GPU vs CPU Limitations

No? GPUs are just extremely parallel much wider SIMD cores

GPUs are pretty tailored and aren't really good for general purpose computing. Branching and cache coherency are much easier in the CPU compared to the GPU. I doubt that any of the advertised gains would be realized by normal users.

GPUs are only useful if your problem is data parallel. The majority of compute intensive problems that are also data parallel at the same time have been shifted to GPUs and SIMD instructions already. A GPU isn't some pixie dust that makes everything faster.

Isn't some of the advantage coming from the fact that gpus are much better designed for massively parallel computations?

GPUs are made to execute a limited set of the same operations operation on on a huge amount of data in parallel. This is a totally different workload than your usual computer programs, that commonly have a long and complicated series of commands. Thus just translating your program 1:1 to your GPU computer would make it way slower. Your GPU runs probably around ~1.5GHz, a third of your CPUs. The GPU does have a few thousand cores that run in parallel while our consumer CPUs does have at best a do

Even processor agnostic programs will run poorly on a GPU. GPUs are about doing very basic tasks very quickly. CPUs can do more but are slower. Most computer programs take advantage of the extra features in a GPU and would not be able to run on a GPU.

Even processor agnostic programs will run poorly on a GPU. GPUs are about doing very basic tasks very quickly. CPUs can do more but are slower. Most computer programs take advantage of the extra features in a GPU and would not be able to run on a GPU.

That seems closer to the architecture of a GPU than a CPU.

No, GPUs are quite different than CPUs with more cores. They're optimized for Singl-Instruction-Multiple-Data algorithms, in which the same operation is done at the same time to many different inputs (think image or video processing). They do very poorly in cases with complex branching logic.

No one has done it because it’s a comically bad idea. It’s like trying to build taxis with only rail infrastructure. GPUs are only fast because they do the same thing across multiple pieces of data at the same time. Branching is possible but incredibly expensive. You would end up with terrible performance and poor power efficiency. And you would still need a cpu to load programs on the GPU and handle io.