AI Compute Constraints

Three observations here. Firstly it has been a really eye opening experience watching the innovation around Stable Diffusion and locally run LLMs and seeing that the unoptimized research code that needed such beefy hardware could actually be optimized to run on consumer hardware given sufficient motivation.Secondly it wasn't obvious that deep learning was going to work as well as it did if you simply threw enough compute at it. Now that this tech has reached critical mass there is a tonn

Yes and no.It’s very expensive to create these models and serve them at scale.Eventually the processing power required to create them will come down, but that’s going to be a while.Even if there was a breakthrough GPU technology announced tomorrow, it would take several years before it could be put into production.And pretty much only TSMC can produce cutting edge chips at scale and they have their hands full.Between Anthropic, xAI and OpenAI, these companies have raised about $84

gonna be awhile before sufficiently powerful hardware is GA for LLM

> but the rate of progression feels like they will soonThe rate of progression seems to be logarithmic - so we got "something looks plausible" but to get that last 10% it's probably going to cost more in HW than just using humans, unless there's some breakthroughs. Just like self driving cars.My impressions at least looking at the developments from a sort of technical perspective - they are hitting all kinds of scaling problems - both in terms of data available, runt

In terms of trajectory, I'm not really convinced we can do much better than we are now. Moore's law is ending in the next decade as we hit fundamental physical limitations in how many transistors we can pack on a chip. The growth in computational power is going to slow down considerably at a time when ai companies are struggling to get more gpu compute to train new models and run inference. OpenAI themselves supposedly stopped sign ups because they're running out of computational

There was just an article from deepmind on HN about this topic the other day[1], but basically IIRC it argues that all of the LLMs are horrendously compute inefficient, which means there’s a ton of room to improve them. So those models will be optimized over time just as the consumer hardware will be improved until eventually one day the two trends will converge. It’s just a question of when that will happen.[1] <a href="https://news.ycombinator.com/item?id=30987885" rel="nofol

Compute has been getting cheaper and models more optimised. So if models can do something it will not be long till they can do this cheap.

Absolutely not. Computers used to be extremely centralized and the decentralization revolution powered a ton of progress in both software development and hardware development.You can run many AI applications locally today that would have required a massive investment in hardware not all that long ago. It's just that the bleeding edge is still in that territory. One major optimization avenue is the improvement of the models themselves, they are large because they have large numbers of par

While OP's reply answers your question, it's important to not apply current costs to predict the future of AI. Hardware for LLMs is one step function away from unimaginable capabilities. That breakthrough could be in performance, cost, or more likely, both.Imagine GPT-4 at 1/1000th the cost. That's where we're going. And you can bet your ass Nvidia is working on it as we speak. Or maybe someone else will leapfrog them like ARM did to Intel.

Haven't people been saying that for the last decade? I mean, eventually they will be right, maybe "about" means next year, or maybe a decade later? They just have to stop making huge improvements for a few years and the investment will dry up.I really wasn't interested in computer hardware anymore (they are fast enough!) until I discovered the world of running LLMs and other AI locally. Now I actually care about computer hardware again. It is weird, I wouldn't have ev