LLM Progress Plateau
The cluster debates whether Large Language Models (LLMs) are hitting a plateau in improvements due to exhausted training data, scaling limits, and diminishing returns, or if exponential progress will continue through new breakthroughs and compute scaling.
Activity Over Time
Top Contributors
Keywords
Sample Comments
Given the rate of improvement wrt to llms, this may not hold true for long
LLMs have been improving exponentially for a few years. let's at least wait until exponential improvements slow down to make a judgement about their potential
Current LLMs are already trained on the entirety of the interwebs, including very likely stuff they really should not have had access to (private github repos and such).GPT-5 and other SoTA models are only slightly better than their predecessors, and not for every problem (while being worse in other metrics).Assuming there is no major architectural breakthrough[1], the trajectory only seems to be slowing down.Not enough new data, new data that is LLM generated (causing a "recompres
Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5).I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
Seems llm progress really is plateauing. I guess that was to be expected.
Good. LLMs progress is too stagnant, it was time they started to play seriously
You seem doubtful that there will be a lot more progress. It's been well documented how much progress we have been through and how many new paradigms have arisen when we think we can't progress.It's spectacularly short-sighted, in my opinion, to assume that we won't make progress.Also, some of the prompted or fine-tuned LLMs today are actually very close.
What you say might be true for the current crop of LLMs. But it's rather unlikely their progress will stop here.
Well, expectations vary widely.On one hand, recent models seem to be less useful than the previous generation of them, the scale needed for training improved networks seems to be following the expected quadratic curve, and we don't have more data to train larger models.On the other hand, many people claim that what tooling integration is the bottleneck, and that the next generation of LLMs are much better than anything we have seen up to now.
That matches my experience too. I wonder how fast they'll improve and if LLMs will hit a wall, as some AI experts think.