Curse of Dimensionality
The cluster discusses the curse of dimensionality in machine learning, focusing on overfitting risks when models have more parameters than data points, and the challenges of high-dimensional spaces despite regularization techniques.
Activity Over Time
Top Contributors
Keywords
Sample Comments
please elaborate - are you thinking of the curse of dimensionality ?
https://en.m.wikipedia.org/wiki/Curse_of_dimensionalitySounds like you are both right?
The problem is not asthetics the problem is overfitting the data. Add enough knobs and you can approximate anything, but your simply encoding data not finding anything new.
Parameters don't have diminishing returns so much as we don't have enough (distinct) data to train models to use that many parameters efficiently.
I'm not an expert on LLMs but my guess would be that this is a result of the curse of dimensionality. As a general rule more dimensions != more better.
In general, it's OK to use more parameters than data points even, if you properly use regularization (such as weight decay), for example. Other times, even significantly fewer parameters than data points can be wrong.
That's not in any way surprising. When you have more parameters than data, this is trivial.
We're all suffering from the curse of dimensionality.
Because your 400 gazillion degrees of freedom machine learning core was not already ill conditioned enough for you?
I see, so it's the high dimensionality of parameters to fit rather than, say, a small number of parameters for millions of points.