AI Training Costs
The cluster focuses on the high financial and computational costs of training large AI models like GPT-3 and GPT-4, with debates on exact figures, feasibility for hobbyists versus corporations, and trends in decreasing expenses.
Activity Over Time
Top Contributors
Keywords
Sample Comments
fully training a ~170 billion parameter model on the cheapest cloud instance probably sets you back at least a few million bucks so unless OP is Tony Stark that might not be as trivial as you think
It costs ~$4.5M to train GPT-3. How big is your group of hobbyists?
Are they even factoring in the cost of compute required to train their systems ?
Yes, but 1) you only need to train the model once and the inference is way cheaper. Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it. 2) the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA
High training times could be cost prohibitive. Currently, its over $100mil to train GPT4 from scratch (which possibly includes other costs related to RLHF and data acquisition). Not sure how this model compares, but its likely not cheap.
Source? Large model training runs cost more than flying a plane across the atlantic, so this doesn’t sound right.
why not? people are spending 6k on cloud costs to train models in this thread. it pays for itself after two models.
It’s the cost of compute hardware required to train a model of that size
Maybe. The problem is that you need billions to train new models.[1] At least with how things are now.[1] https://techcrunch.com/2023/04/06/anthropics-5b-4-year-plan-...
That sounds cheap. Can I really train a 13b model from scratch for just USD 2000?