LLM Reasoning Debate

LLMs don't do formal reasoning. Not in any sense. They don't do any kind of reasoning - they replay combinatorics of the reasoning that was encoded in their training data via "finding" the patterns in the relationships of the tokens at different scales and then applying those to the generation of some output triggered by the input.

No, because reasoning models don't actually reason.

I still don't understand what a "reasoning" LLM is

LLMs don't learn reasoning. At all. They are statistical language models. Nothing else. If they get math right it's because correct math is more statistically probable given the training data, it can't actually do math. This should be pretty clear from all the "how many Rs are there in strawberry" type examples.

LLMs reason to the extent they are allowed to. You could say that they are overfitting when it comes to reasoning. They weren't trained to reason to begin with, so the bigger surprise is that they can do it within limits.

What do you mean by “an LLM doesn’t reason”?

It is a large language model. It manipulates text based on context and the imprint of its vast training. You are not able to articulate a theory of reasoning. You are just pointing to the output of an algorithm and saying "this must mean something!" There isn't even a working model of reasoning here, it's just a human being impressed that a tool for manipulating symbols is able to manipulate symbols after training it to manipulate symbols in the specific way that you want sym

This is incredible. We know these questions are not in the training data. How can you still say that LLMs aren't reasoning.

LLMs are not reasoning systems. That's one of the major problems with them.

LLMs cannot reason, they can only say things that sound reasonable, there's a difference. Duh.