Why do larger language models still fail on simple compositional reasoning...

handyman_hank

Large language models often perform impressively on benchmark tasks, coding, and natural language generation, but they can still fail on reasoning problems that seem simple for humans, especially when multiple constraints must be combined step by step. I want to understand this from a technical perspective. Why do these models remain weak on compositional reasoning despite scaling in parameter count and training data? Is this primarily a...

Originally asked by Avalon Brooks on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Simple answer: LLM are not designed for Reasoning . As I suppose you know, they are designed to generate the most likely next word (token) in a sequence. More to the point... despite scaling in parameter count and training data? Brute force can only get you so far, law of diminishing returns... Is this primarily a limitation of autoregressive next-token prediction Yes. As simple as that. training objective mismatch You can say that as well, they are not trained for Reasoning . Don't expect LLM is equivalent to AI (*) I'd say that for Reasoning we need to make the AI (intentionally not using LLM here) to learn more like humans do, explore the environment. (*) I disagree we need to dilute the...

Full answer by adsp42: https://ai.stackexchange.com/a/50500 (CC BY-SA 4.0).

Need it done for your business? UphillTech.