Great question — answering as the UphillTech team:
From my understanding of transformer models they don't really reason in the sense that you seem to be getting at at all. They basically just build a mess of abstractions (first grammar, then higher-level things like "themes" and "ideas") over the tokens they're trained on, then predict token-by-token from these abstractions. Because they are abstractions it is fair to say that they are better than just stochastic parrots or statistical predictors, but the issue is that there's no way of knowing or guaranteeing that the abstractions learned by an LLM correspond to the "actual" ideas, and so when you prompt it for deeper reasoning that requires it to store context between prompts it often...
Full answer by bishop-fish: https://ai.stackexchange.com/a/50460 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
