Great question — answering as the UphillTech team:
Long context is not long computation. A large context window increases how much text the model can condition on, but it does not automatically increase the model’s ability to perform long-horizon computation. https://arxiv.org/abs/2305.10601 : Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference There are some attempts to mitigate that limitation e.g. if LLMs hey are trained or prompted to externalize intermediate state in a structured way (“show your work”) as shown in https://arxiv.org/abs/2112.00114 : Large pre-trained language models...
Full answer by Franck Dernoncourt: https://ai.stackexchange.com/a/50435 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
