Is next-token prediction sufficient to explain emergent capabilities like...

cafe_carl

I recently watched the video "Large Language Models explained briefly" by 3Blue1Brown ( https://youtu.be/LPZh9BOjkQs ), which explains that a Large Language Model (LLM) is essentially a sophisticated mathematical function designed to predict the next word (or token) in a sequence. While the mechanics of the Transformer architecture and the training process (pre-training on trillions of examples followed by RLHF) are clear, I find it...

Originally asked by Humberto José Bortolossi on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

My question is: Are LLMs truly performing all tasks (including code generation and reasoning-like behavior) solely through next-token prediction Yes, but the complex behavior is understandable when you consider that each token produced affects the next token predicted. It’s a document completion algorithm, and it’s trained on enormous datasets (GBs-TBs of text documents) and so when it has been trained on, say, code databases, it can easily predict an (almost) complete script in Python, for example, because it has captured the necessary patterns from the scripts it was trained on. Attention is key, because it attributes importance to specific words prior to the next token. This is what...

Full answer by Mr. AI Cool: https://ai.stackexchange.com/a/50236 (CC BY-SA 4.0).

Need it done for your business? UphillTech.