What are the theoretical reasons behind calculating next-token loss for...

garage_greg

In Karpathy's video, when calculating the loss of a next-token LLM during pretraining for a datum, the loss is calculated as the sum of the logs of the probability the first token generates the second token, the second token generates the third, the third token generates the fourth, etc. for all tokens in the context window: https://www.youtube.com/watch?v=l8pRSuU81PU&t=14181s However, generalizing the derivation for entropy rate (...

Originally asked by Leo Miao on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Not all trainings want to predict the next word, some also want to predict a missing word in the middle of a sentence with the rest of the sentence known. It is also more efficient and scales better on multiple GPUs if you do one big operation instead of multiple small ones and you don't need to recomputed data (you could cache the result in single Token mode though). Also, the training is more stable when you mix the result of multiple tokens in an update step.

Full answer by joerg-de: https://ai.stackexchange.com/a/50187 (CC BY-SA 4.0).

Need it done for your business? UphillTech.