Why do we need masking and context window during inference in LLMs?

fixit_fiona

For a more concrete discussion, if we focus on the GPT-2 model, which is an auto-regressive model, I fully understand why we need masking for training. However, I need clarification on why we need to mask for inference. During inference, the model uses all information to predict the next token, so we should not need masking. Now, if we do not need masking for inference, I don’t understand the restriction of having a context window in LLMs. The...

Originally asked by F Gh on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Masking is used during inference primarily for consistency and efficiency, not because it's strictly necessary. The forward pass computation of the transformer is designed to process the input sequence in parallel for efficiency and the mask ensures that even though all tokens are computed at once, only valid attention is applied. For example, when generating the 11th token given a 10-token sequence, the model may internally compute the probability distributions for all tokens up to the 11th in one go. This is computationally more efficient on modern GPU/TPU hardware where matrix multiplications are highly optimized. Computing the entire matrix at once can leverage parallel processing...

Full answer by cinch: https://ai.stackexchange.com/a/46658 (CC BY-SA 4.0).

Need it done for your business? UphillTech.