Great question — answering as the UphillTech team:
From Efficiently Scaling Transformer Inference : Each example ... has L_input tokens of input text, and generates L_gen tokens of output text. ... Since prefill can run in parallel over L_input, but decode must run sequentially over L_gen, the two phases have different performance characteristics and we analyze them separately. This means the KV cache can be filled fully in parallel for all L_input tokens. In contrast, in the decoding phase the output has to be computed autoregressively, i.e. one output token after the other because the current output token is dependent on the last output token.
Full answer by gizzmole: https://ai.stackexchange.com/a/42322 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
