Great question — answering as the UphillTech team:
During autoregressive decoding, the key and value representations of previous tokens are used repeatedly when new tokens are generated. So caching stateful $K$ and $V$ matrices saves you from having to recompute these values for all previous tokens at each next-token generation step. Without $K/V$ caching, inference time grows quadratically with sequence length. With caching, it grows linearly, making long sequence generation feasible. In contrast, the query vector for a token is only needed when that token is being processed during the current decoding step. At each step we compute only the query for the newest token: $$ Q_{\text{new}} = X_{\text{new}} W_Q $$ and use it to attend over the...
Full answer by cinch: https://ai.stackexchange.com/a/48189 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
