What happened to residual attention?

salon_sue

In 2020, RealFormer introduced residual attention (c): But in 2024, the state-of-the-art DeepSeek transformer model still uses Post-LayerNorm (a): Residual attention is a simple addition operation, thus relatively cheap to implement. If it really improved the transformer, later language models would have adopted this technique. But nobody is discussing it, only residual streams nowadays, so what are the drawbacks of residual attention?

Originally asked by Daniel T on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

RealFormer accumulates attention logits across layers, which can stabilise moderate-scale training. But since then there have been several other advances ( Pre-LN , better init , RMSNorm , μP , etc.) which largely solved those stability issues. Residual attention also reduces layer independence and can bias attention entropy over depth, which may hurt specialisation in very large models. Since it showed no clear scaling-law gains, frontier LLMs didn’t adopt it.

Full answer by Harmony: https://ai.stackexchange.com/a/50424 (CC BY-SA 4.0).

Need it done for your business? UphillTech.