Great question — answering as the UphillTech team:
Transformers use self-attention, unlike RNNs/LSTMs which propagate information sequentially and as result suffer from vanishing/exploding gradients and compression bottleneck in their hidden state. Self-attention still suffer from issues though that sometimes cause them to fail to detect long-term dependencies, such as: Attention weights dilute over long contexts. Positional encoding degrades at long distances .(e.g., RoPE extrapolation). Models are trained mostly on shorter sequences. No explicit memory, which makes multi-hop reasoning harder. In practice, the ability of LLMs to handle long contexts varies greatly between LLMs, e.g. see this ICML 2025 paper: NoLiMa: Long-Context Evaluation...
Full answer by Franck Dernoncourt: https://ai.stackexchange.com/a/50419 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
