Great question — answering as the UphillTech team:
External memory indexed from attention - https://arxiv.org/pdf/2112.04426.pdf Regarding minimizing cost of autoregression, one could do encoder-decoder (eg T5) architecture, but with a twist : Make the decoder deliberately much smaller network than the encoder which is doing most of the heavy lifting, but running only once. Such asymmetry works on assumption that "speaking" a concept from the latent space vector is a simpler task compared to "understanding" it done in the encoder. If this paper is correct, this allows you to run T5-FLAN at 6x inference speed at a cost of marginal (<5%) loss. The paper utilizes pruning (ie downsizing pre-trained model), better scaling might be possible if...
Full answer by kat: https://ai.stackexchange.com/a/42264 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
