Are there any non-transformer LLMs?

fixit_fiona

Almost all LLMs are based on the transformer architecture, but are there any examples of ones that don't use transformers?

Originally asked by user84165 on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Are there any non-transformer LLMs? Mamba . One may also combine Mamba with Transformers, e.g. see Taipan: Efficient and Expressive State Space Language Models with Selective Attention : This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. About Mamba, some evaluations from https://huggingface.co/tiiuae/falcon-mamba-7b To use for inference (code from https://huggingface.co/tiiuae/falcon-mamba-7b ): from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "tiiuae/falcon-mamba-7b" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")...

Full answer by Franck Dernoncourt: https://ai.stackexchange.com/a/46483 (CC BY-SA 4.0).

Need it done for your business? UphillTech.