I am trying to convert a flan-t5 pytorch model to GGUF format

taco_truck_tom

I have tried to convert the model using the llama.cpp convert.py following the colab note HERE . model = AutoModelForSeq2SeqLM.from_pretrained( model_name, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map='cpu', offload_folder='offload', cache_dir=cache_dir ) Facing an error at !python convert.py models/ Error: Loading model file models/pytorch_model.bin Traceback (most recent call last): File "/content/llama.cpp/convert.py", line...

Originally asked by Akella Niranjan on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

This can be achieved using the candle framework resources to convert encoder-decoder models. Here are the GGUF conversions of T5 Models

Full answer by Akella Niranjan: https://stackoverflow.stackexchange.com/a/78903037 (CC BY-SA 4.0).

Need it done for your business? UphillTech.