Convert safetensors model format(LLaVA model) into gguf format

reno_rita

I want to do LLaVA inference in ollama, so I need to convert it in gguf file format. My model has the file format safetensors.(trained with lora) It seems that ollama supports only llama, but not llava as shown here, https://github.com/ollama/ollama/blob/main/docs/import.md I followed the instruction of llama.cpp, and used the code convert_lora_to_gguf.py here, https://github.com/ggerganov/llama.cpp/blob/master/convert_lora_to_gguf.py But I get...

Originally asked by Jiyong Jeong on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

https://medium.com/@manish.thota1999/an-experiment-to-unlock-ollamas-potential-video-question-answering-e2b4d1bfb5ba python3 /home/myles/llama.cpp/examples/llava/llava_surgery_v2.py -C -m /home/myles/llama.cpp/checkpoint-300 import argparse import os import json import re import torch import numpy as np from gguf import * from transformers import SiglipModel, SiglipProcessor, SiglipVisionModel TEXT = "clip.text" VISION = "clip.vision" def k(raw_key: str, arch: str) -> str: return raw_key.format(arch=arch) def should_skip_tensor(name: str, has_text: bool, has_vision: bool, has_llava: bool) -> bool: if name in ( "logit_scale", "text_model.embeddings.position_ids",...

Full answer by Mylez Goose: https://stackoverflow.stackexchange.com/a/78998355 (CC BY-SA 4.0).

Need it done for your business? UphillTech.