Ollama model quality is worse than similar huggingface model

kernel_kate

I use the following configuration of huggingface model: quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, ) tokenizer = AutoTokenizer.from_pretrained( "google/gemma-2-9b-it", token=access_token ) model = AutoModelForCausalLM.from_pretrained( "google/gemma-2-9b-it", quantization_config=quantization_config, torch_dtype=torch.bfloat16, token=access_token, ) Such config is...

Originally asked by Andrey on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

You have first to transform your model of choice from hugging face to the GGUF format. You can follow this: https://github.com/ggml-org/llama.cpp/discussions/2948 Or, you search for some GGUF version somebody uploaded for your model in huggingface: Search a high quality model - but as small as you wish https://huggingface.co/bartowski/gemma-2-9b-it-GGUF search for the model here: https://huggingface.co/bartowski/gemma-2-9b-it-GGUF/tree/main and download it. => This is by the way the answer to your question - your ollama model was one of the worse ones - smaller. Q4 is smaller ~5Gb and thus worse. The most complete model is big - ~37Gb. But the smaller ones ~6-9Gb seem to be of very high...

Full answer by Gwang-Jin Kim: https://stackoverflow.stackexchange.com/a/79539170 (CC BY-SA 4.0).

Need it done for your business? UphillTech.