Great question — answering as the UphillTech team:
Answer you accepted is completely wrong! Importance Matrix has nothing to do with I-Quant. Importance Matrix is not I-Quant Important Matrix can be used to BOTH K-Quants (Q4_k_m, etc) and I-Quants (IQ4_XS, etc). It's completely different thing. That, is why you see imatrix quants with K-Quants like one here: Or gguf-my-repo which does the quantizations for you, has no difference in quant methods regardless of imatrix option. So what is I-Quant then Quant methods( n here means bpw, # here means suffix): I-Quant: IQn_# (IQ4_XS, IQ3-XXS, IQ4_NL, etc) Introduced in llama.cpp #4773 influenced by QuIP# Acts like compressed file, using lookup table to save extra space, basically trading off speed...
Full answer by jupiterbjy: https://stackoverflow.stackexchange.com/a/79278280 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
