Bus error on mixtral-instructv01-awq with Vllm

grill_gus

I am getting a bus error when trying to initialize "TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ" model from Huggingface, self.model = LLM( model="TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ", quantization="awq", dtype="auto", tensor_parallel_size=tensor_parallel_size, ) The error i get is: ERROR 2024-02-16T21:46:33.751635551Z *** SIGBUS received at time=1708119993 on cpu 67 *** ERROR 2024-02-16T21:46:33.754929304Z PC: @ 0x7e9291287a37 (unknown)...

Originally asked by Tsvi Sabo on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Good thread on this — worth a read before you pay anyone to build it.

Discussion: https://stackoverflow.com/questions/78011667/bus-error-on-mixtral-instructv01-awq-with-vllm.

Want it set up for you? UphillTech.