Ollama, how can I use all the GPUs I have?

reno_queen

I am running Ollma on a 4xA100 GPU server, but it looks like only 1 GPU is used for the LLaMa3:7b model. How can I use all 4 GPUs simultaneously? I am not using a docker , just use ollama serve and ollama run . Or is there a way to run 4 server processes simultaneously (each on different ports) for a large size batch process? Wed May 15 01:24:29 2024 +---------------------------------------------------------------------------------------+ |...

Originally asked by notilas on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Just figured out the way running multiple instances with each on a separate port. OLLAMA_HOST=xxx.yyy.255.43:11432 ./ollama serve OLLAMA_HOST=xxx.yyy.255.43:11433 ./ollama serve OLLAMA_HOST=xxx.yyy.255.43:11434 ./ollama serve OLLAMA_HOST=xxx.yyy.255.43:11435 ./ollama serve

Full answer by notilas: https://stackoverflow.stackexchange.com/a/78498419 (CC BY-SA 4.0).

Need it done for your business? UphillTech.