Ollama isnt using my gpu on a runpod.io pod

soup_sam

I am testing different AI models on runpod.io. One of those models is dolphin-mixtral:8x22b. I followed Runpod's tutorial for setting up the pod with Ollama: https://docs.runpod.io/tutorials/pods/run-ollama , and I used the H100SXM with 80GB VRAM and 16 vCPU 125 GB RAM. However, when I start the model and ask it something like "hey," it uses 100% of the CPU and 0% of the GPU, and the response takes 5-10 minutes. How to make Ollama use my GPU? I...

Originally asked by Anton on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

I ran into the same issue, found the answer in this Reddit post . Set the CUDA_VISIBLE_DEVICES environment variable to 0,1 before running ollama serve : :/# export CUDA_VISIBLE_DEVICES=0,1 :/# echo $CUDA_VISIBLE_DEVICES 0,1 :/# ollama serve Tested on an A4000 pod.

Full answer by Julian Patdu: https://stackoverflow.stackexchange.com/a/78743675 (CC BY-SA 4.0).

Need it done for your business? UphillTech.