Am I using my GPU right with the AI (llama.cpp)?

tux_tina

My machine has NVIDIA GeForce GTX 1050 Ti card, which normally runs my two monitors utilizing about 1846MiB of the 4096 installed. I built the latest (as of yesterday) llama.cpp and tried using it. Some GGUF files wouldn't load at all -- with the software complaining about lack of video RAM on the card -- but another loaded, even though it complained about being unable to use all capabilities of the model: load_tensors: loading model tensors,...

Originally asked by Mikhail T. on Stack Exchange (superuser) (CC BY-SA 4.0).

uphilltech

Good thread on this — worth a read before you pay anyone to build it.

Discussion: https://superuser.com/questions/1930079/am-i-using-my-gpu-right-with-the-ai-llama-cpp.

Want it set up for you? UphillTech.