Can I run open weight models on regular DDR ram instead of GPUs?

soup_sam

If I want to run an open weight model on my local server the general assumption is that I need GPUs with enough ram to fit the model. Nvidia offers various GPUs in the RTX series and there are a few other options as well. I was pondering whether I can use regular DDR4 or DDR5 ram on a server instead. DDR ram is much cheaper than GPUs and in a corporate setting one may already have a server with say 0.5 TB or 1 TB DDR ram already standing around...

Originally asked by quarague on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

There is more and more support for running models on cpus. However, when switching to cpus, the bottleneck for speed is almost always your actual cpu processing power. Gpus are incredibly fast at matrix multiplication, something cpus are not optimized for. As such running any type of Ai model on CPUs is mostly dependent on how much you can optimize the models operations on CPUs, not how fast the transfer is from RAM to CPU. There is also more and more support for keeping weights in RAM and doing computation on GPU, continually transferring weights from RAM to GPU and back. In this case you are bottlenecked by transfer speed like you were describing. But then it's not just access speed. 20x...

Full answer by Robin van Hoorn: https://ai.stackexchange.com/a/50826 (CC BY-SA 4.0).

Need it done for your business? UphillTech.