How do I access a llama.cpp server instance with the Continue extension for...

chef_carol

If I'm running GLM-4.7-Flash-GGUF:Q6_K_XL from the powershell terminal like this .\llama-server.exe -hf unsloth/GLM-4.7-Flash-GGUF:Q6_K_XL --host 127.0.0.1 --port 10000 --ctx-size 32000 --n-gpu-layers 99 , how do I access it from the Continue plugin in VSCodium? The "Add Chat model" optional only shows pre-configured cloud based API option like Claude and ChatGPT, and the only local models I can find is Ollama and a version of Llama.cpp that...

Originally asked by ChristianOConnor on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

What i have seen as i am also a new User using llama.cpp Continue Extension llama.cpp Provider (earlier used ollama with continue) You have to also provide the api base like Below in config.yaml apiBase: http://localhost:8080 I am following those instructions and using Qwen on local system. Screenshot Below

Full answer by TIMEX Peachtree: https://ai.stackexchange.com/a/50484 (CC BY-SA 4.0).

Need it done for your business? UphillTech.