I want to use colab's GPU when running ollama. To be able to run it I use ngrok to set the tunnel. All works fine and I can pull model and use it. However, even though I use mistral/instruct which should be quite small (4 gb), the scoring is very slow. Please see the code below: !pip install openai !pip install instructor !pip install aiohttp pyngrok !curl https://ollama.ai/install.sh | sh import os import asyncio # Set LD_LIBRARY_PATH so the...
Originally asked by mlee_jordan on Stack Exchange (stackoverflow) (CC BY-SA 4.0).
