Why don't we bake system prompts with fine-tuning?

ubuntu_ulysses

I just saw that Claude Code has a system prompt with a length of roughly 20–25K tokens. At a scale like Claude’s, this would add up to millions—or even billions—of tokens processed, potentially costing microseconds of GPU inference time per query, which in aggregate could translate into millions of hours. I was wondering whether a context of that length could be sufficiently represented as a learned mode via a fine-tuned Claude for this task,...

Originally asked by dasdadsaads on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

I assume they utilize prefix caching, which means that mathematical operations needs to be carried out only once for each unique system prompt (or prefix). The per-prompt processing happens only after the common prefix. But since you think that processing the system prompt only takes some microseconds, did you already know about prefix caching? Integrating the system prompt directly into weights is an interesting idea, I'll let others answer that one.

Full answer by NikoNyrh: https://ai.stackexchange.com/a/50265 (CC BY-SA 4.0).

Need it done for your business? UphillTech.