Caching LLM Calls: Prompt, Exact and Semantic

Where the cost goes

Theory

Every call bills every input token, and the model reads the whole prompt before the first word of the answer. When a long guide or set of rules is resent with each question, most of every prompt is a repeat.

Practical example: A support bot

An 8,000-token help guide plus a 200-token question: about 97% of each prompt is the same on every call.

The KV cache

Theory

While reading, the model computes keys and values for each token, which later tokens refer back to. Prompt caching keeps that work for the unchanged start of a prompt, so the next request with the same start only processes the new part. It stores this work, not your text or an answer, so the output is still generated fresh.

Practical example: Guide cached, question fresh

Request two sends the same guide and a new question; the guide is read from the cache and only the question is processed.

Prompt caching in practice

Theory

Some providers cache automatically for supported models; others ask for a cache_control marker at the end of the reusable part. Writes can cost a little more than normal input and reads much less. Caches expire after a few minutes without use, and prompts under a minimum length are not cached, with no error.

Practical example: Mark the system block and check usage

Add cache_control to the system block, then read the usage: cache_read_input_tokens should equal the cached size. If both cache counts are 0, nothing was cached.

GUIDE = open("guide.md").read()
system = [{"type": "text", "text": GUIDE, "cache_control": {"type": "ephemeral"}}]

Prefix order

Theory

A cache hit needs an identical prompt up to the marker, and the cache is layered: tools, then system, then messages. A change at one layer invalidates it and everything after it.

Practical example: Move the clock

Put tools, instructions and the guide first; put the current time, user details and the question last. Don't reshuffle tools or examples per request.

Exact caching

Theory

If whole requests repeat, store the answer yourself under a hash of the full request and skip the model call. Identical requests can get different answers, so a cache freezes one of them; include the model and a prompt version in the key so upgrades don't serve stale answers.

Practical example: One line in Redis

Check the cache first; on a miss, call the model and store the answer under the request hash with a one-day expiry.

import hashlib, json, redis
r = redis.Redis()
req = {"model": "my-model", "temperature": 0, "prompt_version": "v3",
       "messages": [{"role": "user", "content": "Reset my password"}]}
key = "llm:" + hashlib.sha256(json.dumps(req, sort_keys=True).encode()).hexdigest()
answer = r.get(key)
if answer is None:
    answer = "Go to Settings, then Security, then Reset password."  # your model call here
    r.set(key, answer, ex=86400)

Semantic caching

Theory

Embed each question and reuse a cached answer when a new question is similar enough, above a threshold. Loose thresholds return wrong answers for questions that only look alike; strict ones miss real repeats. A single default threshold copied from a tutorial is the common mistake; tune it on your own data.

Practical example: Test the threshold

Label 50 real question pairs as same answer or different, then pick the threshold where wrong hits are rare enough. Never share answers that depend on who is asking.

Which cache, when

Theory

Prompt caching first: it does not change answers and is often one field. Add exact caching for identical repeats where a frozen answer is fine, and semantic caching last, for generic answers only. Track cached tokens and hit rate.

Practical example: The support bot plan

Prompt caching on the guide, exact caching on the top help buttons, semantic caching only if a tested threshold holds.