series

Cutting LLM cost and latency

How to make LLM apps cheaper and faster without worse answers, in watch order: caching first. Hand-drawn doodle explainers, theory then a practical example.

1 video · 10 min · Play all on YouTube

  1. 1. Caching LLM Calls: Prompt, Exact and Semantic

    Stop paying for the same tokens on every call: how prompt caching works, the prompt order that keeps it hitting, and when exact or semantic caching is safe.

play with it

Cost per finished task

Price per token isn't the bill. Change the numbers and watch the cost per 1,000 finished tasks.

Model A

$12.00 per 1,000 tries
$12.63 per 1,000 finished tasks

Model B

$8.76 per 1,000 tries
$12.51 per 1,000 finished tasks

Model B is cheaper per finished task.

Thinking tokens bill as output; cached reads at a tenth of the input price (prompt caching on Claude); a failed try is paid for and retried. Caching also needs the prompt to reach the model's minimum cacheable length.