Cutting LLM cost and latency, part 2 of 2

Why a Cheaper AI Model Can Cost You More

A model that's five times cheaper per token can cost more per finished task. Five hidden costs, added up on one support bot, and how to check before you switch.

9 min · 2026-11-01 · Watch on YouTube · Subscribe

After this video

You can compare two AI models by cost per finished task, using token counts, usage fields and a pass rate from your own test set.

The short version

You pay per token, but your app runs on finished tasks. A model that's 5x cheaper per token can cost more per finished task because of five costs the price page doesn't show: its tokenizer may count your text bigger, reasoning models bill hidden thinking as output, some models write longer answers, wrong answers get retried, and your prompt may be too short to cache on the new model. In our support-bot example, the 5x-cheaper model ended up about 1.5x as expensive per finished ticket.

Cost per task, not price per token

Input and output tokens are priced separately, and output usually costs more. The number that matters is what a finished task costs.

The support bot. 1,000 tickets at 2,000 tokens in and 400 out: model A ($3 in, $15 out per million) costs $12, model B ($0.60 in, $3 out) costs $2.40 on paper.

Tokenizers count differently

Each model family cuts text into tokens its own way, depending on language and content, so the same prompt can be more tokens on another model. Count one real prompt on both models before you switch.

One Hindi sentence. An older OpenAI tokenizer made 36 tokens of it, a newer one 11.

Hidden thinking and chatty answers

Reasoning models think in tokens you never see but pay for as output, the expensive kind. Some models also write longer replies. Read the usage the API returns, set an output limit, lower reasoning effort on easy tasks, and ask for the format you want.

Retries

A wrong answer is paid for and then paid again. Cost per finished task is the cost per try divided by the pass rate, which you get by scoring both models on the same test set.

The math. A: $12 / 0.95 = $12.63 per 1,000 finished tickets. B: $8.76 / 0.70 = $12.51.

The cache that never kicks in

Prompt caching reads cost a tenth of normal input, but each model has a minimum prompt length it will cache. Below it nothing is cached and nothing errors. Check the cache read count in the usage.

The check. Run fifty real tickets through both models, log input, output, thinking and cache tokens, score every answer the same way, and compare cost per finished task.

Check yourself

  1. Model B is 5x cheaper per token but passes 70% of tasks; model A passes 95%. How do you compare them?

    Divide each model's cost per try by its pass rate to get cost per finished task, then compare those.

  2. Why can the same prompt be more tokens on a different model?

    Each model family has its own tokenizer, so the same text can split into more or fewer tokens, depending on language and content.

  3. Your prompt caching stopped saving money after a model switch and nothing errored. What do you check?

    The cache read count in the usage: if it's zero, the prompt may be below the new model's minimum cacheable length, which caches nothing without an error.

Sources

Every claim in this video was checked against these pages. Open one to see the claims it backs and the exact words they rest on.

  • platform.claude.com/…/pricing 4 claims
    • API prices are per million tokens, input and output priced separately, and output costs more than input (on this model 5x).
      Claude Sonnet 5.5 The best combination of speed and intelligence $2 / MTok $10 / MTok
    • How many tokens the same text becomes depends on the model's tokenizer, the language and the type of content.
      How many tokens the same text produces depends on the model’s tokenizer, the language, and the type of content.
    • A newer tokenizer can produce about 30% more tokens for the same text (Anthropic's 4.7+ tokenizer vs the previous one).
      This tokenizer produces approximately 30% more tokens for the same text.
    • You can count how many tokens a request uses on a specific model before switching, with that provider's token counter and the model's id.
      To see how many tokens a request uses on a specific model, count it with Token counting and that model’s ID.
  • raw.githubusercontent.com/…/README.md 4 claims
    • The same Hindi sentence is 36 tokens with OpenAI's older cl100k_base tokenizer and 11 tokens with o200k_base; a short English sentence is 10 tokens in both.
      tiktoken 0.12: len(get_encoding('cl100k_base').encode('आपका ऑर्डर मंगलवार को भेजा जाएगा।')) = 36; o200k_base = 11; 'The quick brown fox jumps over the lazy dog.' = 10 and 10
    • BPE tokenizers turn text into tokens; on average a token is about 4 bytes of text (for OpenAI's tokenizers).
      On average, in practice, each token corresponds to about 4 bytes.
    • BPE tokenizers learn common subwords from text, so frequent words become single tokens and rare words split into pieces.
      It attempts to let the model see common subwords.
    • 'Your ticket has been closed.' is 6 tokens with OpenAI's cl100k_base tokenizer and the same message in Hindi is 28; with o200k_base the English is 6 and the Hindi 8 (a third more).
      tiktoken 0.12, cl100k_base: English 'Your ticket has been closed.' = 6, Hindi 'आपका टिकट बंद हो गया है।' = 28 (28/6 = 4.7); o200k_base: English = 6, Hindi = 8 (8/6 = 1.33)
  • developers.openai.com/…/reasoning 4 claims
    • Reasoning tokens are not shown in the API answer but are billed as output tokens.
      While reasoning tokens are not visible via the API, they still occupy space in the model’s context window and are billed as output tokens
    • The usage object reports reasoning tokens separately; in the docs' example, 1,024 of 1,186 output tokens were reasoning.
      "output_tokens" : 1186 , "output_tokens_details" : { "reasoning_tokens" : 1024 }
    • max_output_tokens caps everything the model generates, reasoning tokens included.
      you can limit the total number of tokens the model generates, including reasoning tokens, visible output tokens, and non-visible formatting tokens, by using the max_output_tokens parameter
    • Reasoning models take a reasoning effort setting with the request.
      specify your reasoning model and reasoning effort
  • platform.claude.com/…/extended-thinking 1 claim
    • Claude reports how many billed output tokens were internal reasoning in usage.output_tokens_details.thinking_tokens.
      monitor the usage.output_tokens_details.thinking_tokens field in the response, which reports how many of the billed output tokens were internal reasoning
  • platform.claude.com/…/prompt-caching 6 claims
    • On Claude, cache reads cost a tenth (0.1x) of the normal input price on most models; some newer ones are lower. Other providers price caching differently.
      Cache read tokens are 0.1 times the base input tokens price (see the table footnote for per-model exceptions).
    • The minimum prompt length that can be cached differs from model to model.
      2,048 tokens for Claude Mythos Preview and Claude Opus 4.7 4,096 tokens for Claude Opus 4.6 and Claude Opus 4.5
    • A prompt shorter than the model's minimum is simply not cached, and no error is returned.
      Any requests to cache fewer than this number of tokens are processed without caching, and no error is returned.
    • If cache_creation_input_tokens and cache_read_input_tokens are both 0, the prompt was not cached.
      if both cache_creation_input_tokens and cache_read_input_tokens are 0, the prompt was not cached
    • Total input tokens on Claude = cache reads + cache writes + input_tokens.
      total_input_tokens = cache_read_input_tokens + cache_creation_input_tokens + input_tokens
    • On Claude, writing to the 5-minute cache costs 1.25x the base input price, so caching pays off only when the prefix is reused.
      Cache write tokens with a 5-minute TTL are 1.25 times the base input tokens price.

Spotted a mistake? Say so in the video's comments. Fixes are listed on corrections.