Same Prompt, Different Token Count: The Hidden Cost of Switching Providers

You benchmarked GPT-4o. The cost looked right. You switched to Claude for the longer context window. Your bill went up 20%.

You didn't change your prompts. You didn't change your usage patterns. The only thing that changed was the tokenizer, and tokenizers disagree, sometimes substantially, on how many tokens the same text is worth.

Three Tokenizers, Three Different Answers ¶

The major frontier models use different tokenization schemes:

OpenAI (GPT-4o, GPT-4 Turbo) uses Byte Pair Encoding (BPE) with a 200,000-token vocabulary, implemented via the tiktoken library. BPE builds its vocabulary by iteratively merging the most frequent byte pairs in the training corpus.

Google (Gemini 1.5 Pro, Gemini 2.0) uses SentencePiece with a 262,144-token vocabulary. SentencePiece operates directly on raw text without pre-tokenization, using unigram language model or BPE internally, with vocabulary construction optimized differently from tiktoken.

Anthropic (Claude 3.5, Claude 4) uses a BPE variant with its own vocabulary, sized and trained on Anthropic's corpus mix.

These aren't minor implementation differences. The vocabulary sizes, merge rules, and training corpora produce meaningfully different token counts for the same input text.

Where the Gaps Are Largest ¶

Tokenizer disagreement is not uniform across content types. English prose from typical web text is where all three tokenizers were most heavily optimized. Gaps are smallest here, typically 2–5%.

The gaps grow significantly in three areas:

Code and technical content. Code uses a lot of tokens that appear rarely in natural language: indentation, brackets, underscores, camelCase identifiers, and language-specific syntax. A Python function with heavy use of list comprehensions and f-strings will tokenize very differently across providers. Gaps of 15–25% on code-heavy prompts are common.

Non-English text. BPE vocabularies trained primarily on English text handle other languages less efficiently. A 500-character Japanese paragraph might use 80 tokens under one tokenizer and 140 under another. For multilingual applications, tokenizer choice can double your effective cost.

Structured data formats. JSON, XML, YAML, and Markdown all contain repeated structural characters that tokenizers handle differently. A large JSON payload with many short string values will produce very different token counts across providers.

Checking Token Counts Before You Commit ¶

Each provider exposes tokenizer access:

``python # OpenAI / tiktoken import tiktoken enc = tiktoken.encoding_for_model("gpt-4o") tokens = enc.encode(your_text) print(len(tokens)) `

`python # Anthropic / count_tokens API method import anthropic client = anthropic.Anthropic() response = client.messages.count_tokens( model="claude-opus-4-7-20251101", messages=[{"role": "user", "content": your_text}] ) print(response.input_tokens) `

For Google's Gemini, the countTokens` method in the Gemini API provides equivalent functionality.

The important discipline: benchmark your actual prompts through each provider's tokenizer before committing to a cost model. Generic token-per-word estimates (the common shortcut of "1 token ≈ 4 characters") obscure provider-specific variation that can swing your budget by 20% or more on technical workloads.

The Compression Lens ¶

Tokenizer differences matter most when you're managing context at scale: long system prompts, large codebase injections, multi-turn agent sessions. This is where compression earns its value.

A 15× compression of a long codebase context doesn't just reduce tokens proportionally. It also reduces the surface area where tokenizer inefficiency compounds. A 100,000-token JSON payload compressed to 6,700 tokens through semantic compression doesn't just cost 15× less; it also eliminates 93,300 tokens worth of tokenizer-specific overhead.

For teams running on multiple providers simultaneously (common for cost arbitrage or reliability), compression normalizes the token footprint before it reaches the provider's tokenizer. The compressed representation uses fewer total tokens on every provider, and the relative gaps between providers shrink because compressed text has a higher information density per character.

Practical Recommendations ¶

If you're switching providers or running a cost comparison:

The tokenizer is part of the pricing mechanism. It deserves the same scrutiny as the per-token rate.

Cut your token footprint across every provider with gotcontext.ai →