Counting LLM Tokens Without Surprise Bills
LLM API bills scale with a number you do not see until the response comes back. Batch jobs are the easy way to be surprised: a request that costs one cent looks harmless until you multiply it by 50,000 items. The short version of this post is to count tokens before you send, cap output explicitly, and pick the cheapest model that meets your quality bar.
Tokens, briefly
Models read tokens, which are sub-word units produced by a tokenizer. Tokenizers differ between providers and sometimes between generations of one provider's models, so count with the right one:
- OpenAI: GPT-4 models use
cl100k_base; GPT-4o and later useo200k_base. Thetiktokenlibrary counts these locally. - Anthropic: the tokenizer is not published.
POST /v1/messages/count_tokensreturns the input count for a specific model, and counts can change between Claude generations, so re-count when you switch. - Gemini: use the API's
countTokensmethod. - Llama: Llama 1 and 2 use SentencePiece; Llama 3 moved to a tiktoken-style tokenizer. Use the tokenizer that ships with the weights you run.
For a quick estimate, OpenAI's rule of thumb for English is about 4 characters or three quarters of a word per token, so 1,000 words is roughly 1,300 tokens. Code, JSON, and non-English text can differ a lot from that, so use a real tokenizer before you commit to a large run.
Cost arithmetic
The bill for one request is:
cost = (input_tokens / 1_000_000) * input_price + (output_tokens / 1_000_000) * output_price
Output usually costs several times more per token than input. Prices change often and vary by model, so take current numbers from the provider: OpenAI, Anthropic, Google.
Here is a worked example with made-up prices of $3 per million input tokens and $15 per million output tokens. A request with 2,000 input tokens and 500 output tokens costs:
input: 2,000 / 1,000,000 * $3 = $0.0060 output: 500 / 1,000,000 * $15 = $0.0075 total per request = $0.0135 over 50,000 requests = $675.00
The token counter gives a side-by-side estimate for pasted text. It estimates from character ratios and its price list is a snapshot, so confirm against the provider's tokenizer and pricing page before a large job.
Ways to control cost
1. Estimate before batches
Make the estimate part of the job, so the script refuses to start when it is over budget. Use your output cap as the output figure, which gives a worst case:
def projected_cost(n_items, avg_input, max_output, input_price, output_price):
per_request = (avg_input * input_price + max_output * output_price) / 1_000_000
return per_request * n_items
if projected_cost(len(items), 2000, 500, 3.00, 15.00) > BUDGET_USD:
raise SystemExit("Projected cost is over budget")
2. Cap output on every request
An output limit sets the worst case for each call. The parameter name varies: max_tokens on Anthropic's Messages API, where it is required; max_completion_tokens on OpenAI Chat Completions and max_output_tokens on its Responses API; maxOutputTokens on Gemini. If you leave it out on OpenAI, the limit is whatever the model allows. On models that reason before answering, reasoning tokens generally count as output, so allow room for them.
Check the stop reason in each response. A reply cut off by the limit is usually unusable, and retrying it means paying twice.
3. Prefer the cheapest model that works
Smaller models are often good enough for summarization, classification, and extraction, but this depends on the task. Build a small set of real inputs with known good outputs, run it across a few model sizes, and use the cheapest one that passes. Run it again when a provider releases new models, since the lineup changes often.
4. Make output parse the first time
Each retry after a malformed response is another billable request. Where the API offers structured outputs or a JSON schema mode, use it instead of parsing free text. Temperature is a weaker control: setting it to 0 reduces variation but does not guarantee identical output, and some current models, including recent Claude models, reject a temperature setting altogether, so check the model's documentation.
5. Use prompt caching for repeated context
If thousands of requests share a long prefix, such as a system prompt, caching reuses it at a discount. The details differ by provider:
- Anthropic: you mark cacheable content with
cache_control. Reads cost about 10% of the base input price on most models. Writes cost 1.25x for the default 5-minute lifetime and 2x for the optional 1-hour lifetime. Prefixes below a model-specific minimum length are not cached, and no error is returned. - OpenAI: caching is automatic for long enough prompts, and the discount depends on the model.
- Gemini: offers context caching, with a storage charge for explicitly cached content.
// Anthropic request body
{
"model": "MODEL_ID",
"max_tokens": 1024,
"system": [
{
"type": "text",
"text": "You are a support agent for...",
"cache_control": {"type": "ephemeral"}
}
],
"messages": [{"role": "user", "content": "..."}]
}
Using the made-up $3 input price and Anthropic's multipliers, a 5,000-token prefix sent 1,000 times inside the cache lifetime costs $15.00 uncached. Cached, it is one write (6,250 token-equivalents) plus 999 reads (499,500 token-equivalents), or about $1.52.
Caching works on an exact prefix match. A timestamp or request ID near the start of the prompt means nothing after it is reused, so keep changing content at the end. Confirm hits in the usage fields of the response.
6. Batch where the API supports it
OpenAI's Batch API, Anthropic's Message Batches, and Gemini's batch mode all price asynchronous work at 50% of the standard rate, in exchange for results that can take up to 24 hours. For non-interactive work such as overnight classification, this is a discount that requires no change to your prompts.
7. Limit retries and loops
Code that re-prompts on failure keeps spending until something stops it, and an agent loop can do the same. Set a fixed limit on attempts per item and on total requests per job, and use the provider console's spending limits or alerts as a backstop.
Do not re-embed what has not changed
Embedding endpoints are cheap per call, which makes repeated work easy to overlook. If a retrieval pipeline embeds the same documents on every query, it pays again for identical results. Compute embeddings at ingestion and store them, keyed by a hash of the text plus the model name, then re-embed only when one of those changes.
Quick checklist
- Estimate cost per request times volume before running batches
- Set an output limit on every API call and check the stop reason
- Pick the cheapest model that passes your own evaluation set
- Use structured outputs to avoid retries on malformed responses
- Keep long static prefixes first and cache them
- Use batch APIs for non-urgent workloads
- Hard-cap retries and total requests
- Compute embeddings once and store them
- Monitor daily spend and set an alert below your budget