The Hidden AI Token Cost Developers Must Know

The why

TL;DR; jump to practicalities here.

In previous articles we talked about AI as a stack and a bit about tokens and LLMs learning path. There is much much more to cover, and we will do that! But so far we know, that AI is a stack.

Not magic,
Not a brain in the cloud,
Not a tiny senior developer trapped inside a GPU cluster, waiting patiently to review your PR.

A stack.

Models, data, training, inference, product wrapper, safety layer, and all the interesting engineering parts that decide whether something works outside of your local demo.

Now let’s talk about the pert that usually appears after the demo – the bill.

Because building with AI is fun until someone from finance, broadly understood, asks why the “small prototype” is now eating money every time a user asks it to summarize a meeting they should probably have skipped anyway.

AI cost looks pretty simple from far away:

Price per milion tokens.

Nice. Clean. Amost harmless.

Almost is a keyword.

Because then you start building.

You add system prompts.
You add conversation history.
You add RAG.
You add tool calls.
You add retries.
You add JSON mode.
You add “please be concise” and the model writes a novel anyway.

Suddenly, the cost is not “one API call”.

It is input tokens, output tokens, cached tokens, embedding tokens, retrieved documents, tool responses, agent loops, failed attempts, logs, evaluations, and that one user who pastes a 300-page PDF into a chat box because apparently we allow that now.

So today, let’s take a proper engineer-to-engineer look at AI cost.

Not the marketing version, no,no – the practical one.

The version you need before putting an AI feature into production and discovering that your unit econimics are held together by optimism and free trial.

First thing first – what is a token – the reminder

You can read more about this topic in this article. Here we’ll just briefly remind you the core concepts.

A token is the unit of text that a language model reads and writes.

Not exactly a word, nor character, nor a byte.

A token can be a whole word, part of a word, punctuation, whitespace, or some weird little text fragment that exists because tokenizers enjoy humbling us.

For example, this sentence:

Hello, developers!

Will not necessarily be counted as three tokens.

Depending on the tokenizer, it may be split into pieces like:

Hello
,
dev
elopers
!

The model does not see text the same way we do. It sees token IDs.

Before text reaches the model, it goes through tokenization. The tokenizer converts text into numbers from a fixed vocabulary. The model then processes those numbers.

In a very simplified version it looks like this:

And then the model generates output token by token.

This all means that cost is usually based on two big buckets:

input tokens = everything you send to the model 
output tokens = everything the model generates back

Oh that’s easy.
-every delevoper at least once in their life before a production incident

Input tokens: the stuff you pay for before the model says anything

Input tokens are all the tokens you send to a model. But remember ALL THE TOKENS, not just the visible plain text. ALL. OF. IT.

That includes:

system instructions,
developer instructions,
user message,
previous conversation history,
retrieved documents,
function or tool definitions,
tool results,
formatting instructions,
examples,
hidden framework wrappers,
and sometimes metadata added by the SDK or platform.

So when a user writes:

> Can you summarise this?

It might seem tiny, but the actual request may look more like:

System: 

You are a helpful assistant for Contoso Support. Follow company tone. Never reveal internal policies. Return valid JSON. 

Developer: 

Classify the request. Use the provided documents. Mention uncertainty. Do not answer from memory. 

Conversation history: 
[user and assistant messages from the last 20 turns] 

Retrieved context: 
Document 1: ... 
Document 2: ... 
Document 3: ... 

User: 

Can you summarize this?

That tiny tiny user message suddenly sits on top of token lasagna.

Delicious, expensive token lasagna.

This is one of the first surprises for developers building AI features: the user prompt is often not the main cost driver. The context around the prompt is.

Especially in RAG systems (more about RAG will come later, I do declare)

Because RAG usually means:

1. Take the user question.
2. Search a document index.
3. Retrieve relevant chunks.
4. Insert those chunks into the prompt.
5. Ask the model to answer using that context.

Step 4 is where your cost meter starts streaching.

A user asks one sentence.
Your app sends 8,000 tokens of documentation.
The model replies with 700 tokens.
You proudly call it “grounded AI”.

Finance call it “interesting”.

Output tokens: the model’s invoice printer

Output tokens are the tokens generated by the model.

These are usually more expensive than input tokens.

Why, you may ask? That’s a very good question – 5 points to Griffindor.

Generating your answer is a sequential work. The model does not generate the full answer in one magical operation. It predicts the next token, then the next one, then the next one, while considering the previous context and generated output.

Again, very simplified

Every generated token costs compute.

And with many providers, output tokens are priced higher than input tokens. Sometimes much higher.

This is why “be concise” is not just about user experience.

It is cost control.

A model that produces 200 tokens instead of 2,000 tokens can be dramatically cheaper, especially at scale.

Let’s say your AI feature handles 100,000 requests per month.

If each response is 200 tokens, that is:

100,000 × 200 = 20,000,000 output tokens

Same product with same number of users, but ten times more output tokens.

Congratulations, you invented a very polite money shredder.

The basic cost formula

Most text model pricing can be simplified like this:

total cost = 
     input_tokens × input_price_per_token 
   + output_tokens × output_price_per_token

Because providers usually publish prices per 1 million tokens, the formula becomes:

input_cost = 
     input_tokens / 1,000,000 × input_price_per_1M_tokens 

output_cost = 
     output_tokens / 1,000,000 × output_price_per_1M_tokens 

total_cost = 
     input_cost + output_cost

for example:

Input tokens: 4,000 
Output tokens: 800 

Input price: $2.00 / 1M tokens 
Output price: $8.00 / 1M tokens

Calculation might look like this:

input_cost = 
     4,000 / 1,000,000 × 2.00 = $0.008 

output_cost = 
     800 / 1,000,000 × 8.00 = $0.0064 

total = $0.0144

Around one and a half cents.

That sounds cheap, doesn’t it?

And per request it is. BUT software doesn’t usually fail financially because of one request.

It fails because of ** MULTIPLICATION **.

$0.0144 × 1,000 requests = $14.40 
$0.0144 × 100,000 requests = $1,440 
$0.0144 × 1,000,000 requests = $14,400

Tiny tiny 1,5 cent becomes a very real, when users retries, background jobs, and agents get involved.

Like logs.

Nobody worries about one line. Dare I say one singular failed request in an ocean of 200s.

But then one day your storagew account looks at you like “We need to talk”. And you know you are in big big trouble.

The context window is not a free buffet

Modern models can support large context windows.

That means you can send a lot of text into a single request.

This is powerful.

It is also dangerous.

A large context window does not mean:

Please paste the entire company knowledge base into every request.

It means:

You can include more information when it is actually needed.

There is a significant difference.

Large context windows are useful for:

• analyzing long documents,
• reviewing large code files,
• handling complex conversations,
• comparing multiple sources,
• working through multi-step problems.

But in production systems, every extra token should justify its existence.

Because the model has to process it.

And you may have to pay for it.

This is where developers need to think like engineers again.

Not:

Can we fit this into the context window?

But:

Should this be in the context window?

Those are very different questions.

The same way “can we put everything in one SQL query?” and “should we?” are technically separate questions, even if Stack Overflow occasionally disagrees.

RAG cost: death by document chunks

Retrieval-Augmented Generation is one of the most common AI application patterns (we’ll look more into it later)

It is also one of the easiest places to accidentally overspend.

A typical RAG flow looks like this:

There are multiple cost areas here:

1. Embedding the user query.
2. Vector database storage and search.
3. Input tokens from retrieved chunks.
4. Output tokens from the answer.
5. Optional reranking.
6. Optional follow-up calls.
7. Optional evaluation or citation checking.

The expensive part is often not the vector search.

It is sending too much retrieved text into the model.

So for example:

User question: 20 tokens 
System prompt: 500 tokens 
Conversation history: 1,500 tokens 
Retrieved chunks: 10 chunks × 800 tokens = 8,000 
tokens Output: 700 tokens

Total input:

20 + 500 + 1,500 + 8,000 = 10,020 input tokens

Most of the request is retrieved context.

Now imagine that the top 10 chunks include repeated introductions, legal footers, markdown navigation, duplicated release notes, and two irrelevant pages because the embedding search was “close enough”.

You are paying the model to read noise.

And then you are surprised when it answers like someone read the noise. Very unfair to the model, honestly.

I, myself, am guilty of charged.

Agents: when the loop becomes the product

A chatbot is usually one request and one response.

An agent is different.

An agent may:

1. receive a task,
2. inspect files,
3. call tools,
4. read tool results,
5. make a plan,
6. call another tool,
7. revise the plan,
8. generate code,
9. run tests,
10. fix errors,
11. summarize the result.

That is not one model call.

That is a loop.

And each loop iteration may include:

• the original prompt,
• system instructions,
• conversation state,
• tool definitions,
• previous tool outputs,
• intermediate reasoning summaries,
• code snippets,
• errors,
• final answer instructions.

This is why agentic systems can become expensive quickly.

The cost is not just:

user asks question → model answers

It is:

user asks task 
   → model plans 
   → tool call 
   → model reads result 
   → tool call 
   → model reads result 
   → model writes file 
   → test fails 
   → model reads error 
   → model fixes file 
   → final answer

And if you are not careful, your agent is basically a junior developer with a corporate card.

Useful? Sometimes.

Should it be allowed to run forever? Absolutely not.

Caching: the closest thing to a discount coupon

Prompt caching is one of the most useful cost-saving features when working with repeated context.

The idea is simple:

If many requests start with the same large prefix, the provider may cache that prefix and charge less when it is reused.

Great candidates for caching:

• long system prompts,
• tool definitions,
• policy instructions,
• product documentation,
• static reference material,
• repeated examples,
• stable application context.

For example, instead of sending this every time as fresh input:

[Long system instructions] 
[Tool definitions] 
[Company policy] 
[Product documentation] 
[User question]

You can structure prompts so the stable part comes first:

[CACHEABLE: Long system instructions] 
[CACHEABLE: Tool definitions] 
[CACHEABLE: Company policy] 
[DYNAMIC: User question]

When cache hits work, repeated input becomes cheaper and often faster.

But caching is not magic either.

It works best when:

• the prefix is stable,
• requests are similar,
• the cache lifetime matches your traffic pattern,
• you place reusable content at the beginning,
• you do not accidentally change whitespace, ordering, timestamps, or generated IDs in the cached section.

That last point is very developer.

You add a timestamp to the top of every prompt for “traceability”, then wonder why caching stopped working.

Classic.

How to save money while developing with AI

Now to the question you probably started with when you opened this article. How to?

I’m not going to tell you “stop using AI”, as this is not a useful advice, AI is a stack in the new reality we all live in.

The useful question is:

How do we make AI features cost-aware from the beginning?

Let’s go through the important parts. I created a 12-step program to help you take over control over your costs (sounds like an intervation or a rehab.. maybe we can consider it as such.. wink wink).

One: Measure tokens before you optimize anything

Do not start with vibes.

Start with numbers.

For every AI feature, log at least:

model 
request type 
input tokens 
cached input tokens 
output tokens 
total tokens 
latency 
estimated cost 
user or tenant 
success/failure 
retry count

Without this, you are blind.

And “we think the prompt is small” is not observability.

It is folklore.

In .NET, I would usually wrap AI calls behind one service and make token/cost tracking part of that abstraction.

Something like:

public sealed record AiUsage(
     string Model, 
     int InputTokens, 
     int CachedInputTokens, 
     int OutputTokens, 
     decimal EstimatedCost, 
     TimeSpan Latency); 

public interface IAiClient 
{ 
     Task> GenerateAsync( 
          AiRequest request, 
          CancellationToken cancellationToken); 
} 

public sealed record AiResponse( 
     T Value, 
     AiUsage Usage);

Then every feature using AI gets usage tracking automatically.

Not as an afterthought.

Not as “we’ll add it before production”.

Before production is where good intentions go to open tickets and die.

Two: Put token budgets in code

Every AI request should have a budget.

Not just a financial budget, but a token budget.

Max input tokens: 8,000 
Max output tokens: 700 
Max retrieved chunks: 5 
Max chunk size: 600 tokens 
Max agent steps: 8 
Max retries: 2

Budgets force design decisions.

If your summarization feature needs 50,000 input tokens for every request, maybe it should not be a single call.

Maybe it should be:

1. chunk,
2. summarize chunks,
3. summarize summaries,
4. cache intermediate results.

Or maybe the feature is simply too expensive for the value it provides.

That is also an answer.

A boring answer, but production is often boring in self-defense.

Three: Control output length

Output tokens are commonly more expensive than input tokens.

So control them.

Use clear instructions:

Answer in maximum 8 bullet points. 
Keep the answer under 250 words. 
Return only JSON. 
Do not include explanations.

Also use API parameters where available:

max_output_tokens
max_tokens
stop sequences
response format constraints

Do not rely only on politeness.

Models are helpful.
Helpful models like explaining.
Explaining costs money.

Four: Do not send the whole conversation forever

Chat history grows.

Token cost grows with it.

At some point, the beginning of the conversation is no longer useful. It is just archaeology.

Strategies:

• keep only the last N messages,
• summarize older conversation,
• store structured state instead of raw chat,
• remove irrelevant turns,
• separate user profile from conversation history,
• avoid resending tool outputs that are no longer needed.

Bad:

Send all previous 120 messages every time.

Better:

Send the last 10 messages + a compact state summary.

Even better:

Send only the facts needed for this request.

The model does not need to remember that the user said “thanks” 47 messages ago.

Unless your product is emotionally attached to token waste.

Five: Be careful with RAG chunking

Chunking strategy affects both quality and cost.

Too small chunks:

• lose context,
• increase retrieval count,
• may require more chunks.

Too large chunks:

• waste tokens,
• include irrelevant information,
• make answers less focused.

A reasonable starting point is often:

chunk size: 300–800 tokens
overlap: 50–150 tokens
top_k: 3–6 chunks

But this depends heavily on your content.

API documentation, legal contracts, code files, release notes, and support articles behave differently.

Please do not use one magical chunk size from a blog post and call it architecture.

Even if this is a blog post.

Especially then.

Six: Retrieve less, but retrieve better

Instead of sending 12 chunks and hoping the model figures it out, improve retrieval.

Useful techniques:

• metadata filtering,
• hybrid search,
• reranking,
• document type filters,
• tenant filters,
• date filters,
• permissions filtering,
• query rewriting,
• deduplication,
• removing boilerplate.

Bad RAG:

Search everything.
Take top 10.
Pray.

Better RAG:

Filter by tenant, product, version, permissions.
Retrieve candidates.
Rerank.
Deduplicate.
Send only the best chunks.

The model should receive the sharpest context possible.

Not a haystack with a motivational message.

Seven: Use cheaper models where possible

Not every task needs the strongest model.

Some tasks are simple:

• classification,
• routing,
• extracting fields,
• rewriting short text,
• detecting intent,
• formatting JSON,
• summarizing small snippets.

These may work well on smaller, cheaper models.

Save expensive models for tasks that actually need them:

• complex reasoning,
• difficult coding,
• ambiguous requirements,
• multi-document synthesis,
• high-risk decisions,
• advanced agent workflows.

This is model routing.

Example:

User request
    ↓
Small model: classify task
    ↓
If simple → small model
If complex → stronger model
If sensitive → stronger model + validation

Think of it like compute tiers.

You would not run every background job on the biggest VM in Azure.

Unless you enjoy explaining cloud bills in meetings.

Eight: Cache what does not change

There are several layers where caching can help:

• provider prompt caching,
• application response caching,
• embedding caching,
• retrieval result caching,
• document summary caching,
• tool result caching.

Example:

If users often ask:

What changed in release 4.2?

Do not regenerate the same answer from scratch every time.

Cache the release summary.

If a document was already summarized, store the summary.

If an embedding already exists for a document chunk, do not regenerate it.

If tool definitions are stable, structure prompts to benefit from provider caching.

Caching is not glamorous.

Neither is paying twice for the same work.

Nine: Avoid retry storms

Retries are good, but retry storms are expensive.

An AI request can fail because of:

• rate limits,
• timeouts,
• invalid JSON,
• tool errors,
• safety filters,
• context length errors,
• provider instability.

If your system blindly retries full prompts, you may multiply cost without improving success.

Use:

• exponential backoff,
• retry limits,
• idempotency keys,
• smaller repair prompts,
• validation before calling the model,
• fallback models,
• circuit breakers.

For invalid JSON, do not always rerun the whole original prompt.

Sometimes you can send only:

Fix this JSON to match the schema.

with the broken output.

Much cheaper than regenerating everything.

Ten: Put hard limits on agents

Agents need boundaries.

Real ones.

Not “please don’t do too much”.

Use limits like:

max_steps = 8
max_tool_calls = 12
max_total_tokens = 100_000
max_runtime_seconds = 60
max_cost_per_task = 0.50

And when the limit is reached, stop gracefully:

I could not complete the task within the configured budget.
Here is what I completed.
Here is what remains.

That is not a failure.

That is engineering.

An unbounded agent is just a while loop with branding.

Eleven: Design prompts like APIs

Prompts are part of your system.

Treat them like code.

That means:

• version them,
• review them,
• test them,
• measure them,
• keep them small,
• remove dead instructions,
• avoid contradictory rules,
• document expected inputs and outputs.

Bad prompt growth looks like this:

You are helpful.
Also concise.
Also detailed.
Also explain everything.
Also never explain too much.
Also use JSON.
Also be friendly.
Also do not include markdown.
Also include markdown tables.

At some point, your prompt becomes a haunted configuration file.

Refactor it.

Twelve: Think in cost per user action

Per-token pricing is too low-level for product decisions.

Translate it into business terms.

For example:

Cost per support answer
Cost per document summary
Cost per code review
Cost per generated test suite
Cost per agent task
Cost per active user per month
Cost per tenant per month

This helps answer useful questions:

• Is this feature profitable?
• Should it be rate limited?
• Should it be included in all plans?
• Should heavy usage require a higher tier?
• Which customers are driving cost?
• Which prompts are inefficient?
• Which workflows need caching?

Nobody outside engineering wants to hear:

We used 973 million output tokens.

They want to hear:

The AI support assistant costs around $0.018 per resolved ticket.

Now we can have a real conversation.

The sneaky costs outside tokens

Tokens are only part of the story.

Depending on your architecture, AI cost may also include:

• vector database,
• embedding generation,
• document ingestion,
• storage,
• OCR,
• speech-to-text,
• text-to-speech,
• image processing,
• search grounding,
• monitoring,
• evaluation,
• logging,
• data retention,
• human review,
• security controls,
• latency overhead,
• engineering time.

That last one matters.

A cheaper model that takes three months of prompt gymnastics to behave may not be cheaper.

Developer time is also a resource.

Sadly, not usually billed per token.

Although some meetings suggest it should be.

A small practical checklist

Before shipping an AI feature, ask:

Do we log input and output tokens?
Do we estimate cost per request?
Do we know the cost per user action?
Do we limit output length?
Do we limit conversation history?
Do we limit retrieved context?
Do we cache stable prompts or documents?
Do we route simple tasks to cheaper models?
Do we have retry limits?
Do agents have step and cost budgets?
Do we know which tenants/users/features drive cost?
Do we have alerts for cost spikes?

If the answer to all of this is “no”, then you do not have an AI feature.

You have an invoice generator with a chat interface.

So what actually makes AI expensive?

Not tokens alone.

Tokens are just the unit.

AI becomes expensive when we combine:

• large prompts,
• long histories,
• verbose outputs,
• too much retrieved context,
• unnecessary powerful models,
• unbounded agents,
• repeated work,
• missing caching,
• retries,
• and no observability.

The model is not usually the only problem.

The product design is.

The architecture is.

The missing limits are.

The “we’ll optimize later” is.

AI cost is not something to think about after launch. It is part of the system design.

Like security.
Like logging.
Like permissions.
Like deployment strategy.
Like that one environment variable nobody documented and now production depends on.

The engineering version

From a developer’s point of view, AI cost is not mysterious.

It is mostly this:

How much text do we send?
How much text do we generate?
How often do we do it?
How expensive is the model?
How much work do we repeat?
How many loops do we allow?

That is it.

The hard part is not understanding the formula.

The hard part is building the discipline around it.

Because AI makes it very easy to hide complexity behind one innocent-looking API call.

But behind that call there may be a whole system:

• tokenizer,
• model inference,
• context processing,
• output generation,
• tool orchestration,
• caching,
• retrieval,
• validation,
• retries,
• billing.

So next time someone says:

It is just one AI call.

Smile politely.

Then ask:

How many tokens?

Sources and nice-to-knows

Gemini API – billing.

GEmini API – Understand and count tokens.

OpenAI – What are tokens and how to count them?

Antropic – prompt caching.

Microsoft Learn – understand tokens.