How to stop runaway AI agent costs
Runaway-cost incidents are the most common way teams discover their agent infrastructure is broken. Here is how to prevent them — and how to limit the blast radius when one slips through.
Anatomy of a runaway-cost incident
A runaway-cost incident is what happens when an agent enters a state — usually a loop, sometimes a context-window thrash, occasionally a recursive sub-agent storm — that causes it to burn tokens faster than any reasonable workflow should.
The pattern is depressingly consistent: an agent retries a failed tool call, the failure mode causes it to retry with more context, the larger context makes the next call more expensive and more likely to fail, and the spiral compounds until either the model provider's rate limiter or your finance team's pager intervenes. By that point you have spent five thousand dollars in forty minutes.
The three layers of cost defense
You want defense in depth. Any single layer can fail or be misconfigured; together they make a runaway nearly impossible.
- Per-agent hourly token budget — a soft ceiling that catches loops within a single agent in the window where they actually do damage.
- Per-agent daily cost ceiling in dollars — a hard ceiling expressed in the unit your CFO actually cares about.
- Per-workspace monthly event quota — the backstop that protects against a misconfigured guardrail letting through what a single agent's limits should have caught.
Why per-hour beats per-month
Many teams set only a monthly limit and call it a day. This is the equivalent of a fire alarm that goes off at the end of the month. By the time your monthly quota is breached, the agent has already burned through your monthly budget in a single afternoon, and the only question left is how loudly your CFO yells.
Hourly token budgets catch runaway loops in the window where the damage is recoverable. A well-behaved agent rarely burns more than a few thousand tokens per hour. An agent stuck in a tool-call loop will hit a million in under sixty minutes. The signal-to-noise ratio is exceptional.
Stop versus warn
agentwach guardrails let you choose between 'stop' (the agent's next ingest call returns stop:true and your SDK halts it) and 'warn' (a token_warnings row is written, but the agent keeps running). Default to stop. Warn-only is appropriate for two cases: a critical production agent where any false-positive halt is unacceptable, and the first week of a new guardrail where you are still calibrating thresholds.
What to do when one slips through
Sooner or later a runaway will escape your guardrails — a new agent gets deployed without limits configured, a model price change blows past a stale dollar cap, an edge case nobody anticipated. When it happens, your post-mortem should answer three questions: what was the loop signature, what guardrail should have caught it, and what made the existing guardrail miss?
The point of observability is not preventing every incident. The point is making sure each incident teaches you something durable.