Skip to content
Back to blog
AI Agents

Runaway AI Agents: Evals, Spend Caps, and Kill Switches to Set Before You Scale

HTHaseeb Tariq · Co-FounderNov 2, 202613 min read
AI agent control panel with spending meters, kill switch, and network connections representing cost governance and failure controls
AI Agents

On November 1, 2026, Salesforce expanded Agent Fabric with a suite of controls aimed at a problem every team deploying agents at scale has encountered: agents that fail silently, loop indefinitely, and rack up token costs before anyone notices. The release includes Agent Evals, API Evals, a now-GA Agent Kill Switch, and a FinOps Dashboard with Model Wallet hard spend limits. Salesforce reports that Agent Fabric handles over 1.2 billion requests daily, and their cost optimization policies cut token usage by up to 75% in test scenarios.

Four days earlier, on October 28, ServiceNow launched the Control Tower Alliance, a coalition including Anthropic, AWS, Cisco, HiddenLayer, Kong, NVIDIA, Okta, OpenAI, Stripe, WitnessAI, and Zenity. The alliance focuses on centralized governance, observation, and token metering for enterprise AI. When the largest AI providers and enterprise platforms form a coalition around agent governance and cost control, it signals that the problem is no longer edge cases. It is the central operational challenge of agent deployment.

This guide covers the controls you need before scaling AI agents: pre-deploy and production evals, per-run step/token/time ceilings, per-period budgets enforced in the request path (not just dashboards you check after the bill arrives), tiered thresholds with cheaper-model fallback, approval gates by cost and reversibility, kill switches that work without a deploy, and cost attribution per agent and team. These are the controls that separate teams burning money on runaway agents from teams deploying with confidence.

The failure modes: why agents run away

Traditional automation fails loudly. A script throws an exception, a workflow step errors out, and someone gets paged. Agents fail quietly. An agent pursuing an objective can get stuck in a loop: retrying a failed API call, re-processing the same data, or cycling through tool calls that do not advance the goal. Each iteration burns tokens. By the time someone notices the bill, the damage is done.

nOps identifies agent loops as a primary driver of runaway spend. An agent without step limits can call the same tool hundreds of times trying to "fix" a problem that cannot be fixed. An agent without time limits can run for hours on a task that should take minutes. An agent without token limits can generate increasingly elaborate responses to increasingly off-track reasoning. Without hard limits enforced outside the agent's control, the agent's persistence becomes a liability.

The second failure mode is silent degradation. An agent that worked correctly last month may fail differently when underlying data changes, model versions shift, or integrated systems update their APIs. Without production evals that continuously verify agent behavior, you will not know the agent is producing wrong answers until a customer complains or an audit reveals the damage.

Pre-deploy evals: catch failures before production

Salesforce's Agent Evals and API Evals, launching in beta November 2026 with GA in January 2027, address the first line of defense: testing agents before they reach production. Agent Evals run the agent against defined test cases and evaluate whether outputs meet quality criteria. API Evals test specific tool calls for correct behavior. Both run before deployment, blocking releases that do not meet thresholds.

The pattern is not unique to Salesforce. Any agent deployment pipeline should include evaluation gates. The question is what to evaluate and how strictly to enforce it.

What to include in pre-deploy evals

  • Task completion: Does the agent accomplish the objective for a representative set of inputs?
  • Accuracy: For tasks with verifiable answers, does the agent produce correct results?
  • Tool usage: Does the agent call the right tools in the right sequence? Does it avoid forbidden tools?
  • Boundary adherence: Does the agent stay within defined scope, or does it attempt unauthorized actions?
  • Cost per task: How many tokens does the agent consume on typical tasks? On edge cases?
  • Failure recovery: When a tool call fails, does the agent recover gracefully or loop indefinitely?

Evaluation thresholds should be explicit. A pass rate of 95% on task completion, maximum 10,000 tokens per task, zero attempts to call forbidden tools. If the build does not meet thresholds, the deploy does not proceed. This is CI/CD for agents: automated quality gates that prevent regressions from reaching production.

Production evals: continuous verification

Pre-deploy evals catch known failure modes. Production evals catch the failures that emerge when agents encounter real-world data. Salesforce's approach includes sampling live agent runs and evaluating them against the same criteria used in pre-deploy testing. The FinOps Dashboard surfaces metrics on task success, tool usage, and cost, providing visibility into how agents behave at scale.

Production evals serve multiple purposes. They detect drift: an agent that passed pre-deploy tests but fails on production data. They detect adversarial inputs: prompts or data designed to manipulate the agent. They detect infrastructure issues: tool endpoints that have changed, latency that causes timeouts, rate limits that trigger retries.

Implementing continuous evaluation

  • Sample agent runs for async evaluation. Not every run needs evaluation, but a statistically significant sample does.
  • Evaluate against success criteria: did the agent complete the task? Did it produce correct output? Did it stay within bounds?
  • Track metrics over time: task success rate, average token usage, tool call patterns, failure types.
  • Alert on anomalies: a sudden drop in success rate, a spike in token usage, an increase in specific error types.
  • Compare cohorts: do agents behave differently for different user segments, input types, or time periods?

The agent governance framework we recommend includes observability as a core pillar. Production evals build on that observability: you cannot evaluate what you do not log.

Per-run ceilings: step, token, and time limits

Evals detect problems. Per-run limits prevent problems from becoming expensive. Every agent run should have hard ceilings on the resources it can consume.

Per-run limits to enforce on agent executions.
Limit TypeWhat It ControlsWhy It Matters
Step limitMaximum number of reasoning steps or tool calls per runPrevents infinite loops where agent keeps retrying
Token limitMaximum input + output tokens per runCaps cost of any single execution
Time limitMaximum wall-clock duration per runPrevents stuck agents from running indefinitely
Tool call limitMaximum calls to specific expensive toolsLimits exposure to paid APIs or heavy compute

These limits must be enforced outside the agent's control. A limit implemented as a prompt instruction can be reasoned around. A limit implemented in the orchestration layer cannot. When the agent hits the ceiling, the run terminates regardless of whether the agent "thinks" it is done.

Cloudflare AI Gateway demonstrates this pattern for token spend: configure a maximum budget per gateway, and requests that would exceed the budget receive a 429 response. The limit is enforced at the infrastructure layer before the model call executes. Cloudflare notes that enforcement is eventually consistent, so brief bursts may exceed the limit before it takes effect, but the ceiling holds over any meaningful time window.

Per-period budgets: daily, weekly, monthly caps

Per-run limits cap individual executions. Per-period budgets cap aggregate spend over time. Salesforce's Model Wallet enforces hard spend limits at the daily, weekly, or monthly level. When the budget is exhausted, agents stop making model calls until the period resets or the budget is increased.

The critical detail is where the budget is enforced. A dashboard that shows you have exceeded your budget is not a control. A dashboard you check on Monday that shows you overspent last week is too late. The budget must be enforced in the request path: every model call checks the remaining budget before executing, and calls that would exceed the budget are blocked.

Budget enforcement architecture

  • Centralized budget tracker: a service that maintains current spend per agent, team, or application.
  • Pre-request check: before any model call, query the budget tracker for remaining allocation.
  • Cost estimation: estimate the cost of the pending request (based on model, expected tokens, or historical averages).
  • Enforcement decision: if estimated cost would exceed budget, reject the request with a clear error.
  • Post-request update: after the call completes, update the budget tracker with actual spend.
  • Alerting: notify stakeholders when spend reaches 50%, 75%, 90% of budget, not just at 100%.

AWS published a sample implementation for per-user budget enforcement before model calls using AgentCore Gateway interceptors. The pattern applies regardless of cloud provider: intercept the request, check the budget, proceed or reject.

Tiered thresholds and model fallback

Hard limits stop agents when budgets are exhausted. Tiered thresholds degrade gracefully before hitting hard limits. Salesforce's cost optimization policies include model routing that directs requests to cheaper models when spending approaches thresholds. nOps describes a similar pattern: define tiers where the first threshold triggers a warning, the second triggers fallback to a cheaper model, and the third triggers a full stop.

Example tiered threshold configuration.
Threshold% of BudgetAction
Warning50%Alert ops team, continue with current model
Fallback75%Route new requests to cheaper model tier
Degraded90%Limit agent to read-only operations
Stop100%Block all agent operations until reset

Fallback models need validation. If your agent was tested with GPT-4 and falls back to GPT-3.5 under budget pressure, the cheaper model may not complete tasks correctly. Pre-deploy evals should include runs with fallback models to verify acceptable performance at each tier. A fallback that produces wrong answers is worse than a hard stop.

Approval gates by cost and reversibility

Not all agent actions carry equal risk. A read-only query costs tokens but cannot cause business harm. A write operation that modifies customer data has downstream consequences. A high-cost operation that triggers an expensive third-party API call might be fine at low volume but dangerous if it loops.

Approval gates based on cost and reversibility add a human checkpoint where risk is highest. The payment controls we documented follow this pattern: low-value transactions execute autonomously, high-value transactions require human approval. The same principle applies to cost-based gates.

When to require approval

  • Operations exceeding a cost threshold (e.g., any single call estimated to cost more than $1).
  • Aggregate session spend exceeding a threshold (e.g., session has already consumed $50, next operation requires review).
  • Irreversible operations regardless of cost (deletes, sends, commits).
  • Operations touching high-value entities (key accounts, financial records, access controls).
  • Operations that have failed and are being retried (the third retry of a failed API call should trigger review, not continue automatically).

The approval interface should show the estimated cost, what the agent wants to do, and why. An approver who sees "Agent wants to make an expensive API call" cannot make an informed decision. An approver who sees "Agent wants to call Enrichment API for 500 records, estimated cost $75, because task requires firmographic data" can.

The kill switch: stopping agents without a deploy

Salesforce's Agent Kill Switch, now generally available, provides exactly what the name implies: the ability to stop an agent immediately, without deploying new code or waiting for running tasks to complete. ServiceNow's Control Tower and NVIDIA OpenShell integration achieves similar immediate revocation, with sub-second propagation to hardware-level enforcement.

A kill switch is not the same as a feature flag. A feature flag typically takes effect on new requests; in-flight operations continue. A kill switch must stop running operations. nOps emphasizes that a kill switch which only prevents new runs while existing runs continue to loop is not adequate protection against runaway spend.

Kill switch requirements

  • Immediate effect: revocation takes effect within seconds, not minutes or hours.
  • Stops in-flight operations: running agent tasks are interrupted, not just new tasks blocked.
  • No deploy required: the switch can be flipped from a dashboard, API call, or automated trigger, not a code change.
  • Accessible to multiple roles: not dependent on a single admin being available.
  • Audited: every activation and deactivation is logged with timestamp, actor, and reason.
  • Tested: the team has actually tested the kill switch under realistic conditions, not just documented it.

Testing the kill switch is critical. A feature you have never exercised is a feature that might not work when you need it. Run a kill switch drill: start an agent on a long-running task, hit the switch, verify the agent stops, verify logs capture the event. Do this before an incident forces you to find out whether it works.

Cost attribution: per-agent and per-team visibility

Controls work better when accountability is clear. If all agent spend aggregates into one line item on the cloud bill, no team owns the cost. If spend is attributed to specific agents, specific teams, and specific use cases, cost becomes a first-class metric that teams can track and optimize.

Salesforce's FinOps Dashboard provides this visibility: spend broken down by agent, by application, and by cost category. The data enables both reactive analysis ("which agent drove last week's spike?") and proactive budgeting ("this team's agents are allocated $10,000/month").

Attribution dimensions

  • Agent identity: which specific agent incurred the cost.
  • Team or owner: which team is responsible for the agent.
  • Use case or application: what business purpose the agent serves.
  • Model: which underlying model was called (costs vary by model).
  • Operation type: reasoning, tool calls, embeddings, etc.
  • Time period: when the spend occurred (for trend analysis and anomaly detection).

Attribution enables chargebacks, where teams pay for what they use. Chargebacks create incentives: a team that sees their agent spend compared to others will optimize. Without attribution, cost optimization is someone else's problem.

Putting it together: a control checklist

The controls described in this guide form layers. Evals catch problems early. Per-run limits contain individual failures. Per-period budgets cap aggregate exposure. Tiered thresholds degrade gracefully. Approval gates add human oversight where risk is highest. Kill switches provide emergency stop. Attribution enables accountability.

Control checklist before scaling AI agents.
ControlLayerEnforcement Point
Pre-deploy evalsPreventionCI/CD pipeline, deployment gate
Production evalsDetectionSampling, async evaluation
Step/token/time limitsContainmentOrchestration layer, per-run
Per-period budgetsContainmentRequest path, budget service
Tiered model fallbackDegradationRouting layer, spend thresholds
Cost-based approval gatesOversightPre-execution checkpoint
Kill switchEmergency stopCentral control plane, immediate
Cost attributionAccountabilityLogging, billing integration

No single control is sufficient. An agent with pre-deploy evals but no per-run limits can still loop in production. An agent with per-run limits but no budget enforcement can run many expensive runs. An agent with budgets but no kill switch cannot be stopped quickly when something goes wrong. The controls compose: each layer catches what the previous layer missed.

Internal links: related governance controls

This guide focuses on cost and failure controls. Related governance areas include permission boundaries and action whitelists, which control what agents can do; MCP gateways, which enforce tool-level policies; and approval workflows for high-stakes actions, which add human oversight for consequential operations. The distrust tax describes the ongoing cost of running agents without adequate controls. The objective-driven agent readiness guide covers preparing your systems for agents that pursue goals autonomously.

How Vyntrix Labs helps

When we deploy AI agents for clients, cost and failure controls are part of the baseline architecture. Every agent ships with per-run limits, budget enforcement, and observability that surfaces cost by agent, team, and use case. We implement kill switches that work immediately, not kill switches that exist on paper. We build eval pipelines that gate deployments and continuously monitor production.

For organizations already running agents, we conduct AI automation audits that assess your current cost exposure, identify which agents lack limits, and design the control layer to close the gaps. See AI agents for custom development with built-in governance and AI automation for workflow automation with spend controls from day one.

Conclusion

Agents fail silently. They loop. They burn through token budgets before anyone notices. The controls that prevent runaway costs are not exotic: pre-deploy evals that catch failures before production, production evals that detect drift, per-run ceilings that contain individual failures, per-period budgets enforced in the request path, tiered thresholds that degrade gracefully, approval gates for high-cost operations, kill switches that stop agents immediately, and attribution that makes cost visible and accountable.

Salesforce reports that their cost optimization policies cut token usage by up to 75% in test scenarios. The ServiceNow Control Tower Alliance includes the major AI providers and enterprise platforms because everyone recognizes the problem. The controls are not optional at scale. They are how you scale without runaway costs becoming the story.

Frequently asked questions

Why do AI agents cause runaway costs?

Agents fail silently instead of throwing errors. An agent pursuing an objective can get stuck in loops, retrying failed operations, re-processing data, or cycling through tool calls that do not advance the goal. Each iteration consumes tokens. Without hard limits enforced outside the agent's control, the agent's persistence burns budget until someone notices the bill.

What is the difference between per-run limits and per-period budgets?

Per-run limits cap individual agent executions: maximum steps, tokens, or time for a single run. Per-period budgets cap aggregate spend over a time window: maximum daily, weekly, or monthly cost across all runs. You need both. Per-run limits prevent any single execution from being expensive. Per-period budgets prevent many executions from adding up to excessive spend.

Why must budget enforcement be in the request path, not just a dashboard?

A dashboard that shows you have exceeded your budget after the fact is not a control. Budget enforcement must check remaining allocation before every model call and block requests that would exceed the budget. If enforcement is not in the request path, you will only discover overspend when the bill arrives, too late to prevent the damage.

What should a kill switch actually do?

A kill switch must stop running operations immediately, not just prevent new ones. It should take effect within seconds, require no code deploy, be accessible to multiple people in your organization, and be tested before an incident. A kill switch you have never exercised might not work when you need it.

How do tiered thresholds with model fallback work?

Define spending thresholds that trigger progressively stronger responses. At 50% of budget, alert the ops team. At 75%, route new requests to a cheaper model. At 90%, limit agents to read-only operations. At 100%, stop all operations. Fallback models must be validated in pre-deploy evals to ensure acceptable performance at each tier.

What should pre-deploy evals test?

Pre-deploy evals should verify task completion, output accuracy, correct tool usage, boundary adherence (no unauthorized actions), cost per task, and failure recovery (does the agent loop or recover gracefully?). Define explicit pass thresholds and block deployments that do not meet them. This is CI/CD for agents.

HT

Haseeb Tariq

Co-Founder at Vyntrix Labs

Let's build something intelligent

Ready to automate the busywork and scale with AI?

Book a free discovery call and we'll map your highest-impact automation opportunities, no obligation, no jargon.