Agent Health Score: Measuring AI Behavioral Stability

Your agent’s Health Score is a single number — ranging from 0 to 100 — that represents its overall behavioral wellness at a glance. Rather than sifting through raw logs or decoding opaque error traces, you get one authoritative signal: is your agent performing the way it should? The Health Score distills every observable dimension of agent behavior into a unified, trackable metric so you can spot problems early, respond confidently, and demonstrate reliability over time.

Score Ranges

Each score range maps to a named status and a recommended level of attention.

Score	Status	Meaning
70 – 100	🟢 Healthy	Agent is behaving as expected. Continue normal monitoring.
60 – 69	🟡 Monitoring Needed	Minor anomalies detected. Review recent sessions and watch for trends.
30 – 59	🟠 Issues Detected	Meaningful behavioral problems are present. Investigate and consider an intervention.
0 – 29	🔴 Critical	Severe dysfunction. Immediate intervention is required before the agent causes downstream harm.

Score thresholds are used to trigger automated alerts. See Setting Score Thresholds below to configure them for your team.

What Goes Into the Score

The Health Score is built from four core dimensions, each measuring a distinct aspect of agent behavior.

Instruction Adherence

Measures how faithfully your agent follows the directives in its system prompt. An agent that progressively ignores constraints, adopts new personas, or fulfills requests it was explicitly told to refuse will score poorly here.

Output Consistency

Tracks the stability and predictability of the agent’s responses across similar inputs. High variance — where the agent gives contradictory answers, wildly different formats, or inconsistent tone — signals a consistency problem even when individual outputs seem acceptable.

Tool Use Accuracy

Evaluates whether the agent calls the right tools, supplies correct arguments, and handles tool responses appropriately. Agents that misuse tools, hallucinate function names, or enter failure cascades after bad tool responses incur penalties in this dimension.

Context Coherence

Assesses how well the agent retains and correctly applies information from earlier in the conversation. An agent that forgets user-provided facts, contradicts its own prior statements, or loses track of the original goal over a long session will score lower here.

How the Score Is Calculated

Agent Therapy computes the Health Score at two granularities:

Per-session score: Calculated at the end of each therapy session using the behavior logs submitted for that session. Each of the four dimensions is scored independently, then combined into a single 0–100 composite using a weighted formula. Dimension weights reflect their relative impact on end-user experience and are tuned continuously based on aggregate platform data.
Rolling 7-day average: In your dashboard and API responses, you’ll also see a 7-day rolling average that smooths out one-off anomalies and gives you a more reliable picture of your agent’s health trend over time. This is the number most useful for reporting and SLA tracking.

Watch the rolling average more than any single session score. A single low-scoring session may reflect a difficult conversation rather than a systemic problem — a downward trend over several days is the real warning sign.

Viewing Your Agent’s Score

In the dashboard: Navigate to Agents → select your agent → the Health Score is displayed prominently at the top of the agent detail page. You’ll see both the most recent session score and the 7-day rolling average, along with a sparkline chart of score history. Via the API: Retrieve the current score and recent history by calling:

GET /agents/{id}

The response includes a health object:

{
  "id": "agt_01hx9z3kqp8fvb2mjd7c",
  "name": "Support Bot",
  "health": {
    "current_score": 74,
    "rolling_7d_average": 81,
    "status": "monitoring_needed",
    "last_evaluated_at": "2024-11-15T14:32:00Z",
    "dimensions": {
      "instruction_adherence": 78,
      "output_consistency": 71,
      "tool_use_accuracy": 83,
      "context_coherence": 65
    }
  }
}

This lets you integrate Health Score data directly into your own monitoring dashboards, incident management tools, or Slack notifications.

Setting Score Thresholds

You can configure Agent Therapy to alert you automatically whenever your agent’s Health Score drops below a value you define. This is the fastest way to catch regressions before they affect users at scale. Thresholds are set per-agent and support multiple severity levels — for example, a warning at 70 and a critical alert at 50. Alerts can be delivered via email, Slack, PagerDuty, or webhook.

For production agents, we recommend setting a warning threshold at 70 and a critical threshold at 50 as a starting point, then tuning based on your agent’s historical score range.

To configure score threshold alerts, see the Setting Up Alerts guide.

Get Started

Core Concepts

Guides

Integrations

Support

Agent Health Score: Measuring AI Behavioral Stability

Score Ranges

What Goes Into the Score

Instruction Adherence

Output Consistency

Tool Use Accuracy

Context Coherence

How the Score Is Calculated

Viewing Your Agent’s Score

Setting Score Thresholds

​Score Ranges

​What Goes Into the Score

​Instruction Adherence

​Output Consistency

​Tool Use Accuracy

​Context Coherence

​How the Score Is Calculated

​Viewing Your Agent’s Score

​Setting Score Thresholds

Score Ranges

What Goes Into the Score

Instruction Adherence

Output Consistency

Tool Use Accuracy

Context Coherence

How the Score Is Calculated

Viewing Your Agent’s Score

Setting Score Thresholds