Configure AI Agent Health Score Alerts in Agent Therapy

AI agents degrade quietly. A prompt change, a model update, or a subtle shift in user behavior can cause your agent’s health to erode over days before anyone notices — by which point real users have already been impacted. Alerts let you catch those regressions the moment they happen, so you can investigate and intervene before the problem compounds. This guide covers every alert type Agent Therapy supports and walks you through configuring notifications across all available channels.

Alert Types

Health Score Threshold

Fires when your agent’s rolling health score drops below a value you set. The rolling score is the weighted average of the agent’s last 10 sessions, so this alert catches gradual degradation trends rather than one-off anomalies.

Pattern Detected

Fires when a specific behavioral pattern is found in any session — for example, any detection of hallucination_burst or goal_hijacking. Use this to zero in on the failure modes that matter most for your use case.

Session Failed

Fires when a diagnostic session cannot be completed due to an error. This usually indicates a problem with your log pipeline — malformed input, an invalid agent ID, or an upstream integration issue — so it signals infrastructure problems as well as agent problems.

Critical Finding

Fires immediately when any finding with severity: critical is detected in a completed session. This alert bypasses the rolling average so you hear about it right away, regardless of the agent’s overall score trend.

Configuring Alerts

Open the Alerts settings

In the Agent Therapy dashboard, go to Settings → Alerts and click New Alert.

Choose the alert type

Select one of the four alert types described above. The form adjusts to show the relevant condition fields for the type you choose.

Set the conditions

Configure the trigger conditions for your alert:

Health Score Threshold — enter the minimum acceptable score (e.g., 65). The alert fires when the rolling score falls at or below this number.
Pattern Detected — select one or more patterns from the dropdown (e.g., hallucination_burst, instruction_drift). The alert fires if any of the selected patterns appear in a session.
Session Failed — no additional conditions needed; the alert fires on any session failure for the selected agents.
Critical Finding — no additional conditions needed; the alert fires on any severity: critical finding.

You can scope any alert to all agents or to a specific agent or tag group.

Choose a notification channel

Select how you want to be notified: Email, Slack, or Webhook. You can add multiple channels to the same alert. See Notification Channels for setup details for each option.

Save the alert

Click Save Alert. The alert activates immediately and will begin firing on the next qualifying event.

Notification Channels

Email Enter one or more email addresses. Agent Therapy sends a formatted alert email that includes the agent name, alert type, current health score (where applicable), a link to the triggering session, and a summary of findings. Slack Provide an Incoming Webhook URL from your Slack workspace. Agent Therapy posts a structured message to the target channel containing the alert type, agent name, health score, and a direct link to the session report. To create a Slack incoming webhook, visit api.slack.com/messaging/webhooks and follow the setup steps for your workspace. HTTP Webhook Provide any HTTPS endpoint. Agent Therapy sends a POST request with a JSON payload whenever the alert fires. The payload includes the full alert event object so you can route it to PagerDuty, an internal Slack bot, a ticketing system, or any other destination. See Alert via API below for the event payload schema.

Alert via API

You can register webhook endpoints programmatically using the REST API. This is useful when managing alert configuration as code or when spinning up per-environment monitoring in a deployment pipeline. Endpoint: POST https://api.agenttherapy.ai/v1/webhooks

curl --request POST \
  --url https://api.agenttherapy.ai/v1/webhooks \
  --header "Authorization: Bearer YOUR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "url": "https://your-service.example.com/hooks/agent-therapy",
    "events": [
      "session.completed.critical_finding",
      "session.failed",
      "agent.health_score.threshold_breached"
    ],
    "secret": "whsec_your_hmac_signing_secret"
  }'

import httpx

response = httpx.post(
    "https://api.agenttherapy.ai/v1/webhooks",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://your-service.example.com/hooks/agent-therapy",
        "events": [
            "session.completed.critical_finding",
            "session.failed",
            "agent.health_score.threshold_breached",
        ],
        "secret": "whsec_your_hmac_signing_secret",
    },
)

webhook = response.json()
print(webhook["id"])  # whk_xxxxxxxxxx

Request body fields:

Field	Type	Required	Description
`url`	string	Yes	The HTTPS endpoint that will receive `POST` requests when an event fires.
`events`	array of strings	Yes	The event types to subscribe to. See the table below for all valid values.
`secret`	string	Yes	An HMAC signing secret. Agent Therapy signs every request with this secret using SHA-256 and includes the signature in the `X-AgentTherapy-Signature` header so you can verify authenticity.

Supported event types:

Event	Fires when
`session.completed.critical_finding`	A completed session contains one or more `critical` findings.
`session.completed.pattern_detected`	A completed session contains a pattern you subscribed to.
`session.failed`	A session fails before completing.
`agent.health_score.threshold_breached`	An agent’s rolling health score drops at or below your configured threshold.

Listing Registered Webhooks

To retrieve all webhook endpoints registered for your account, send a GET request to /v1/webhooks. Endpoint: GET https://api.agenttherapy.ai/v1/webhooks

curl --request GET \
  --url https://api.agenttherapy.ai/v1/webhooks \
  --header "Authorization: Bearer YOUR_API_KEY"

import httpx

response = httpx.get(
    "https://api.agenttherapy.ai/v1/webhooks",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)

webhooks = response.json()
for wh in webhooks["data"]:
    print(wh["id"], wh["url"], wh["events"])

Example response:

{
  "data": [
    {
      "id": "whk_3fTqNpVrCx",
      "url": "https://your-service.example.com/hooks/agent-therapy",
      "events": [
        "session.completed.critical_finding",
        "session.failed",
        "agent.health_score.threshold_breached"
      ],
      "created_at": "2024-01-15T09:30:00Z"
    }
  ]
}

The response contains a data array of all registered webhook objects. Each object includes the id (needed to update or delete the webhook), the target url, the subscribed events, and the created_at timestamp.

Silencing & Pausing Alerts

Sometimes you need to take an agent offline for maintenance, deploy a major prompt update, or run load tests — situations where a flood of alerts would be noise rather than signal. Agent Therapy lets you put any agent into Maintenance Mode to suppress all alerts for that agent temporarily. To enable Maintenance Mode:

Go to Agents and open the agent’s detail page.
Toggle the Maintenance Mode switch in the top-right corner of the page.
Optionally enter an end time — when the timer expires, Maintenance Mode disables automatically and alerting resumes.

While an agent is in Maintenance Mode, sessions continue to run and reports continue to be generated — only the notifications are suppressed. The alert history log still records what would have fired, so you can review it after maintenance is complete. You can also pause a specific alert rule without disabling all alerts for an agent. Go to Settings → Alerts, find the rule, and click the Pause toggle next to it.

Start with a health score threshold of 65 for production agents. This gives your agents enough headroom to absorb the natural session-to-session variation that comes from diverse user inputs, while still catching meaningful degradation early. Once you have a few weeks of baseline data, you can tighten the threshold based on your agent’s typical score range.

Get Started

Core Concepts

Guides

Integrations

Support

Configure AI Agent Health Score Alerts in Agent Therapy

Alert Types

Health Score Threshold

Pattern Detected

Session Failed

Critical Finding

Configuring Alerts

Notification Channels

Alert via API

Listing Registered Webhooks

Silencing & Pausing Alerts

​Alert Types

Health Score Threshold

Pattern Detected

Session Failed

Critical Finding

​Configuring Alerts

​Notification Channels

​Alert via API

​Listing Registered Webhooks

​Silencing & Pausing Alerts

Alert Types

Configuring Alerts

Notification Channels

Alert via API

Listing Registered Webhooks

Silencing & Pausing Alerts