Agentic Incident Response Platform

Response agents that work with your stack.

Warrn

Your teams

Wwarrn engg

All teams

You're on call

Ends in 6d 3h

VVighnesh KS

MEDIUM

StatusResolved

Teamwarrn engg

Source

CloudWatch (SNS)

Created

Jun 17 at 7:31 PM (18 days ago)

Resolved by

System

Deduplication key

ap-south-1:warrn-prod-tg-healthy-hosts-low

Affected Customers

HHubSpotNNotion

Add customer

Extra Properties

CloudWatch

CloudWatch View alarm in AWS Console

State

OKALARM

Metric

HealthyHostCount in AWS/ApplicationELB

Threshold

LessThanThreshold 2.0

Investigation Complete

6/24/2026, 2:58:06 PM

95%Request Re-run

The healthy-host-count alarm fired because a Terraform-managed rolling deployment inadvertently kept HealthyHostCount at 1 for 5 consecutive minutes (13:55–13:59 UTC), tripping the 4-minute sustained threshold designed to distinguish real outages from normal rolling-deploy churn. CloudTrail shows Terraform deregistered warrn-api-1 at 13:55:12, re-registered it at 13:58:00, then deregistered warrn-api-2 only 21 seconds later — before api-1 had passed its ALB health checks — so the count never recovered to 2 between the two operations. CloudWatch fired ALARM at 14:01:22 and auto-resolved to OK at 14:03:22. Both EC2 instances remained running with passing status checks throughout; no crash, no stop-instance event, and no 5xx errors were observed. The service was degraded to a single host for ~5 minutes but was never completely down.

ACTIONS0/4 done

Increase inter-step delay in Terraform rolling deploy

Soon

Insert a minimum wait of 90–120 seconds between RegisterTargets for instance N and DeregisterTargets for instance N+1, keeping HealthyHostCount at 2 throughout the deploy.

Add a health-check gate before proceeding to next deploy step

When Possible

Have the deploy script poll DescribeTargetHealth after each RegisterTargets and only proceed once HealthyHostCount returns to 2.

Review alarm evaluation_periods vs. worst-case deploy overlap

When Possible

Consider raising evaluation_periods to 6 to add headroom, or suppress the healthy-hosts alarm during active Terraform runs.

Resolved · Was open for 1mActivity

Join WarroomDeclare Incident

Warrnagen

Why did the healthy-hosts alarm fire during the deploy?

Ask about this alert...

Everything for Incident Response

From detection to resolution, Warrn provides the complete toolkit for modern DevOps teams.

On-call

Rotations, overrides, and escalations across your teams and services.

Incidents

Declare an incident — the warroom builds itself.

Heartbeats & Monitors

Websites, APIs, and cron jobs, checked from six regions.

Status Pages

Public and private pages, updated from the incident in one click.

Incident Response, Done Right

Incident kanban. Drag incidents through triage, mitigation, and review — the board stays in sync with Slack and your SLAs.

Warrn Scribe. Joins the incident call, transcribes in real time, and keeps a live view of what is known and what is next.

RCAs that write themselves. Drafted from the transcript and timeline in your template — refined by the team in Spaces.

Action items that get done. Extracted from the warroom, assigned to owners, and visible until they ship.

SLAs that enforce themselves. Update cadences, status page posts, and RCA deadlines set per severity — reminders land before anything slips.

iOS · Android · Slack

On-call, in your pocket

Acknowledge alerts, follow incidents, and manage your on-call from anywhere.

App Store Google Play

Acknowledge & escalate from your phone
Ask @warrn for context and similar incidents
Every action synced back to the timeline

Home

DMs

Activity

Later

# warrn-7450-rds-connections-high

Warrn AGENT 12:28 PM

You joined #warrn-7450-rds-connections-high

Sarah 12:28 PM

@warrn What's going on?

Warrn AGENT 12:28 PM

Here's the latest:

Connections on warrn-postgres crossed 80 (max ~112), starting 7 min ago and still climbing — currently at 96.
The API fleet is steady, so the growth looks like leaked connections from the worker pool deployed about 20 minutes ago.
No customer-facing errors yet. Warrnagen is investigating and has paged the on-call.

Dev 12:29 PM

@warrn Have we had any similar incidents recently?

Warrn AGENT 12:29 PM

Recent similar incidents include:

INC-643: Connection pool leak in billing worker caused the same climb in March — resolved in 30 minutes by rolling back.
INC-611: RDS hit max_connections during a migration showed the same pattern and latency spikes.

BIS| ≔ ⋮≡ | </>

Message the channel