Warrn
Your teams
Wwarrn engg
All teams
You're on call

Ends in 6d 3h

VVighnesh KS
MEDIUM
StatusResolved
Teamwarrn engg
SourceCloudWatchCloudWatch (SNS)

Created

Jun 17 at 7:31 PM (18 days ago)

Resolved by

System

Deduplication key

ap-south-1:warrn-prod-tg-healthy-hosts-low

Affected Customers

HHubSpotNNotion

Add customer

Extra Properties

CloudWatch

CloudWatchView alarm in AWS Console

State

OKALARM

Metric

HealthyHostCount in AWS/ApplicationELB

Threshold

LessThanThreshold 2.0

Investigation Complete

6/24/2026, 2:58:06 PM

95%Request Re-run

The healthy-host-count alarm fired because a Terraform-managed rolling deployment inadvertently kept HealthyHostCount at 1 for 5 consecutive minutes (13:55–13:59 UTC), tripping the 4-minute sustained threshold designed to distinguish real outages from normal rolling-deploy churn. CloudTrail shows Terraform deregistered warrn-api-1 at 13:55:12, re-registered it at 13:58:00, then deregistered warrn-api-2 only 21 seconds later — before api-1 had passed its ALB health checks — so the count never recovered to 2 between the two operations. CloudWatch fired ALARM at 14:01:22 and auto-resolved to OK at 14:03:22. Both EC2 instances remained running with passing status checks throughout; no crash, no stop-instance event, and no 5xx errors were observed. The service was degraded to a single host for ~5 minutes but was never completely down.

ACTIONS0/4 done

Increase inter-step delay in Terraform rolling deploy

Soon

Insert a minimum wait of 90–120 seconds between RegisterTargets for instance N and DeregisterTargets for instance N+1, keeping HealthyHostCount at 2 throughout the deploy.

Add a health-check gate before proceeding to next deploy step

When Possible

Have the deploy script poll DescribeTargetHealth after each RegisterTargets and only proceed once HealthyHostCount returns to 2.

Review alarm evaluation_periods vs. worst-case deploy overlap

When Possible

Consider raising evaluation_periods to 6 to add headroom, or suppress the healthy-hosts alarm during active Terraform runs.

Resolved · Was open for 1mActivity
Join WarroomDeclare Incident
Warrnagen
Why did the healthy-hosts alarm fire during the deploy?
Ask about this alert...

Everything for Incident Response

From detection to resolution, Warrn provides the complete toolkit for modern DevOps teams.

On-call. Rotation schedules with automatic escalations and overrides, across your teams and services.

Incident Response, Done Right

Incident kanban background

Incident kanban. Drag incidents through triage, mitigation, and review — the board stays in sync with Slack and your SLAs.

Warrn Scribe background

Warrn Scribe. Joins the incident call, transcribes in real time, and keeps a live view of what is known and what is next.

bg c4

RCAs that write themselves. Drafted from the transcript and timeline in your template — refined by the team in Spaces.

bg c3

Action items that get done. Extracted from the warroom, assigned to owners, and visible until they ship.

bg c1

SLAs that enforce themselves. Update cadences, status page posts, and RCA deadlines set per severity — reminders land before anything slips.

iOS · Android · Slack

On-call, in your pocket

Acknowledge alerts, follow incidents, and manage your on-call from anywhere.

  • Acknowledge & escalate from your phone
  • Ask @warrn for context and similar incidents
  • Every action synced back to the timeline

Integrates with your stack

Connect with the tools you already use. Get alerts from anywhere, notify your team everywhere.

View all integrations

Warrnagen

Agent that can triage, respond, and resolve incidents in real-time.

Switch in minutes, not months

Import your policies, schedules, and integrations automatically. We help you switch.

Built for Modern Teams

Everything you need to detect, respond, and resolve incidents faster.

Agentic. Intelligent triage from day one.

Modern. Built for how teams work today.

Simple. Setup in minutes, not days.

Start with your noisiest service

Point one alert source at Warrn and watch it get quiet.

  • 14-day free trial
  • No credit card required

Starting at

$0/month

Free for small teams up to 5 users

Start Free Trial