.NET › CI/CD, Publishing & Deployment

How do you implement automated rollback in a CI/CD pipeline triggered by post-deployment monitoring signals (error rate, latency)?

Advanced

Answer

Automated rollback typically involves: deploying the new version (often to a canary subset or staging slot first), running an automated monitoring window that queries metrics (error rate, p99 latency, custom business KPIs) from an observability platform, comparing against a defined threshold or against the previous version's baseline, and automatically triggering a rollback (reverting a slot swap, scaling canary traffic back to zero, or redeploying the previous known-good artifact) if the new version's metrics breach the threshold within the observation window -- without requiring a human to manually notice and react to a problem.
# Simplified rollback logic
- name: Monitor post-deploy
  run: |
    sleep 300  # observation window
    error_rate=$(curl -s "$METRICS_API/error_rate?window=5m")
    if (( $(echo "$error_rate > 0.05" | bc -l) )); then
      echo "Error rate too high, rolling back"
      az webapp deployment slot swap --slot staging --target-slot production --action swap
      exit 1
    fi
Real-world example A high-traffic API automatically rolls back a deployment within 3 minutes if its post-deployment error rate exceeds 5%, avoiding hours of degraded service that would otherwise require an engineer to notice an alert and manually intervene at 2 AM.

Common follow-ups: What metrics are most reliable signals for automated rollback decisions?;How do you avoid false-positive rollbacks from normal traffic variance?