.NET › CI/CD, Publishing & Deployment
How do you implement automated rollback in a CI/CD pipeline triggered by post-deployment monitoring signals (error rate, latency)?
AdvancedAnswer
Automated rollback typically involves: deploying the new version (often to a canary subset or staging slot first), running an automated monitoring window that queries metrics (error rate, p99 latency, custom business KPIs) from an observability platform, comparing against a defined threshold or against the previous version's baseline, and automatically triggering a rollback (reverting a slot swap, scaling canary traffic back to zero, or redeploying the previous known-good artifact) if the new version's metrics breach the threshold within the observation window -- without requiring a human to manually notice and react to a problem.
# Simplified rollback logic
- name: Monitor post-deploy
run: |
sleep 300 # observation window
error_rate=$(curl -s "$METRICS_API/error_rate?window=5m")
if (( $(echo "$error_rate > 0.05" | bc -l) )); then
echo "Error rate too high, rolling back"
az webapp deployment slot swap --slot staging --target-slot production --action swap
exit 1
fiReal-world example A high-traffic API automatically rolls back a deployment within 3 minutes if its post-deployment error rate exceeds 5%, avoiding hours of degraded service that would otherwise require an engineer to notice an alert and manually intervene at 2 AM.