Service reliability targets and alerting plan

You are a site reliability engineer. Design reliability targets and alerting for one service, based on what its users actually experience. Inputs - The service and what it does: {{service_description}} - The user journeys that matter most: {{user_journeys}} - Current metrics or incident history (availability, latency, error rates): {{current_metrics}} - Team constraints (on-call size, tooling, business hours): {{team_constraints}} How to work 1. For each key journey, decide what 'good' means to a user: the request succeeded, and it was fast enough. 2. Define each indicator as a ratio of good events to valid events, and say exactly where it is measured (load balancer, service, client). 3. Propose a target and window for each, justified by current performance and business need. Do not choose extra nines without a reason; each one is much more expensive. 4. Convert each target into an error budget: allowed bad events or minutes per 30 days. Show the arithmetic. 5. Design alerts on budget burn rate using a short and a long window together, so pages fire for fast burns and tickets for slow ones. Give the thresholds. 6. List dashboard panels covering latency (successful and failed separately), traffic, errors by type, and saturation (CPU, memory, queues, connection pools). 7. Write an error budget policy: what the team does when the budget is healthy, at risk, or spent. 8. Name two or three repetitive operational tasks worth automating and the safer rollout practice (staged or canary) for changes. Output format - Indicator and target table: Journey, Indicator (good/valid), Measured at, Target, Window, Budget - Alert table: Name, Condition, Windows, Severity (page or ticket), First action - Dashboard panel list - Error budget policy (3-5 bullets) - Automation and rollout recommendations Quality bar - Mark every number you assumed rather than read from the inputs. - Every page must be actionable by the on-call person; if not, make it a ticket. - Plain language so product owners can agree to the targets.

You are a site reliability engineer. Design reliability targets and alerting for one service, based on what its users actually experience. Inputs - The service and what it does: {{service_description}} - The user journeys that matter most: {{user_journeys}} - Current metrics or incident history (availability, latency, error rates): {{current_metrics}} - Team constraints (on-call size, tooling, business hours): {{team_constraints}} How to work 1. For each key journey, decide what 'good' means to a user: the request succeeded, and it was fast enough. 2. Define each indicator as a ratio of good events to valid events, and say exactly where it is measured (load balancer, service, client). 3. Propose a target and window for each, justified by current performance and business need. Do not choose extra nines without a reason; each one is much more expensive. 4. Convert each target into an error budget: allowed bad events or minutes per 30 days. Show the arithmetic. 5. Design alerts on budget burn rate using a short and a long window together, so pages fire for fast burns and tickets for slow ones. Give the thresholds. 6. List dashboard panels covering latency (successful and failed separately), traffic, errors by type, and saturation (CPU, memory, queues, connection pools). 7. Write an error budget policy: what the team does when the budget is healthy, at risk, or spent. 8. Name two or three repetitive operational tasks worth automating and the safer rollout practice (staged or canary) for changes. Output format - Indicator and target table: Journey, Indicator (good/valid), Measured at, Target, Window, Budget - Alert table: Name, Condition, Windows, Severity (page or ticket), First action - Dashboard panel list - Error budget policy (3-5 bullets) - Automation and rollout recommendations Quality bar - Mark every number you assumed rather than read from the inputs. - Every page must be actionable by the on-call person; if not, make it a ticket. - Plain language so product owners can agree to the targets.

| Journey | Indicator | Measured at | Target | Window | Budget | |---|---|---|---|---|---| | Checkout | Requests not returning a server error / all valid checkout requests | Load balancer | 99.9% | 30 days | 43 minutes of full outage, or about 1,300 failed checkouts at current volume (assumed 1.3M/month) | | Checkout speed | Requests under 800 ms / all successful requests | Service | 99% | 30 days | 13,000 slow requests | Alerts | Name | Condition | Windows | Severity | First action | |---|---|---|---|---| | Checkout fast burn | Burn rate above 14x | 5 min and 1 h | Page | Check the latest deploy, roll back if it correlates | | Checkout slow burn | Burn rate above 3x | 6 h and 3 days | Ticket | Review error types next business day | Policy: if more than 75 percent of the budget is used with half the window left, pause feature releases for this service...