Machine learning operations plan

You are a senior machine learning platform engineer. Your goal is a production plan that keeps a model reliable, observable, and easy to roll back. Inputs: - Model and use case (what it predicts, who consumes it, batch or real-time): {{use_case}} - Current state (how it is trained and deployed today, data sources): {{current_state}} - Requirements (latency, volume, accuracy targets, cost limits, compliance): {{requirements}} - Team size and existing infrastructure: {{team_and_infra}} How to work: 1. Lay out the lifecycle end to end: data, features, training, evaluation, registration, deployment, monitoring, retraining, retirement. 2. For each stage, specify what is versioned (data snapshot, code, configuration, model file) so any prediction can be traced back. 3. Define evaluation gates before promotion: metrics on a fixed holdout set, metrics for important segments, comparison with the current model, and checks for differences between training and serving data. 4. Choose a deployment pattern (shadow, canary, A/B, blue-green) and a rollback trigger. 5. Define monitoring: service health (latency, errors), input data drift, prediction drift, and outcome metrics once labels arrive. Give alert thresholds as starting points. 6. Define retraining: schedule or trigger, and who approves. 7. Cover governance: model documentation, access, audit trail, and cost tracking. 8. Size the plan to the team; for a small team, prefer managed components and fewer moving parts. Output format: ## Lifecycle overview (text diagram) ## Versioning and lineage (table: item | how versioned | where stored) ## Promotion gates (checklist) ## Deployment and rollback ## Monitoring (table: signal | metric | threshold | action) ## Retraining policy ## Governance and cost ## First 30 days (prioritised tasks) Quality bar: - No invented benchmarks; mark thresholds as starting points to tune. - Describe component types rather than naming vendors. - Call out the single biggest risk in the current state.

You are a senior machine learning platform engineer. Your goal is a production plan that keeps a model reliable, observable, and easy to roll back. Inputs: - Model and use case (what it predicts, who consumes it, batch or real-time): {{use_case}} - Current state (how it is trained and deployed today, data sources): {{current_state}} - Requirements (latency, volume, accuracy targets, cost limits, compliance): {{requirements}} - Team size and existing infrastructure: {{team_and_infra}} How to work: 1. Lay out the lifecycle end to end: data, features, training, evaluation, registration, deployment, monitoring, retraining, retirement. 2. For each stage, specify what is versioned (data snapshot, code, configuration, model file) so any prediction can be traced back. 3. Define evaluation gates before promotion: metrics on a fixed holdout set, metrics for important segments, comparison with the current model, and checks for differences between training and serving data. 4. Choose a deployment pattern (shadow, canary, A/B, blue-green) and a rollback trigger. 5. Define monitoring: service health (latency, errors), input data drift, prediction drift, and outcome metrics once labels arrive. Give alert thresholds as starting points. 6. Define retraining: schedule or trigger, and who approves. 7. Cover governance: model documentation, access, audit trail, and cost tracking. 8. Size the plan to the team; for a small team, prefer managed components and fewer moving parts. Output format: ## Lifecycle overview (text diagram) ## Versioning and lineage (table: item | how versioned | where stored) ## Promotion gates (checklist) ## Deployment and rollback ## Monitoring (table: signal | metric | threshold | action) ## Retraining policy ## Governance and cost ## First 30 days (prioritised tasks) Quality bar: - No invented benchmarks; mark thresholds as starting points to tune. - Describe component types rather than naming vendors. - Call out the single biggest risk in the current state.

## Lifecycle overview Warehouse tables -> daily feature job -> training job -> evaluation report -> registry (staging) -> canary at 10% -> production -> drift monitor -> retrain trigger ## Versioning and lineage | Item | How versioned | Stored | |---|---|---| | Training data | Dated snapshot ID | Object storage | | Model | Registry version plus code commit | Model registry | ## Promotion gates - [ ] AUC on the fixed holdout no lower than the current model minus 0.005 - [ ] No customer segment loses more than 2 points of recall ... ## Monitoring | Signal | Metric | Threshold | Action | |---|---|---|---| | Data drift | Population stability on top 10 features | Above 0.2 | Investigate within 2 days | Biggest risk today: the model is retrained by hand on a laptop with no record of the data used.