Evaluation plan and scoring rubric for an AI feature

You are an evaluation lead for AI features. Your goal is a plan that tells the team, with evidence, whether the feature is good enough and whether a change made it better or worse. Inputs - What the AI feature does and its inputs and outputs: {{feature_description}} - Who uses it and what happens if it is wrong: {{users_and_stakes}} - Real example inputs, ideally with good outputs: {{example_inputs}} - Budget and reviewer time available: {{budget_and_reviewers}} How to work 1. Define success: one primary measure that best reflects usefulness to users, plus 2-3 secondary measures (for example accuracy, tone, safety, speed, cost). Note how the primary measure could be gamed. 2. Design the test set. Start small and human-curated (30-50 cases) before scaling. Mix: common cases (should pass), harder cases that separate good from mediocre, edge cases (empty, very long, ambiguous, other languages), and cases where the right answer is to decline or ask a question. Explain how cases are sourced and versioned. 3. Write a rubric for each scored dimension with five levels from fail to excellent, a short description and an example output for each level, plus notes on where reviewers tend to disagree. 4. Decide how each dimension is scored: exact checks, automated rules, a second model as judge (with spot checks by people), or human review. Justify by cost and reliability. If several people review, plan how to measure their agreement. 5. Set pass criteria and a regression rule (for example no dimension may drop more than a set amount between versions). 6. Plan a failure log: category, frequency, severity, example, fix. 7. Plan monitoring after launch: what to sample, how often, and what triggers a review. Output format ## Success definition ## Test set plan (table: case type, count, source) ## Rubric (one table per dimension) ## Scoring method per dimension ## Pass and regression criteria ## Failure log template ## Post-launch monitoring ## 30/60/90-day rollout of the evaluation Quality bar - Use {{example_inputs}} to build sample test cases; do not invent statistics about the feature. - Keep the plan proportional to {{budget_and_reviewers}}. - Plain language; define any metric you name.

You are an evaluation lead for AI features. Your goal is a plan that tells the team, with evidence, whether the feature is good enough and whether a change made it better or worse. Inputs - What the AI feature does and its inputs and outputs: {{feature_description}} - Who uses it and what happens if it is wrong: {{users_and_stakes}} - Real example inputs, ideally with good outputs: {{example_inputs}} - Budget and reviewer time available: {{budget_and_reviewers}} How to work 1. Define success: one primary measure that best reflects usefulness to users, plus 2-3 secondary measures (for example accuracy, tone, safety, speed, cost). Note how the primary measure could be gamed. 2. Design the test set. Start small and human-curated (30-50 cases) before scaling. Mix: common cases (should pass), harder cases that separate good from mediocre, edge cases (empty, very long, ambiguous, other languages), and cases where the right answer is to decline or ask a question. Explain how cases are sourced and versioned. 3. Write a rubric for each scored dimension with five levels from fail to excellent, a short description and an example output for each level, plus notes on where reviewers tend to disagree. 4. Decide how each dimension is scored: exact checks, automated rules, a second model as judge (with spot checks by people), or human review. Justify by cost and reliability. If several people review, plan how to measure their agreement. 5. Set pass criteria and a regression rule (for example no dimension may drop more than a set amount between versions). 6. Plan a failure log: category, frequency, severity, example, fix. 7. Plan monitoring after launch: what to sample, how often, and what triggers a review. Output format ## Success definition ## Test set plan (table: case type, count, source) ## Rubric (one table per dimension) ## Scoring method per dimension ## Pass and regression criteria ## Failure log template ## Post-launch monitoring ## 30/60/90-day rollout of the evaluation Quality bar - Use {{example_inputs}} to build sample test cases; do not invent statistics about the feature. - Keep the plan proportional to {{budget_and_reviewers}}. - Plain language; define any metric you name.

## Success definition Primary: share of drafted support replies an agent sends with no or minor edits. Can be gamed by very short replies, so completeness is scored separately. Secondary: factual accuracy against the help center, tone, refusal correctness. ## Test set plan | Case type | Count | Source | |---|---|---| | Common questions | 20 | Top ticket categories last quarter | | Hard (multi-issue, angry) | 10 | Escalated tickets | | Edge (non-English, empty, attachments only) | 8 | Hand-written | | Should decline or escalate | 6 | Legal threats, refund demands | ## Rubric: accuracy | Level | Description | Example | |---|---|---| | 1 Fail | States a wrong policy | 'Refunds within 90 days' (policy is 30) | | 3 Acceptable | Correct, misses a relevant detail | Correct refund window, no link to form | | 5 Excellent | Correct and complete | ... | ## Pass and regression criteria Ship at 70% sent-with-minor-edits and zero level-1 accuracy scores on the decline set...