The workflow should operate as a closed evaluation flywheel:
Business Goal
↓
Success Criteria
↓
Tracing and Logging
↓
Production Error Analysis
↓
Eval Dataset and Slices
↓
Human Rubric and Golden Set
↓
Code + Human + LLM Evaluators
↓
Offline Experiments
↓
CI/CD Quality Gates
↓
Production Monitoring
↓
New Failures → Regression Dataset
↺
One terminology adjustment: experimentation belongs mainly in the offline stage. The CI/CD stage should primarily apply repeatable evaluation suites and release gates rather than conduct uncontrolled experiments.
Define the business process the AI system should improve—not merely the model capability being demonstrated.
A weak goal is:
Build an LLM assistant for warranty claims.
A stronger goal is:
Reduce the average warranty-claim review time by 40% while maintaining the current financial accuracy, compliance, and customer-escalation standards.
Define the user, workflow and decision. Document who uses the system, what information is available, what decision or action it supports, and what happens after the AI returns an output.
Establish the current baseline. Measure the existing process before introducing AI:
Without a baseline, the organization cannot prove that the AI created value.
Separate three types of outcomes.
| Outcome level | Question | Warranty example |
|---|---|---|
| Business | Did the process improve? | Review time and claim leakage |
| Product | Did the user complete the task? | Reviewer can confidently decide |
| AI system | Did each AI component work? | Extraction, retrieval and decision accuracy |
Define the autonomy boundary. State whether the system:
Greater autonomy requires stricter release criteria.
Document unacceptable outcomes. For example:
Create a one-page AI Task Contract containing:
Business problem
Target users
Current baseline
Expected business outcome
AI-supported task
Autonomy level
In-scope and out-of-scope scenarios
Critical risks
Human escalation policy
Business owner
The business owner, product owner and domain expert agree on what business success and unacceptable failure mean.
Translate the business goal into measurable and testable product, quality, safety and operational requirements.
Anthropic recommends that success criteria be specific, measurable, achievable and relevant, and notes that most LLM applications require multidimensional evaluation rather than one overall quality score. (Claude Platform Docs)
Use multiple dimensions.
A production scorecard should normally include:
| Dimension | Example metric |
|---|---|
| Task quality | Correct claim decision |
| Completeness | Required fields included |
| Grounding | Claims supported by policy |
| Safety | Unauthorized actions |
| Privacy | PII exposure |
| Reliability | Success across repeated runs |
| Human interaction | Correct escalation |
| Operations | Latency and availability |
| Economics | Cost per successful task |
| Business | Review time and overturn rate |
Separate hard gates from optimization metrics.
Hard gates are non-negotiable:
Critical unauthorized actions = 0
PII leakage = 0
Required output schema validity = 100%
Optimization metrics can trade off against one another:
Task accuracy
Latency
Cost
Response length
Define thresholds by risk and slice. Do not rely only on an overall average.
Overall decision accuracy ≥ 95%
High-value claim accuracy ≥ 99%
Policy-exception accuracy ≥ 98%
Critical financial errors = 0
Specify the measurement method. “Groundedness must be good” is incomplete. Define:
Include the human baseline. Determine how qualified employees perform on the same cases. An AI target should be interpreted relative to human accuracy, cost and processing time.
Define uncertainty behavior. Include criteria for:
Create an Eval Scorecard:
| Metric | Method | Dataset | Threshold | Hard gate? | Owner |
|---|---|---|---|---|---|
| Decision correctness | Human-calibrated Judge | Golden set | ≥95% | Yes | Product |
| High-value errors | Code + human | High-risk slice | 0 | Yes | Business |
| Citation accuracy | Code + Judge | RAG set | ≥98% | Yes | AI team |
| p95 latency | Telemetry | Production-like load | ≤8 sec | No | Platform |
Every important requirement has a metric, evaluator, dataset, threshold and owner.
Make every AI result explainable, reproducible and debuggable.
Distributed tracing should capture the execution flow across model calls, retrieval, tools, agents and dependent services. OpenTelemetry semantic conventions provide standardized naming across traces, metrics and logs, helping teams analyze telemetry consistently across applications and platforms. (Microsoft Learn)
Use one end-to-end trace ID. Every user task should have a trace containing child spans for:
Request received
Input processing
Prompt construction
Retrieval
Reranking
Model invocation
Guardrail check
Tool call
External API or database call
Post-processing
Final response
Business-system update
Capture complete execution context.
At minimum, record:
trace_id and session_id
timestamp and environment
application version
prompt/template version
model and model version
model parameters
input and output
retrieved document IDs and versions
retrieval and reranking scores
tool names and arguments
tool responses and errors
guardrail decisions
latency by component
input/output tokens
estimated cost
user feedback
human correction
final business-system state
Version everything that can change.
This includes:
Make traces replayable. The team should be able to reconstruct the configuration that produced an output. Where external systems are nondeterministic, save sanitized tool results or immutable references to them.
Use structured events instead of free-text-only logs. For example:
{
"event_name": "warranty_policy_retrieved",
"trace_id": "abc123",
"policy_id": "EXT-2026-04",
"policy_version": "7",
"retrieval_score": 0.91
}
Protect sensitive data.
Log evaluator outputs as first-class data. Store the evaluator version, score, rationale category, confidence and timestamp—not just the final score.
Create a Trace and Logging Specification with field definitions, data classifications, retention rules and access policies.
For any failed output, the team can determine which version ran, what evidence was retrieved, which tools were called and what final state was created.
Convert vague reports such as “the chatbot gave a bad answer” into specific, reproducible and prioritized failure modes.
Microsoft recommends combining production monitoring with trace analysis, continuous evaluation, scheduled testing and failure clustering to understand and improve deployed AI applications. (Microsoft Learn)
Review a combination of targeted and random traffic.
Cases should enter the review queue through:
Random sampling is necessary because users do not report every problem.
Review the full trace. Do not evaluate only the final answer. Identify whether the primary failure originated in:
Input quality
Task definition
Prompt or instruction
Retrieval
Document freshness
Model reasoning
Tool selection
Tool arguments
External-system response
Business rule
Guardrail
Post-processing
Human workflow
Separate root cause from symptom.
Example:
Symptom:
The assistant gave the wrong coverage decision.
Root cause:
The retrieval system selected an obsolete policy version.
Contributing factor:
The prompt did not require policy-effective-date validation.
Use a consistent failure taxonomy.
Each incident should capture:
Failure category
Root cause
Severity
Affected project level
Affected business slice
Reproducibility
Frequency
Business impact
Detection method
Owner
Corrective action
Use severity before frequency. A rare unauthorized financial action may have greater priority than a frequent formatting problem.
Cluster errors. Group semantically similar failures and use a Pareto view:
38% incorrect retrieval
23% missing input validation
17% inappropriate escalation
12% calculation issues
10% miscellaneous
Assign one primary cause. Multiple tags are useful, but one primary cause helps teams assign ownership and measure whether the fix worked.
Run a recurring failure-review meeting. A weekly review should decide:
Maintain a searchable Failure Registry linked to traces, Eval cases, fixes and release versions.
Every meaningful production error has a reproducible case, severity, root cause, owner and disposition.
Create a representative but intentionally challenging dataset that measures both normal performance and critical risks.
OpenAI recommends combining production, historical, human-curated, domain-specific and synthetic examples, including typical, edge and adversarial cases. Google recommends using examples representative of production inputs and suggests approximately 100 examples as a useful starting point for aggregated metrics, although actual size should be driven by task and slice coverage. (OpenAI Developers)
Use multiple data sources.
Recommended hierarchy:
Synthetic data should expand coverage—not define reality.
Represent production distribution and deliberately oversample risk.
A dataset may contain:
60% common production cases
15% edge cases
10% historical failures
10% high-risk cases
5% adversarial cases
The exact mix depends on the application. Critical cases should have enough examples to support meaningful decisions even if they are rare in production.
Define slices before running the experiment.
Example warranty slices:
Standard claim
Extended warranty
Expired coverage
Missing repair date
Conflicting information
High claim value
Policy exception
Duplicate claim
Non-English notes
Long attachment
Prompt injection
Human escalation required
Store rich metadata.
{
"case_id": "WC-00125",
"source": "production_anonymized",
"risk_level": "critical",
"slice_tags": [
"high_value",
"policy_exception"
],
"input": {},
"expected_behavior": {},
"reference_answer": null,
"required_evidence": [],
"rubric_version": "v1.3"
}
Separate datasets by purpose.
| Dataset | Purpose |
|---|---|
| Development set | Prompt and architecture iteration |
| Calibration set | Human and Judge alignment |
| Regression set | Previously fixed failures |
| Release set | Predeployment quality gate |
| Hidden holdout | Unbiased final comparison |
| Production sample | Drift and real-world validation |
Do not repeatedly optimize against the hidden holdout.
Deduplicate cases. Near-duplicate examples can create misleadingly high confidence and overemphasize one scenario.
Prevent data leakage. Keep Eval references and expected answers outside the model’s retrieval corpus and prompts unless the task explicitly requires them.
Version the dataset. Never silently modify an existing release dataset. Publish a new version with change notes.
Every important user intent, risk type and failure category is represented by a named slice with adequate test coverage.
Create the authoritative definition of correct and acceptable behavior.
Use qualified domain experts. General annotators may judge clarity, but policy correctness, financial eligibility and regulatory compliance should be labeled by people who understand those decisions.
Write criterion-specific rubrics. Avoid one vague question such as:
Is this a good answer?
Instead, separate:
Decision correctness
Evidence support
Completeness
Policy compliance
Escalation correctness
Communication quality
Prefer clear decision boundaries.
Example:
PASS:
The coverage decision matches the policy effective on the
repair date, incorporates all applicable exclusions and requests
human review when required information is missing.
FAIL:
The decision contradicts an applicable policy, ignores an
exclusion or makes a definitive determination without required data.
NOT APPLICABLE:
The criterion does not apply to this case.
UNCERTAIN:
The policy itself is ambiguous and requires adjudication.
Include examples around the boundary. Provide positive, negative and borderline examples. These are often more useful than abstract descriptions.
Define acceptable alternatives. For generative tasks, the golden set should not require one exact sentence. Store:
Blind the reviewers. Reviewers should not know which model or prompt generated each result when comparing candidates.
Double-label an important subset. Have multiple experts independently label high-risk and calibration cases. Resolve disagreements through adjudication.
Measure reviewer agreement. Low agreement may indicate:
Do not immediately treat it as a model problem.
Separate Judge calibration data from Judge validation data. Avoid evaluating the Judge only on examples used to tune its prompt.
Refresh the golden set carefully. Policy and business-process changes may invalidate old labels. Preserve history and effective dates instead of overwriting labels.
Create a version-controlled Rubric Package containing definitions, examples, adjudication rules, reviewers and policy references.
Independent experts can apply the rubric consistently, and disagreements have a defined adjudication process.
Anthropic recommends choosing the fastest, most reliable and scalable grading method suitable for the task: code-based grading for objective checks, human evaluation for nuanced judgments, and LLM-based grading for scalable semantic assessment after reliability has been validated. (Claude Platform Docs)
0 or 1Example:
{
"passed": false,
"metric": "claim_amount_accuracy",
"expected": 2850.00,
"actual": 2580.00,
"difference": -270.00
}
Pairwise comparison is often easier and more consistent than independently assigning two absolute quality scores.
Use one focused criterion per Judge where possible.
Better:
Does every coverage statement have support in the supplied policy?
Worse:
Is this response accurate, relevant, safe, useful and professional?
Provide a detailed rubric and relevant evidence.
Use structured outputs.
{
"label": "FAIL",
"failure_category": "unsupported_decision",
"severity": "critical",
"evidence": [
"Policy section 4.2 excludes this repair type."
],
"confidence": 0.94
}
Prefer discrimination tasks. Classification, pairwise comparison and criterion-based scoring tend to be more reliable than asking a Judge for an unconstrained critique. OpenAI explicitly recommends aligning evaluators with models’ relative strength at discriminating between options. (OpenAI Developers)
Calibrate against humans.
Measure:
Human–Judge agreement
False-positive rate
False-negative rate
Agreement by slice
Agreement by severity
Stability across repeated runs
Version the entire Judge.
Judge model
Judge prompt
Rubric
Temperature
Input schema
Output parser
Threshold
Do not use the candidate model as the only Judge of itself for high-stakes decisions.
Route uncertain and critical cases to humans.
Code check
↓
If objective failure → Fail
↓
LLM Judge for semantic criteria
↓
If critical, uncertain or disagreement → Human review
Every metric has the least expensive evaluator that remains sufficiently reliable for its risk level.
Determine whether a candidate configuration is genuinely better than the current baseline before deployment.
Compare candidate and baseline on the same cases. Paired comparisons reduce noise from dataset composition.
Change one major variable at a time where practical.
Examples:
Prompt A vs Prompt B
Model A vs Model B
Chunking method A vs B
Retriever A vs B
Agent policy A vs B
For architectural experiments where multiple elements must change, document the full configuration.
Freeze the experiment definition before running it.
Record:
Dataset version
Baseline configuration
Candidate configuration
Evaluator versions
Success thresholds
Primary metrics
Critical slices
Decision rule
Use repeated trials for stochastic behavior. Run important cases multiple times when temperature, tools or agent planning can produce materially different trajectories.
Track:
Evaluate both components and end-to-end outcomes.
For RAG:
Retrieval → Context → Answer → Citation → Task success
For agents:
Plan → Tool selection → Arguments → Tool result → State → Outcome
Google’s agent evaluation guidance distinguishes final-response evaluation from trajectory evaluation, including whether the agent used the expected tools in the expected sequence. (Google Cloud Documentation)
Always report slices.
Overall accuracy: 95%
Standard claims: 98%
High-value claims: 99%
Policy exceptions: 82%
Missing-information: 76%
The candidate should not win based on the overall average while regressing on critical slices.
Measure the quality–cost–latency frontier. A 1% quality improvement may not justify a 300% cost increase.
Perform qualitative error analysis. Review representative wins, losses and disagreements instead of relying solely on aggregate scores.
Preserve an untouched holdout. Use it only for significant release decisions, not daily prompt tuning.
Create an Experiment Report:
Hypothesis
Baseline
Candidate
Dataset
Metrics
Overall results
Slice results
Critical failures
Cost and latency
Representative examples
Decision
Known limitations
A candidate is promoted only when it satisfies the predeclared decision rule and introduces no unacceptable slice-level regression.
Prevent changes to prompts, models, tools, retrieval, policies or application code from introducing known failures.
OpenAI recommends continuous evaluation on every change and growing the Eval set over time. Microsoft similarly recommends cloud-based evaluation for scalable predeployment testing and CI/CD integration. (OpenAI Developers)
Use a tiered test strategy.
Fast and inexpensive:
Schema tests
Critical unit Evals
Prompt-instruction tests
Tool-argument tests
Small regression sample
Full regression dataset
RAG component tests
Agent trajectory tests
Safety and security tests
Repeated-run tests
Cost and latency checks
Complete frozen release set
Hidden holdout
Load testing
Red-team suite
End-to-end integration tests
Canary readiness checks
Use both absolute and relative gates.
Absolute:
Critical failures must equal zero.
Relative:
Candidate cannot regress more than 2 percentage points
against production on any critical slice.
Do not gate only on averages. Gates should operate on:
Separate hard failure, warning and informational metrics.
FAIL: unauthorized action
WARN: cost increased by 12%
INFO: average response length increased
Control flaky Evals.
Store all evaluation artifacts. Retain configuration, outputs, individual scores and logs for every release candidate.
Set cost and runtime budgets. A CI suite that takes hours and costs too much will eventually be bypassed.
Include rollback criteria. A deployment plan should define the signals that trigger automatic or manual rollback.
Critical safety violations = 0
Critical financial errors = 0
Schema validity = 100%
Overall task success ≥ 95%
Every critical slice ≥ approved threshold
No critical slice regression > 2 percentage points
p95 latency ≤ 8 seconds
Average cost ≤ $0.10 per completed task
No production change can bypass the agreed quality, risk, cost and latency gates.
Detect degradation, new failure modes and changes in business performance after release.
AWS groups production monitoring into three pillars: application and system health, business and user-interaction health, and model/AI quality health. Microsoft additionally recommends sampled continuous evaluation, scheduled dataset evaluation, scheduled red teaming and threshold-based alerts. (AWS Documentation)
Monitor:
Availability
Error rate
Timeout rate
Latency p50/p95/p99
Throughput
Tool/API failures
Database failures
Token usage
Infrastructure usage
Cost
Monitor:
Task success
Groundedness
Unsupported claim rate
Retrieval quality
Tool-call correctness
Escalation correctness
Safety violations
PII leakage
Conversation abandonment
Repeated-agent loops
Monitor:
User adoption
Task completion
Human acceptance rate
Override rate
Processing-time reduction
Cost savings
Claim leakage
Customer satisfaction
Escalation volume
Combine different evaluation frequencies.
| Evaluation type | Recommended use |
|---|---|
| 100% deterministic checks | Schema, permissions, forbidden actions |
| Sampled semantic Evals | Groundedness, relevance and completeness |
| Targeted high-risk Evals | High-value or sensitive transactions |
| Scheduled regression run | Detect system or corpus drift |
| Scheduled red teaming | Discover new attack patterns |
| Human audit sample | Validate automated evaluators |
MLflow’s production evaluation guidance similarly distinguishes offline evaluation on curated datasets from automatic evaluation on sampled live traces, and recommends using higher sampling during development and targeted filters for high-value or high-risk production traffic. (MLflow AI Platform)
Segment dashboards by version and slice.
Always make it possible to compare:
Model version
Prompt version
Application version
User group
Intent
Language
Risk level
Document source
Tool
Business unit
Alert on sustained trends. Avoid excessive alerts from individual low-confidence Judge decisions. Use rate, duration and severity thresholds.
Example:
Alert when groundedness failures exceed 5%
for 15 minutes on at least 100 evaluated traces.
Critical deterministic violations may warrant immediate alerts.
Use canary or shadow deployment.
Capture delayed ground truth. Some business outcomes become available days later—for example, a human overturn, customer complaint or claim audit. Join those outcomes back to the original trace.
Monitor the evaluators. A Judge score trend may change because the Judge model or prompt changed, not because the application changed.
The team can detect operational, AI-quality and business degradation by version and slice before it creates widespread impact.
Ensure that the organization does not repeatedly experience the same failure.
Create a reproducible Eval case before or alongside the fix.
The sequence should normally be:
Production failure detected
↓
Trace reviewed
↓
Failure reproduced
↓
Eval case created
↓
Current system confirmed to fail
↓
Fix implemented
↓
Eval case passes
↓
Case added to permanent regression suite
Sanitize production data. Remove or tokenize personal and confidential information while preserving the characteristics that caused the failure.
Capture the complete expected behavior. Do not merely store the expected final sentence. Store:
Expected decision
Required evidence
Prohibited behavior
Expected tool use
Expected final state
Escalation requirements
Add neighboring cases. A single incident may encourage an overly narrow fix.
If the failure involves an expired policy, add variations such as:
This tests whether the system learned the general rule rather than memorizing one input.
Tag the case appropriately.
regression
high_value
policy_version
date_boundary
critical
production_2026_Q3
Link everything.
The Eval case should link to:
Set an intake SLA. For example:
Critical failure:
Eval case added within one business day
High-severity failure:
Added within one week
Deduplicate and maintain the suite. More cases do not automatically mean a better dataset. Merge near-duplicates, retire obsolete cases carefully and preserve historical versions.
Track coverage growth. Report how much of the regression suite originates from:
Production incidents
User feedback
Human review
Red teaming
Domain-expert scenarios
Synthetic expansion
Every important resolved failure has become a permanent, versioned regression test that fails on the old implementation and passes on the corrected one.
| Stage | Primary artifact |
|---|---|
| Business goal | AI Task Contract |
| Success criteria | Eval Scorecard |
| Tracing/logging | Trace Schema and Logging Standard |
| Error analysis | Failure Registry |
| Dataset and slices | Versioned Eval Dataset |
| Human rubric | Golden Set and Rubric Package |
| Evaluators | Evaluator Catalog |
| Offline experiments | Experiment Report |
| CI/CD | Release-Gate Policy |
| Production monitoring | Quality, Operations and Business Dashboards |
| Failure feedback | Regression Case and Linked Incident Record |
| Area | Accountable role | Key contributors |
|---|---|---|
| Business goal | Business owner | Product manager |
| Success criteria | Product manager | Domain experts, AI team |
| Tracing | Platform lead | AI and data engineers |
| Error analysis | AI product lead | Operations and SMEs |
| Dataset | AI/data lead | SMEs and data governance |
| Golden set | Domain lead | Product and AI engineers |
| Evaluators | AI engineering lead | SMEs and platform |
| Offline experiments | AI team | Product and data science |
| CI/CD gates | Engineering lead | Security and platform |
| Monitoring | Production owner | AI, platform and business |
| Failure feedback | Product owner | All participating teams |
The maturity of the framework can ultimately be measured by one question:
When an AI failure occurs in production, how quickly can the organization explain it, reproduce it, fix it and guarantee that it will not recur?