ANS · 6 cases
Answer
Evidence and permission are sufficient.
Evaluation set
Test whether the agent knows when not to answer
The synthetic set contains 18 questions: six answerable, six requiring a certified definition, and six that should be refused.
DEF · 6 cases
Needs definition
Data exists, but meaning is incomplete.
REF · 6 cases
Refuse
Data, permission or approved action is absent.
Report a four-part scorecard
| Measure | What it reveals |
|---|---|
| Answer rate | How often the system attempts an answer. This is coverage, not accuracy. |
| Correct-answer rate | Correct answers divided by answered questions, with result-based grading where possible. |
| Appropriate-refusal rate | How often refusal-required questions are safely declined. |
| Harmful-answer rate | Unsupported, unauthorized or materially wrong answers divided by all questions. Set an explicit release threshold. |
Recommended protocol
- Choose a meaningful but bounded slice of the organization’s schema.
- Replace the synthetic field names with equivalents while preserving the three bands.
- Run the same set across every candidate model, configuration and verification mode.
- Grade returned data, not only similarity between SQL strings.
- Inspect results by band, persona and control domain.
- Re-run after schema, semantic, policy, model or prompt changes.
Do not use the sample as a benchmark claim. It is a template for building an organization-specific evaluation set. Performance on one schema does not predict performance on another.