Mid-market clients ask for a grounded assistant over a narrow domain, from pharma specifications to insurance clauses to industrial SOPs, and almost none of them have a labelled test set to score it with. Teams ship on manual spot checks and vibes. The first regression is caught by an end user. This note sets up the harness we plan to publish for that gap.
The failure mode
A new domain assistant goes live. Nobody has written down what "correct" looks like, because writing down 10,000 question-answer pairs is more work than building the assistant was. Over the next three months, two things happen in parallel. The retriever changes because the corpus grew. The prompt changes because an engineer tightened the format. Nobody can tell if the assistant got better or worse, because there is no score to compare against. The regression is detected when a user escalates.
This is not a model-quality problem. It is an evaluation-infrastructure problem, and it blocks every subsequent improvement. One industry survey put the share of production RAG systems suffering from hallucinations, missed retrieval or broken citations at roughly 35%, which is the kind of number that is only visible when someone is actually measuring.
The measurable question
For a domain assistant without a labelled golden set, can a combination of (a) subject-matter-expert-written seed questions, (b) synthetic adversarials generated from the corpus, and (c) reference-free quality signals, produce a stable regression score that moves predictably when the model, retriever or prompt is changed?
"Stable" here means test-retest variance under 5% on an unchanged pipeline across three runs. "Moves predictably" means a deliberate degradation (swap the embedding model for a weaker one; shorten the context window; corrupt 10% of retrieved chunks) is detected.
Conditions we plan to publish with the result
- Anchor engagement: one of the live grounded assistants in ScroBits' delivery portfolio, de-identified. Candidate anchors include the infrastructure query agent in production at a 300-person IT company and the MetricStream GRC assistant engagement. One, not both, will be the reference.
- Corpus: between 500 and 10,000 documents, actual client content, no synthetic corpus.
- Seed set: 50 SME-written questions, 50 synthetic questions generated from random passages, 50 adversarial questions (unanswerable, ambiguous, out-of-scope).
- Harness: reference-free scoring using RAGAS-style metrics (faithfulness, context precision, answer relevance), extended with a citation-correctness check and a refusal-correctness check.
- Judge: a frontier LLM used as judge, with calibration against a 50-item SME-labelled subset to establish judge-vs-human agreement before any score is reported.
- Baseline runs: three repeated runs on the unchanged pipeline to estimate variance.
- Perturbation runs: controlled degradations introduced one at a time to confirm the harness detects them.
Method we plan to test
The harness has five parts. None of them are new in the literature. The contribution is the stable combination and the honesty gates on top.
- Seed question extraction. A short interview protocol for an SME to produce 50 questions grounded in the live corpus. These are the hardest to automate and the most valuable, so we optimise the SME's time, not ours.
- Synthetic question generation. For each sampled passage, generate a question whose answer is grounded in that passage. Filter out any question whose answer the model can produce from parametric knowledge alone (otherwise the test does not stress retrieval). Confirm 10 to 15% of the final set is intentionally unanswerable so refusal behaviour is tested.
- Reference-free scoring. For each generated answer:
- Faithfulness: fraction of claims in the answer supported by the retrieved context (RAGAS implementation).
- Context precision: fraction of retrieved chunks that are relevant to the question.
- Answer relevance: semantic overlap between the answer and the question.
- Citation correctness: every cited chunk is retrievable, and the cited passage contains the claim.
- Refusal correctness: for the unanswerable subset, the assistant refuses or hedges rather than inventing an answer.
- Judge calibration. Against a small SME-labelled subset, measure judge-vs-human agreement. If agreement is below 70%, the judge is retuned or replaced before any score is published.
- Regression gate. The composite score, with a confidence interval from the three-run variance, becomes the gate. Pipeline changes that do not clear the gate do not ship.
The design borrows from RAGAS (reference-free metric catalogue), the Hugging Face RAG evaluation cookbook pattern (synthetic set plus LLM-as-judge), and RAGEval's grounded-generation discipline. The adversarial subset is informed by the RAGUARD finding that current RAG systems can perform worse than their zero-shot baselines when exposed to misleading retrievals.
Observations
None yet. This note is published at Hypothesis stage.
Where this fails
- Judge bias. An LLM judge inherits the biases of its base model. If judge-vs-human agreement is below 70% on the calibration subset, the harness is not shipped.
- Synthetic question narrowness. Questions generated from passages tend to be too close to the passage. Real users ask across passages, with partial names and stale terminology. We expect the synthetic score to be optimistic against real traffic and will report the gap explicitly once we have production samples.
- Reference-free faithfulness missing deep errors. Reference-free metrics can mark a plausible wrong answer as faithful if the retrieved context is itself wrong or outdated. The harness therefore has to be paired with the version-conflict detection work (see the companion note under /scrolabs) rather than treated as a complete safety net.
- Small-domain adversarials. For very narrow domains (a single pharma product's regulatory file), the number of credible adversarial questions is small, and the score becomes noisy. We will not pretend the harness is calibrated at corpus sizes below our tested range.
What delivery teams can take from this
An evaluation harness delivery pods can lift into any new Launchpad or Sherlock-backed assistant engagement in week one of the project, before any model or prompt is tuned:
- A one-page SME interview protocol for seed-question collection.
- A synthetic question generator with a groundedness filter and an unanswerable-question quota.
- A reference-free scoring suite (RAGAS metrics, citation-correctness, refusal-correctness).
- A judge-calibration script against a small SME-labelled subset.
- A CI gate that fails pipeline changes which do not clear the regression threshold.
The deliverable is a repository, not a document. It ships only if the Controlled test confirms the stability and perturbation-detection claims above.
Open questions
- Which of our live engagements will anchor the reference numbers? Decision point before Controlled test.
- Does any client require the judge to run inside their deployment boundary (which caps the judge choices)?
- What is the minimum SME time we can get away with for the seed set? Thirty minutes is the goal; we have not measured it.
- Can we reliably generate adversarial questions that are "unanswerable" without the generator leaking the answer in the question itself?
Resolving these moves this note from Hypothesis to Controlled test.
Running a grounded assistant without a golden set, and want a harness that scores regression honestly? Propose a challenge →