← caelith labs
artefact · 03 · 03
eval suite + audit trail — spec.
Agent build engagement · v0.1 · 6 sections
Every Agent Build engagement ships with an eval suite that gates production deployment and an audit trail that captures every agent decision. This is the spec. Real engagement deliverables include named test cases, named eval thresholds, and engagement-specific guardrails — adapted from the structure below.
§ 1what gets evaluated
Three things get evaluated per build, in order of binding-ness:
- Correctness — does the agent do what it's supposed to do, on a representative test set?
- Safety / guardrails — does it refuse out-of-scope requests and stay inside policy boundaries?
- Cost + latency — does it meet the per-call cost ceiling and the p95 latency budget?
An eval failure on any of the three blocks production deployment. Correctness fails are the most common; safety fails are the most expensive; cost/latency fails are the most ignored (until they aren't).
§ 2test set construction
how we build it
- ~30–60 input cases per agent, hand-curated with the Controller during engagement Week 1
- Drawn from three sources: real historical inputs (anonymized), edge cases the Controller has actually hit, and adversarial cases (off-topic, malformed, jailbreak attempts)
- Each case carries a known good output OR a known property (e.g., "must include X", "must not include Y", "must be ≤200 tokens")
- Test set committed to the engagement repo at
evals/<agent-name>/cases.jsonl
example case structure
{
"id": "case-042",
"input": "Can you summarise the attached invoice and extract the VAT number?",
"expected_properties": [
"contains a VAT number in DE format",
"does not invent line items",
"≤200 tokens"
],
"category": "core",
"regression_for": null
}
§ 3scoring rubric
| Dimension | Threshold | Method |
|---|
| Correctness | ≥90% pass | Per-case property check (deterministic where possible, LLM-as-judge where not) |
| Safety | 100% refusal on adversarial set | Hand-graded by Processor + Controller jointly during W3 |
| Cost | ≤€X per 1k calls (set per engagement) | Aggregated from Anthropic billing API over a 24h smoke run |
| Latency | p95 ≤Xs (set per engagement) | Aggregated from agent's own request log |
Thresholds are set during W1 architecture and signed off in the engagement proposal. If a threshold can't be met, we re-spec before continuing — not paper over.
§ 4audit trail
Every agent invocation in production writes a record to the engagement's audit log:
- Timestamp + request ID (correlation across the call chain)
- Input (or a hash of input if sensitive)
- Model + version (Claude Sonnet 4.5, etc.)
- Tools called + their responses
- Output
- Token usage + cost
- Latency
- Any guardrail trips
Default storage: PostgreSQL on the Controller's stack, with 365-day retention (configurable). If the engagement uses Caelith Labs's hosting, retention is identical and data is returned/deleted per the DPA § 9.
note on hash-of-input
For engagements touching end-customer PII (DATEV bookkeeping, Holded ERP, etc.), inputs are hashed before storage and the original is not retained. The hash is enough to detect "this exact input was seen before" without storing the PII itself. Adds ~5% to per-call latency.
§ 5regression cadence
The eval suite runs on three triggers:
- Pre-deployment — every code change to the agent must pass the full eval suite before merge. Default: GitHub Actions on PR.
- Weekly — full eval re-run against the live model (Anthropic ships model updates; what passed yesterday might fail today). Default: scheduled at 03:00 UTC Sunday.
- On user report — if a Controller's team flags a bad agent response, the input becomes a new test case (with the expected behaviour). Test set grows over time.
Weekly re-runs cost ~€2–8 per agent depending on test set size and model. Cheap insurance.
§ 6what the Controller gets at handover
- The eval suite (cases + scoring rubric + run scripts) in the engagement repo, owned by the Controller
- A baseline run report (correctness/safety/cost/latency at handover)
- GitHub Actions workflow for pre-deployment eval gate
- Weekly scheduled eval workflow + alerting on threshold breach (email or Slack — Controller's choice)
- Audit-log table schema + dashboard query (Metabase / similar — Controller's choice)
- Section 4 of the engagement runbook explains how to add a new test case and how to read the audit log
The Controller runs evals themselves after handover. If they want Caelith Labs to keep running them, that's the optional retainer (€1,500–€2,500/month).