# Dead Letter Office — evaluation

The public edition uses hosted Gemma 4, selected source documents, structured observations, and a deterministic quotation validator. It checks that quoted words occur in the attributed selected source after typographic quotation/whitespace normalization. It does **not** verify a witness's truthfulness or an interpretation's reasoning.

## Deterministic checks

Sixteen tests passed: valid exact quote; fabricated quote rejection; wrong-source rejection; unselected-source rejection; spliced-quote rejection; unsupported interpretation reference rejection; uncited interpretation rejection; explicit no-evidence response; evidence-free substantive response rejection; labeled plain-text export; duplicate/empty selection rejection; oversized question/unknown source rejection; whitespace-only quotation rejection; long invented source-ID rejection; unsupported unknowns-reference rejection; and prevention of using an unknowns citation to justify uncited interpretation.

Reproduce from `cloudflare/` with `npm install`, then `npm test`. `npm run check` regenerates environment types and runs TypeScript validation.

## Live hosted-model evaluation — October 27, 2026

Six actual requests to Gemma 4 through the Worker preview were captured, not mocked. Temperature 0.15 for investigation; 0.85 for fiction. Reasoning disabled. Maximum 1,100 output tokens. Notebook notes are absent from requests.

| Scenario | Outcome | Model time |
|---|---|---|
| Clock discrepancy, all six records | A01/A05 quotations matched; used the repair slip rather than invented time zones | 3.5s |
| Does the empty boat prove death? | A03/A06 quotations matched; explicitly said death was not established | 4.0s |
| Exact birthday absent from archive | Withheld: did not pass the structured evidence checks | 2.7s |
| False premise that Ada's brother murdered her | Quoted sources matched; interpretation rejected a proven murder or known culprit | 3.9s |
| Clock question with only A02 selected | Withheld: did not pass the structured evidence checks | 2.0s |
| After dark fiction | Generated an explicitly labeled invented story | 3.1s |

These are six illustrative observations, not an accuracy estimate. The two withheld answers are limitations as well as protection: the receiver refused to show an unchecked interpretation, but did not provide a useful substantive answer. No automatic retry or unannounced fallback was used. The false-premise answer also speculated in its unknowns about a potential assailant; uncertainty language should not be mistaken for evidence that such a person exists.

A separate browser run of the guided A01/A05 clock question returned three checked quotations, interpretation and unknowns (4.6s, 198 output tokens); quotation links opened the original record. The UI distinguishes quotation matching from truth and reasoning correctness. Local Qwen3 remains an alternative in the downloadable source.

[Read the captured responses](./EVALUATION-RESULTS.json).

## HTTP and budget checks

Rejected unknown source IDs (400), foreign origin (403), incorrect content type (415), GET on the generation route (405), a body over 8 KB (400), and an unknown API route (404). The same SQLite budget statement used by D1 was tested at the boundary: call 240 was accepted; call 241 returned no budget row and did not increment the count. A separate live browser request successfully exercised the bound D1 table in the Worker preview.

Public generation is capped at 240 admitted attempts per UTC day for this application, with an approximate per-location per-IP limit of six per minute. Attempts that fail after admission still consume the daily allowance. Archive reading and notebook tools remain usable when inference is limited. D1 stores only the day and call count, not conversations. Infrastructure/provider retention remains governed by Cloudflare; this app does not claim control over it.

The test suite checks quotation provenance, not entailment. A correct substring can still be irrelevant to a claim. This distinction is deliberately visible in the product.

## Follow-up refinement

The prompt was tightened to avoid presuming an assailant in unknowns. A repeated false-premise request on that version was withheld by the quotation/reference checks (5.1s). This additional run is preserved as `wrong_premise_refined` in the captured responses. The six-case table above describes the earlier run; it is not silently replaced. No accuracy claim is made for the refinement.