Case study
An assistant for the first-pass assessment in pharmacovigilance. It reads an adverse event report and drafts the seriousness, expectedness and causality assessments with citations, for a human to approve. The goal is not to replace the judgment but to assemble what the judgment needs.
Try the demo →For every report that arrives, a safety team has to decide whether the company assesses it as serious, whether the reaction is already described in the product label, and whether a causal link can be ruled out — then report it to regulators within a deadline if it qualifies. The volume is high, the deadlines are short, and the evidence is unstructured text.
The domain is strongly asymmetric. An over-call costs time; an under-call is a missed report. Failing to escalate a serious, unlabelled reaction has real regulatory and clinical consequences. That makes 'when in doubt, stay on the safe side' a design principle rather than a preference.
The hardest cases are those where the deciding fact is not in the structured fields but in the narrative. Dizziness may be reported as non-serious, while the narrative describes a fall, a fracture and an admission — which makes the company assessment serious on the hospitalisation criterion. No amount of field extraction finds that.
The design was driven less by what to ask the model for, and more by what not to let it decide.
The LLM is never asked whether something is serious. It is asked which ICH E2A criteria the text positively supports, with the quote. A deterministic policy turns those into serious / needs-review / non-serious. That separation is why the safety property can be proven by test rather than hoped for: the decision layer is checked exhaustively, so no model output can produce an under-call.
Defaults land on the conservative side: needs-review for seriousness, cannot-be-excluded for causality, and not-expected unless the label match is confident. A needs-review verdict is never rounded down to non-serious — it is raised to the reviewer with its evidence attached.
A case with no own-company product is gated out and never assessed, rather than assessed with a caveat. Company assessment presupposes company scope, so producing verdicts for a case outside it is itself the error.
Not every step runs on the same model. The judgments that hinge on attributing a fact in the narrative get the stronger model; faithful transcription and coding get the cheaper one. That split came from measuring each step, not from guessing. Cost is measured on every run and shown on screen: about $0.03–0.04 and 15–18 s per case, ~85% of it the two judgments on the stronger model. An out-of-scope case stops at the gate for $0.
Running locally would keep sensitive case text off third-party APIs, so it was benchmarked properly. The finding: no local 7–8B model preserved the safety invariant — every one missed a serious event. So the fully-local tier was not shipped. What shipped is a split: transcription and coding run locally, the clinical judgments stay on the frontier model.
A code the reviewer judges medically important is promoted to a list, so later cases coded to it are flagged serious automatically. Approved cases accumulate as precedent and raise a warning when a new draft disagrees with them. Past data may only move a verdict toward the safe side, and never decides 'serious' on its own.
So an English-speaking reader can follow a case end to end, the interface, the verdict vocabulary and MedDRA terms are in English. The case text and the model's rationales stay Japanese: the pipeline and its evaluation are Japanese, and translating the rationales would add a model call and rewrite the audit evidence itself. When the screen says 'Needs review', what is sent is still 要確認 — and a test proves it.
Assessments are scored against a hand-built gold set. The headline number is not accuracy — it is that under-calls are zero, which also held across 3 repeated runs and on held-out cases.
The sample is small (7 synthetic cases; 5 in scope, 23 events), and the gold set was drafted with an LLM and corrected, from a PV perspective, by me — the same person who built the system. Some of the agreement is therefore the system agreeing with its own author — this is not an independent evaluation.
The cases are synthetic and tidier than real reports, which are messy, incomplete and inconsistently formatted. Run-to-run variation was measured on only 3 runs, with no confidence intervals.
MedDRA coding is measured on a 56-term fictional demo terminology, not on MedDRA itself, which is licensed. Real MedDRA has about 27,000 PTs, so 100% shows that retrieval plus selection works, not that this accuracy holds at full scale. The single-LLM 29% was measured while the gold set still held real codes, and it cannot be reproduced from the current repository.
'Past-case feedback improves accuracy' was not demonstrated. On held-out cases precedent changed nothing, and the one earlier improvement turned out to depend on seed data.
So the claim being supported is not 'this is accurate'. It is that the safe-side invariant holds, and that there is a repeatable procedure for measuring it.
My background is in pharmacovigilance, not web frontends or cloud infrastructure. Those parts were built with AI assistance. That is not hidden — it is part of what this project is meant to demonstrate.
It is not 'it runs, so it is fine', though. Every significant decision is recorded as an architecture decision record alongside the reproducible experiment that produced its evidence, so the reasoning can be audited rather than taken on trust.
And I do not trust code I did not write line by line. The frontend's types are checked automatically against the backend models, so drift fails a test; the safety invariants are pinned the same way. Where I cannot vouch for a layer by reading it, a machine vouches for it instead.
The UI gets the same treatment: a real browser walks the reviewer's path against backend responses recorded from the real system — free and deterministic — and checks that what is sent on approval is exactly what the real backend accepted, in both languages. After writing a test, I break the code on purpose to confirm the test actually fails.
All case reports and product labels used here are fictional data created for this project. No real patient information is involved. MedDRA is not included: coding uses a fictional demo terminology (codes DEMO-xxxx). MedDRA® is a registered trademark of ICH.