How we validate and benchmark the model

Most agent demonstrations stop at “it produced something plausible.” That is the wrong bar. If I am going to claim the system works, I need a story for what works means at each layer, and a few benchmarks I would be willing to defend in public. The short version: tooling fidelity is the floor, agentic reasoning is the middle, and scientific output quality is what matters. I lean hardest on the last layer, scoring generated briefs against expert-curated ground truth.

Three layers of evaluation

Layer 1: Tooling fidelity

Does the system retrieve and parse the correct underlying records from UniProt, NCBI, Ensembl, and Reactome? This is the mechanical floor: a held-out gene set, hand-curated canonical answers, precision and recall. It is rarely reported in agent papers, but if the tool layer is unreliable every downstream claim is unsupported. I treat it as a gate, not a headline number.

Layer 2: Agentic reasoning quality

Given a query, does the agent make defensible decisions about which tools to call, in which order, and when to stop? Best evaluated through trajectory grading: an expert scores each tool call as necessary, sufficient, or redundant, with a composite for the whole resolution path, reported as expert-agreement rates against a held-out query set. It is subjective, and I would rather under-claim here than pretend trajectory grading is fully objective.

Layer 3: Scientific output quality

This is the layer that matters most. A brief can pass Layers 1 and 2, retrieve every correct record and call every tool sensibly, and still overstate a mechanism or bury the one caveat a reviewer cares about. Layer 3 is where that gets caught.

Verifying output against ground truth

Expert-curated reference set

Build a gene-brief gold-standard set: roughly 30 well-characterized genes across AD (APOE, MAPT, TREM2, APP, PSEN1), PD (SNCA, LRRK2, GBA, PRKN, PINK1), ALS (SOD1, TARDBP, C9orf72, FUS), and a non-neuro control set. For each, an expert-curated reference brief fixes the canonical facts; outputs are then scored claim by claim for accuracy, completeness, citation grounding, and false-claim rate. A missed fact is a recall problem; a confidently asserted false one is a different and worse failure, scored separately. Where a brief makes a computable claim, the expert recomputes it independently and we report the correlation between system and expert values.

Negative controls

A negative control is the most informative test here, and the one most demonstrations skip. The system is run against an adversarial input battery: fictional gene symbols, ambiguous queries, real genes paired with false-premise context (“describe SNCA’s role in glycolysis”), and queries about superseded findings. The benchmark is not accuracy on these inputs; it is appropriate-rejection rate. Does the system flag the problem, or produce a confidently wrong brief.

Calibration

When the system expresses confidence, whether implicit in tone or explicit if instrumented, does that confidence track actual accuracy? Once expert-scored outputs exist, a reliability diagram is straightforward. I expect the first version to be poorly calibrated, and naming that up front is the honest position.

What the network uniquely enables

The rate-the-edge feedback loop on the cross-disease network is itself a benchmarking instrument. The graph now carries 311 nodes and 474 edges across 11 disease groups, and the node set is one-to-one with the trainable model panel, so anything you see in the graph is something you can train and test. Each edge is a model-proposed gene–gene relationship anchored by the PubMed citations the brief surfaces, and each user rating is a domain-expert judgement on that edge. Three analyses follow: inter-rater agreement (do independent experts converge on which edges are real?), model–expert agreement (does the system’s edge confidence track expert ratings?), and held-out enrichment (are highly rated edges enriched for interactions independently validated in STRING, BioGRID, or curated pathways?). Over time the loop produces a labeled dataset of expert-judged biological edges, a resource with publishable value in its own right.

The Layer-3 scoring rubric

Each brief in the gold-standard evaluation is scored on four dimensions. The same four are the knobs the training loop reweights, so the rubric the expert applies and the rubric the model optimizes against are the same object.

One design note worth stating plainly: the mock brief pool penalizes evasiveness, not falsehood. The low-quality variants hedge and omit identifiers but are never made factually wrong, because a deliberately incorrect brief would turn the training set into a misinformation source.

Two benchmarks to prioritize first

Benchmark A: Gene-brief gold-standard evaluation

Roughly 30 expert-curated reference briefs across AD, PD, ALS, and a non-neuro control set, scored on factual accuracy, recall of canonical facts, false-claim rate, and citation grounding. Lightweight, presentable, and a defensible foundation for any further claim. Build it first because it is the cheapest credible thing, and the gold-standard briefs double as new few-shot exemplars.

Benchmark B: Adversarial rejection

Fictional-symbol inputs, ambiguous queries, false-premise queries, and outdated-finding queries, scored on appropriate-rejection rate and willingness to flag uncertainty. The test most aligned with the field’s stated needs, and the one I would most want to put on a public leaderboard. It is also the one I expect the current system to do worst on, which is exactly why it is worth reporting.

What could undermine these numbers