How we benchmark, in full.

What's published vs what's not

The suites you see here are a curated subset of NOPE's full benchmark corpus. The full corpus contains 126 test suites and 3,271 crisis-shaped conversations; we publish 17 of them. The selection is biased toward:

• Methodologically clean fixtures with peer-reviewed clinical sources.
• Demographic representation: youth, LGBTQ+, Indigenous populations, elder care.
• A deliberate mix of wins and losses — suites where NOPE leads, and suites where it doesn't.
• Public-research-derived cases, not partner-tranche or operator-flagged incident data.

Within each published suite, the first 8 cases are kept verbatim. The remaining cases show only their case_id, description, and a minimal per-model pass/fail map — the full conversation and rich model output are redacted. The aggregate F1, precision, and recall numbers are computed on the full case set in the original run — they remain honest; only the per-case detail is truncated.

This policy ("holdout vibes") protects the benchmark's integrity over time: if every case is public, comparator models can implicitly tune against them, and the benchmark stops measuring real generalization. The methodology, on the other hand, is fully open below.

Talk to us if you want methodology access for due diligence, a regulator review, or partnership-level deep-dive on the full corpus.

F1, precision, recall

All comparator metrics on this site are flag-level binary. Each case has an expected_flag — whether a safety layer should flag this conversation for follow-up — and each model returns flagged: true | false. Confusion matrix terms:

• TP — model flags, case is a real crisis.
• FP — model flags, case is benign.
• TN — model doesn't flag, case is benign.
• FN — model doesn't flag, case is a real crisis.

Precision = TP / (TP + FP). When the model flags, how often is it right?
Recall = TP / (TP + FN). Of the real crises, how many did the model catch?
F1 = 2 × (P × R) / (P + R). The harmonic mean — a single number you can sort on.

The headline Accuracy column on this site uses a different definition: it bakes in severity-band correctness (whether the model returned "moderate" when expected was "moderate"). On suites where severity is fine-grained, Accuracy and F1 can diverge by 30+ percentage points. The F1 / precision / recall numbers are the comparator-fair ones to cite.

Latency

Latency numbers cited on this site (and on nope.net) are p50, end-to-end, measured from the test runner over public internet. Each adapter is hit from the same machine with the same network path; numbers are not network-isolated. The runner uses concurrency 20 by default, with rate-limited adapters auto-throttled to 5 (Azure, LlamaGuard, gpt-oss-safeguard) with 2/5/10-second backoff on HTTP 429.

NOPE's own production endpoints (/v1/evaluate, /v1/ocular) include the inbound API key check, the model call, and any post-processing — not just the raw model latency.

Suite selection criteria

The 17 suites on this site were chosen on:

1. Clinical methodology grounding — citations to C-SSRS, HCR-20, DASH, peer-reviewed research with DOIs. We exclude suites built on operator-flagged incidents or unattributed corpora.
2. Population coverage — both crisis-detection (where a missed flag is the dangerous error) and false-positive prevention (where over-flagging erodes trust). Both directions are represented.
3. Comparator parity — preference for suites where each comparator was called via its standard API with its standard interpretation. No comparator was specifically degraded for these suites.
4. Intellectual honesty — the published set deliberately includes suites where NOPE doesn't lead (e.g. indigenous-crisis-patterns, algospeak-obfuscation, kms-hyperbole-calibration). A board where NOPE wins every row would read as cherry-picked.
5. License compatibility — published cases are either NOPE-synthetic (research-informed vignettes), Apache-licensed (PIPPA), or derived from cited public research. Reddit-derived, partner-tranche, and operator-incident-derived corpora stay internal.

Per-suite sources

Each suite's result JSON includes a sources field with full citations (DOIs and URLs where available). Open any suite from the home page to see the source list inline. The same sources are also stamped into the fixture files themselves; on request we can share the fixture JSONs (which contain ground-truth labels and per-case rationale).

Methodology non-claims

NOPE's classifiers are not predictive, not diagnostic, not therapeutic, and not a replacement for clinical judgment. Scores reflect what is present in the conversation — they do not forecast outcomes or replace a clinical assessment. The benchmark measures detection of crisis-shaped signal, not prediction of crisis-shaped events.