EntonBench

A benchmark of open-ended quantitative trading research tasks, graded by a deterministic verifier — not a judge model. A model passes only when the artifact it produces — a gate, a parameter cell, a repaired or newly created strategy template, a reconstructed feature set — survives deterministic out-of-sample backtest verification against committed reference data.

Rankings come exclusively from hidden holdout tasks the public has never seen. The same verification ladder that grades this benchmark governs ENTON's internal live-capital program.

Leaderboard

2026-07-06 · 3 independent repeats per task · ranked by the hidden holdout set only — the public split is a transparency column.

Hidden holdout set — determines rank

Gemini 3 Flash (preview)
50%3/6
Gemini 2.5 Flash
50%3/6
Gemini 3.1 Flash Lite
50%3/6
GPT-5.5†
33%2/6
Claude Opus 4.7†
0%0/6
0%25%50%75%100%
Hidden holdout (ranking)Public split (transparency)

Gemini 3 Flash (preview)

#1

teacher/judge model for the EntonBench program

Hidden
50%3/6
Public
82%37/45
verifier_failedmissing_expected_failure_modedecision_mismatch

Gemini 2.5 Flash

#2

student baseline row (ticket-44 handoff: ranks as the SFT-program base)

Hidden
50%3/6
Public
73%33/45
verifier_faileddecision_mismatchagent_loop_no_submit

Gemini 3.1 Flash Lite

#3
Hidden
50%3/6
Public
73%33/45
verifier_failedagent_loop_no_submitunexpected_failure_mode

GPT-5.5†

#4
Hidden
33%2/6
Public
72%28/39
verifier_faileddecision_mismatch

Claude Opus 4.7†

#5
Hidden
0%0/6
Public
77%30/39
verifier_failed

† Preliminary — access configuration for these rows is being standardized; figures update as they finalize.

View as data table
ModelHidden setPublic splitTop failure axes
Gemini 3 Flash (preview)50% (3/6)82% (37/45)verifier_failed, missing_expected_failure_mode, decision_mismatch
Gemini 2.5 Flash50% (3/6)73% (33/45)verifier_failed, decision_mismatch, agent_loop_no_submit
Gemini 3.1 Flash Lite50% (3/6)73% (33/45)verifier_failed, agent_loop_no_submit, unexpected_failure_mode
GPT-5.5†33% (2/6)72% (28/39)verifier_failed, decision_mismatch
Claude Opus 4.7†0% (0/6)77% (30/39)verifier_failed
  • Ranking uses only the hidden set. Published tasks are assumed to enter training corpora eventually — by design. The public split exists for transparency, not ranking.
  • The hidden-set rule has already caught contamination in practice. In our own tuning experiments, a model trained on task-family data tied the leader on the public split while collapsing on hidden siblings — exactly the answer-transfer signature this ranking rule exists to catch (see the program note below).
  • No model cracks the hidden set. The harder hidden sibling — a 497-cell parameter-neighborhood robustness task — has defeated every model in every round to date. It marks the benchmark's current ceiling.
  • verifier_failed dominates every failure taxonomy. The artifact didn't survive deterministic recompute. The bench measures research execution, not formatting.

The honest-rejection exhibit

The hardest task class asks a model to create a strategy from a research brief, run it through a real backtest engine, read its own results, and decide honestly. This step-through follows the committed trajectory of a frontier model doing exactly that.

Step 1 of 8

The brief

The task hands the model a research brief: create a midday compression→expansion strategy template for NQ futures, from scratch, against the engine's strict template contract.

How it works

  1. 01Every task ships a manifest: objective, allowed and forbidden files, committed evidence, an output contract, and a verifier command composed of deterministic recompute stages — TypeScript compilation, template-contract checks, smoke-backtest parity against byte-pinned tick fixtures, full factory runs, trajectory metric thresholds, feature-cache matching.
  2. 02Models attempt tasks through two runners: single-shot (one structured response) and agent-loop (write code → run → read verifier feedback → iterate → submit) with a hard-policed tool set. Verifier failure output is redacted so it cannot be used as an answer oracle.
  3. 03Grading is adversarial by construction: snapshot diffs around every attempt hard-zero out-of-scope changes, fixtures are sha-pinned, and results use a failure-mode vocabulary instead of a bare pass/fail.
  4. 04Rewards are denominated in dollars because backtest ledgers are. Every figure is a historical out-of-sample recompute — never a live result, never a forward-looking claim.

The DaVinci program

EntonBench doubles as the reward surface for ENTON's verifier-graded research-model program: models trained on verified research trajectories, released only through pre-registered evaluation gates. The current iteration already beats its base model decisively on open-ended research tasks (12/16 vs 7/16) and is held back by its gate until it generalizes to unseen task splits — a DaVinci row joins this leaderboard when a version passes. The gates are real because their verdicts are published either way.

Limitations

  • The hidden set is currently two sibling tasks (× 3 repeats): ranking resolution is coarse. Hidden siblings for the template-repair, template-creation, and feature families are a named follow-up.
  • No accept exemplar exists yet — no live-verified GO has occurred — so over-rejection is unmeasurable until one exists. We will not fabricate one.
  • A pass is not profitability, returns, alpha, or any forward-looking performance claim. Dollar figures are historical out-of-sample recompute units. Nothing here is investment advice.

Reproduce this

The public split runs from a fresh clone with a single API key: install, run the verifier suite, then attempt tasks live through the included single-shot and agent-loop runners. The public repository opens after final review — tasks under CC BY-NC 4.0, harness under MIT. Hidden holdout tasks and all live-trading data are permanently excluded from the public repository.