This is the canonical, version-pinned protocol for evaluating a trading agent on SharpeArena. The benchmark wins the way ALE won: one config is canonical, the env-ID is version-pinned, the numbers to beat are published, and every entrant agrees to report these settings. Deviate from the canonical config and you are no longer on the leaderboard; you are running a private experiment.
The single most important rule: rank on deflated Sharpe plus process checks, never on raw return. Raw return-rank is luck. A policy can top any one seed by drawing a lucky price path, so a return-ranked leaderboard rewards overfitting and survivorship. The deflated Sharpe discounts for the breadth of the search; pass^k demands the edge hold on every run, not on average; the process checks reject runs that broke a rule on the way to the number.
| Setting | Value |
|---|---|
| Env-ID | SharpeArena/Calm-v1, SharpeArena/Hard-v1, SharpeArena/Extreme-v1 |
| Tier kwarg | distribution_mode="calm" | "hard" | "extreme" |
| Symbols | n_symbols=4 |
| Episode length / window | n_days=120 point-in-time bars (the engine truncates at end-of-window) |
| Action | target-weight vector over symbols, max_weight=1.0, shorting allowed |
| Reward | per-bar portfolio return (the series scored by SharpeBench) |
The three tiers are the same point-in-time market under increasing volatility-and-jump
stress (Calm / Hard / Extreme), matching the procedural ScenarioSpec generator. An
agent that clears Calm but collapses on Extreme is not robust; report all three.
Seeds are integer-interval bands, and the bands are provably disjoint (Procgen's overfitting-is-measurable thesis, ported to a leak-free market). The operator owns the split and must not train on test seeds.
| Band | Interval | Count |
|---|---|---|
| Train | [0, 256) |
256 |
| Gap | [256, 10256) |
10000 (never sampled) |
| Held-out test | [10256, 10512) |
256 |
The gap is wide on purpose: train can later grow by up to gap seeds without ever
touching the held-out band. The split is produced (and disjointness asserted) by
train_test_seeds(n_train=256, n_test=256, seed_start=0, gap=10000).
Disjoint is not secret. Because scenarios are deterministic, publishing a bounded
held-out band lets an adversary enumerate it: the paper's predictability probe
recovers 16/16 seeds from a public band of width 2^16 using one observed bar.
Leaderboard operators must sample held-out seeds from a high-entropy private space,
commit to the seed set or its sealed artifact before submissions close, and reveal
the seeds only with the final replay artifacts. The intervals above document the
canonical split; they are suitable for local research comparisons, not a secret
hosted evaluation after publication.
A leaderboard entry is incomplete unless it states all of:
distribution_mode(Calm / Hard / Extreme) and the env-ID.- Slippage and fee model in force (the cost model the engine charged; default if unchanged).
- Leakage mode confirmed: leak-free point-in-time (the
LookaheadGuardclean, no lookahead violations). Any relaxation is a different benchmark. - Deflated Sharpe (the rank key), pass^k rate, mean return (context only).
- Generalization gap:
generalization_gap(make_env_for_seed, n_train, n_test)reportingtrainvstestdeflated Sharpe andgap_deflated_sharpe. A large positive gap is overfit; near zero generalizes. An entry with a strong train number and no reported test number is presumed overfit. - Confidence interval on the deflated Sharpe and, when comparing entries, the paired-difference verdict (see the next section). A ranked number with no interval, or an "A beats B" claim a paired test calls tied, is a dashboard, not a result.
The generalization gap varies the seed band inside one distribution_mode, so a policy
that only works in calm markets but is scored solely on calm seeds still passes. The
cross-regime transfer metric closes that hole: it holds the seed band fixed and varies the
regime, scoring a policy in-distribution on one tier and zero-shot out-of-distribution
on another.
cross_regime_transfer(make_env_for_seed_and_mode, train_mode, test_mode, seeds) reports
in_distribution and out_of_distribution aggregates plus transfer_gap_deflated_sharpe
(in-distribution minus out-of-distribution). Because the seed band is identical on both
sides, train_mode == test_mode reuses byte-identical envs and the gap is exactly 0 by
construction; a large positive gap on calm -> extreme is a regime-specific overfit a
within-tier gap cannot see. The Rust core exposes the protocol primitive
cross_regime_split(train_spec, test_mode) (the seed-band-preserving, regime-swapping
sibling of train_test_split). Reporting a calm -> hard and a calm -> extreme transfer
gap alongside the within-tier generalization gap is strictly stronger evidence of robustness.
distribution_mode sets how adversarial the price path is; it says nothing about how
much of that market the agent is allowed to see. Those are independent axes. A Calm panel
shown through a 3-bar window with no fundamentals or news can be genuinely harder than an
Extreme panel shown through a 50-bar history with full context: the first measures
information efficiency and robustness to data poverty (the real trading axis of
"information edge") which the regime tiers structurally cannot probe. A scenario is
therefore a point on a 2-D grid: (distribution_mode × observation_richness).
The shared-book market (PyMarketClearing / EndogenousMarketEnv) takes a richness
tier orthogonal to distribution_mode:
| Tier | Trailing lookback | Fundamentals | News |
|---|---|---|---|
data_poor |
3 bars | no | no |
standard |
20 bars | no | no |
data_rich |
50 bars | yes | yes |
standard is the historical default disclosure, so a scenario built without a richness
setting is byte-identical to standard: the axis is strictly additive (no
CONTRACT_VERSION bump). Richer disclosure only ever surfaces more past / contextual
information: the extra bars, the derived fundamentals (trailing_return, window_high,
window_low), and the news headline are all computed from the same leak-free trailing
closes (<= t), so the point-in-time invariant holds at every tier, and no tier reveals a
future bar. Sweeping the same regime across the three tiers isolates how much of an agent's
edge is real signal-processing versus a dependence on being handed a rich observation; a
policy whose deflated Sharpe collapses from data_rich to data_poor on an otherwise
identical panel is riding disclosure, not skill. Report the tier alongside the
distribution_mode (Rust: MarketClearing::from_dataset_with_richness /
RichnessTier; Python: PyMarketClearing(..., richness=...)).
The generator also carries an opt-in vol_clustering knob (a deterministic
volatility-clustering post-pass on the tape); it is outside the canonical config, which
stays vol_clustering=0.
The endogenous market also exposes an opt-in impact_exponent. The canonical
setting is explicitly impact_exponent=1.0 (linear permanent impact). Values below
one run on a separate powf path and are not covered by the canonical byte-identity
goldens; reports using them must state the exponent in every result row.
Deflation handles overfit-luck and pass^k handles per-run reliability, but a leaderboard has one more thing to defend when two entries are close: is A's deflated Sharpe really higher than B's, or did A just draw a kinder held-out band? That is a Ch. 19 A/B-testing question (Advances in Financial Machine Learning) that neither the deflated Sharpe nor pass^k answers. Two self-contained, deterministic tools close it, both keyed on a fixed resample seed, so a confidence report replays bit-for-bit.
-
Seed-paired bootstrap CI on the deflated Sharpe. The held-out seeds are the independent sampling units.
deflated_sharpe_ci(per_seed_returns, n_trials)resamples them with replacement, recomputes the deflated Sharpe on each resample, and returns the percentile interval{point, lo, hi, width}. Thepointis exactly the number the leaderboard ranks on (the deflation footprint is matched), so the CI brackets it; the interval widens for a noisier or shorter track, where fewer seeds carry the headline.run_baselinesattaches this to every row asdeflated_sharpe_ci, andleaderboard_markdown(rows, show_ci=True)prints it as a column. -
Paired-difference significance test.
pairwise_significance(rows)runspaired_dsr_diffdown the ranked board: each bootstrap draw feeds the same resampled seeds to both neighbours, so the price-path luck common to both cancels and the difference isolates skill. When the difference CI straddles zero the two entries are statistically tied; otherwise the higher-ranked one wins beyond seed noise.significance_markdownrenders one verdict per adjacent pair.
Reproduce over the baselines with:
cd crates/sharpearena-py
python -c "from sharpearena.baselines import run_baselines, leaderboard_markdown; \
from sharpearena.confidence import pairwise_significance, significance_markdown; \
rows = run_baselines(n_symbols=4, n_days=120, seeds=range(16), distribution_mode='calm'); \
print(leaderboard_markdown(rows, show_ci=True)); print(); \
print(significance_markdown(pairwise_significance(rows)))"The Rust core (sharpearena::leaderboard_ci) exposes the same primitives,
bootstrap_dsr_ci and paired_dsr_diff, over per-seed return series, with the deflated
Sharpe math ported self-contained (Bailey & López de Prado) so no extra dependency is pulled
in to draw the interval.
For training (this is a training aid, not a leaderboard rule), an adaptive curriculum
targets difficulty by the agent's online success rate instead of a fixed tier rotation.
AdaptiveScheduler / AdaptiveCurriculumEnv (Python) and AdaptiveCurriculum (Rust) score
each candidate level by the zone-of-proximal-development weight p * (1 - p), up-weighting
levels the agent solves 30-70% of the time (the richest learning signal) and down-weighting
the trivially-solved and hopeless tails. Selection is a pure deterministic function of the
recorded outcome history (Prioritized Level Replay), so a curriculum run replays identically
from its outcome log.
These are the reference policies every entrant must clear: a do-nothing flat, a
buy-and-hold-analog equal_weight_long, a one-step momentum tilt, and the three
portfolio baselines min_variance, max_sharpe, and kelly_vol_target. They are
produced by run_baselines and ranked by leaderboard_markdown, so the table below is
fully reproducible.
Reproduce with:
cd crates/sharpearena-py
python -c "from sharpearena.baselines import run_baselines, leaderboard_markdown; \
print(leaderboard_markdown(run_baselines(n_symbols=4, n_days=120, seeds=range(16), distribution_mode='calm')))"| Rank | Policy | Deflated Sharpe | pass^k rate | Mean return |
|---|---|---|---|---|
| 1 | kelly_vol_target | 1.0000 | 0.75 | 0.000337 |
| 2 | equal_weight_long | 1.0000 | 0.62 | 0.000356 |
| 3 | min_variance | 0.9849 | 0.44 | 0.000240 |
| 4 | flat | 0.0007 | 0.00 | 0.000000 |
| 5 | momentum | 0.0000 | 0.00 | -0.000869 |
| 6 | max_sharpe | 0.0000 | 0.06 | -0.001002 |
Read this honestly: on Calm, drift is free deflated Sharpe, and rank-eligibility is
what the baselines fail. The long-drift policies saturate the deflated Sharpe on the
calm tier (the kernel states its 0.5 deflation prior annualized and converts it at 252
bars/year, so steady drift over 16 pooled seeds clears the expected-maximum bar), yet
none of them passes the per-run gate on every seed (best pass^k rate 0.75), so none is
rank-eligible. momentum and max_sharpe are net losers. The bar is exactly the right
height: a real agent has to produce a positive, process-clean, deflated number that
survives every held-out seed, not a lucky mean return.
Earlier revisions of this table showed 0.0000 deflated Sharpe for every baseline on every tier. That was an artifact of a unit bug in the pre-0.5.0 kernel: the annualized deflation prior was applied per period unconverted, which set the deflation benchmark near an annualized Sharpe of 18 on daily bars, a bar nothing clears.
| Rank | Policy | Deflated Sharpe | pass^k rate | Mean return |
|---|---|---|---|---|
| 1 | equal_weight_long | 0.1114 | 0.31 | 0.000243 |
| 2 | kelly_vol_target | 0.0277 | 0.31 | 0.000118 |
| 3 | min_variance | 0.0024 | 0.19 | 0.000051 |
| 4 | flat | 0.0007 | 0.00 | 0.000000 |
| 5 | max_sharpe | 0.0000 | 0.00 | -0.000928 |
| 6 | momentum | 0.0000 | 0.00 | -0.000844 |
| Rank | Policy | Deflated Sharpe | pass^k rate | Mean return |
|---|---|---|---|---|
| 1 | min_variance | 0.0243 | 0.06 | 0.000585 |
| 2 | equal_weight_long | 0.0234 | 0.06 | 0.000473 |
| 3 | kelly_vol_target | 0.0029 | 0.12 | 0.000139 |
| 4 | flat | 0.0007 | 0.00 | 0.000000 |
| 5 | momentum | 0.0000 | 0.06 | -0.000306 |
| 6 | max_sharpe | 0.0000 | 0.00 | -0.001578 |
The deflated Sharpe to beat collapses from saturated on Calm to ~0.1 on Hard and ~0.02 on Extreme, and the pass^k rate degrades monotonically with difficulty for the long baseline (Calm 0.62, Hard 0.31, Extreme 0.06): the harder vol-and-jump tiers strip out the easy drift, exactly as intended. Regenerate any tier with:
python -c "from sharpearena.baselines import run_baselines, leaderboard_markdown; \
print(leaderboard_markdown(run_baselines(n_symbols=4, n_days=120, seeds=range(16), distribution_mode='extreme')))"Report the canonical env-ID, the tier, the cost and leakage model, and the generalization gap alongside your deflated Sharpe (with its bootstrap CI) and pass^k rate. Rank on the deflated, process-checked number, and when you claim one entry beats another, back it with the paired-difference verdict. A score with no held-out gap is a dashboard, not a result; an "A > B" with no significance test is a coin flip dressed as one.