diff --git a/README.md b/README.md index c2ed711..7ad5d62 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ The strategic bet is **interface ownership**: if every trading agent in the open ## Status: published, active (pre-1.0) -Published to **crates.io**, **npm**, and **PyPI** at **v0.1.0**, depending on the **published** `sharpebench-sim 0.0.7` engine (not a vendored copy). CI is green across four surfaces: Rust (`fmt`, `clippy -D warnings`, tests, a WASM target build), `cargo-deny`, the npm package, and the Python wheel (`maturin` + `pytest`). +Published to **crates.io**, **npm**, and **PyPI** at **v0.4.0**, depending on the **published** `sharpebench-sim 0.0.7` engine (not a vendored copy). CI is green across four surfaces: Rust (`fmt`, `clippy -D warnings`, tests, a WASM target build), `cargo-deny`, the npm package, and the Python wheel (`maturin` + `pytest`). Beyond the core `reset`/`step` lifecycle, the environment now ships a full **reinforcement-learning training surface**: @@ -45,7 +45,12 @@ Beyond the core `reset`/`step` lifecycle, the environment now ships a full **rei | **Procedural scenarios** | A seeded generator (`ScenarioSpec` / `generate_scenario`) using Procgen's integer-seed-interval model, with `Calm` / `Hard` / `Extreme` volatility-and-jump tiers and provably disjoint `train_test_split`, cross-runtime golden-hashed for byte-identical generation. | | **Generalization gap** | `generalization_gap` measures train-vs-held-out deflated Sharpe over disjoint seed bands, turning "did it overfit" into one number scored by the SharpeBench kernel. | | **`verifiers` training env** | A PrimeIntellect `verifiers` `MultiTurnEnv` that steps the market bar-by-bar, over a multi-row scenario `Dataset`, with an `XMLParser` decision protocol and a GRPO-safe bounded reward scored by the real SharpeBench `score_run` (deflated Sharpe, pass^k, process checks). | -| **Vectorized rollouts** | `VecTradingEnv` runs B scenario lanes in lockstep (rayon, structure-of-arrays JSON, current-Gymnasium `AutoresetMode`), exposed as a `gymnasium.vector` env. | +| **Pluggable reward schemes** | `load_environment(reward_scheme=...)` selects from a registry: an online differential Sharpe (Moody-Saffell, aligns the training signal with the deflated-Sharpe score), Sortino, drawdown-penalized, turnover-penalized, loss-averse. Schemes shape *training only*; the rank key stays the SharpeBench kernel. | +| **More env tasks** | Beyond the position env: a `PortfolioEnv` (simplex allocation, log-return), an `ExecutionEnv` (VWAP/TWAP implementation-shortfall MDP), and a `MarketMakingEnv` (Avellaneda-Stoikov) that ships its **closed-form analytical-optimal policy** as a baseline, so agents are scored on regret-vs-provably-optimal. | +| **Scenario families** | Calm/Hard/Extreme vol-jump tiers plus `CointegratedPairs` (genuine mean-reverting spread), `RegimeShift` (trend-to-whipsaw), curriculum chaining (`CurriculumEnv`/`regime_curriculum`), and a frozen named held-out eval-seed regression set. | +| **Rich observations** | Opt-in, causal, leak-free obs augmentations computed from the point-in-time history: technical indicators (RSI/MACD/Bollinger/...), multi-timescale momentum, rolling covariance, a spread z-score, a synthetic seed-derived news/sentiment channel, and time-to-horizon. Declarative via `PreprocessingConfig`. | +| **Risk + eval axes** | Drawdown stop-out, turbulence-halt, and liquidation-cascade wrappers; a forecast-quality calibrated eval axis (FinPILOT), per-regime breakdown, efficient-frontier/Kelly baselines, and a full risk/profit metrics panel (Calmar/Sortino/VaR/CVaR/tail/turnover). | +| **Vectorized rollouts** | `VecTradingEnv` runs B scenario lanes in lockstep (rayon, structure-of-arrays JSON, current-Gymnasium `AutoresetMode`, async `send`/`recv`), exposed as a `gymnasium.vector` env. | | **Point-in-time-safe wrappers** | Causal normalize (no future-bar leak), `TimeLimit`, `FrameStack`, `RecordEpisodeStatistics`, vector-env variants, and `flatten`/`unflatten` Dict-obs helpers, plus a `check_env` conformance harness that *proves* seed-determinism (and adopts Gymnasium's own `check_env`). | | **Gymnasium registration** | Versioned, namespaced IDs: `gymnasium.make("OpenOutcry/Hard-v1")` and `make_vec(...)` route to the scalar and vector envs, with `-Eval-v1` variants on a disjoint held-out seed band. | | **Multi-agent markets** | A PettingZoo `MultiAgentOpenOutcryEnv` (batched competition: N agents on one frozen scenario, SharpeBench-ranked), and `EndogenousMarketEnv`, a real shared-book market where aggregate flow *moves* the cleared price (Kyle permanent + Almgren-Chriss temporary impact). | diff --git a/crates/openoutcry-py/python/openoutcry/__init__.py b/crates/openoutcry-py/python/openoutcry/__init__.py index 9b4ce64..cecde95 100644 --- a/crates/openoutcry-py/python/openoutcry/__init__.py +++ b/crates/openoutcry-py/python/openoutcry/__init__.py @@ -92,6 +92,13 @@ sample_block_window, make_block_env, ) +from .cascade import LiquidationCascadeEnv, cascade_survived, cascade_summary +from .reward_misspecification import ( + MISSPECIFIED_REWARDS, + MISSPECIFIED_PROXY_POLICIES, + misspecification_gap, + demonstrate_punishment, +) from .minari_export import to_minari, to_minari_train_test from .pettingzoo_env import MultiAgentOpenOutcryEnv, make_aec_env from .market_env import EndogenousMarketEnv @@ -227,6 +234,13 @@ "block_windows", "sample_block_window", "make_block_env", + "LiquidationCascadeEnv", + "cascade_survived", + "cascade_summary", + "MISSPECIFIED_REWARDS", + "MISSPECIFIED_PROXY_POLICIES", + "misspecification_gap", + "demonstrate_punishment", "register_envs", ] __version__ = "0.4.0" diff --git a/crates/openoutcry-py/python/openoutcry/cascade.py b/crates/openoutcry-py/python/openoutcry/cascade.py new file mode 100644 index 0000000..2bc397b --- /dev/null +++ b/crates/openoutcry-py/python/openoutcry/cascade.py @@ -0,0 +1,178 @@ +"""Forced-liquidation cascade scenario wrapper over :class:`~openoutcry.gym.OpenOutcryEnv`. + +A drawdown that breaches the maintenance-margin floor mid-episode triggers a *cascade*: a +margin call forces a position reduction, the forced selling depresses the mark, the lower +mark deepens the equity deficit, and the chain repeats. :class:`LiquidationCascadeEnv` +models that loop as a **deterministic, bounded, in-step** projection — not a generic +event-driven engine — so it stays reproducible and leak-free while still emitting the +structured events (``margin_call`` -> ``forced_reduce`` -> ``cascade_impact``) that the +verifiers ``process_check_reward`` / ``mandate_reward`` machinery already consume from +``info["events"]``. + +The cascade is a pure function of the realized post-step NAV, the running peak, and the +three parameters — no RNG, no future bars — so the same path yields byte-identical events. +It is bounded by ``cascade_steps``: each step compounds one ``impact_per_step`` mark drop, +and the chain stops early only when equity is wiped. + +terminated vs truncated (mirroring :class:`~openoutcry.risk.DrawdownStopper`): + +* a cascade that wipes equity (``final_nav <= 0``) is **terminated** — bankruptcy is an + absorbing MDP state, there is no future to bootstrap past. +* a cascade that the position survives (``final_nav > 0``) is **truncated** — a margin + stop-out is an episode *cut*, not an absorbing state; the value estimate should bootstrap + past it. + +While liquidating, the action handed to the underlying env is overridden to flat (zeros): +the operator's position is being force-reduced, so the agent cannot re-lever into the +breach until equity recovers above the margin floor. +""" + +from __future__ import annotations + +from typing import Any, Optional + +import numpy as np +import gymnasium as gym + + +def _run_cascade( + nav: float, + peak: float, + *, + maintenance_margin: float, + cascade_steps: int, + impact_per_step: float, +) -> tuple[list[dict[str, Any]], float]: + """The deterministic cascade chain for a breached step. Pure: events + final NAV. + + Emits one ``margin_call`` then up to ``cascade_steps`` ``forced_reduce`` / + ``cascade_impact`` pairs. ``forced_reduce.fraction`` is the cumulative reduction toward + flat; ``cascade_impact.mark_drop`` is the equity lost to that step's forced selling — + ``impact_per_step`` of the liquidated notional (the running ``peak`` is the high-water + notional proxy), the notional shrinking each step as the position is reduced, so the + drops compound down. The chain breaks early only if equity is wiped (``<= 0``). + """ + threshold = maintenance_margin * peak + events: list[dict[str, Any]] = [ + {"event": "margin_call", "nav": nav, "deficit": threshold - nav} + ] + cur = nav + notional = peak + for k in range(1, cascade_steps + 1): + events.append({"event": "forced_reduce", "fraction": min(1.0, k / cascade_steps)}) + mark_drop = notional * impact_per_step + cur -= mark_drop + notional *= 1.0 - impact_per_step + events.append({"event": "cascade_impact", "step": k, "mark_drop": mark_drop}) + if cur <= 0.0: + break + return events, cur + + +class LiquidationCascadeEnv(gym.Wrapper): + """Wrap an env so a maintenance-margin breach fires a deterministic liquidation cascade. + + After each ``env.step`` the realized equity (``info["nav"]``) is compared to + ``maintenance_margin * running_peak``. On breach the in-step cascade runs, its events are + appended to ``info["events"]``, and ``info["cascade"] = {"triggered", "events", + "final_nav"}`` is surfaced. ``terminated`` is set when the cascade wipes equity, else + ``truncated`` (stop-out convention). While the breach persists the executed action is + forced flat. + """ + + def __init__( + self, + env: gym.Env, + *, + maintenance_margin: float = 0.4, + cascade_steps: int = 3, + impact_per_step: float = 0.02, + ) -> None: + super().__init__(env) + if not 0.0 < maintenance_margin <= 1.0: + raise ValueError("maintenance_margin must be in (0, 1]") + if cascade_steps < 1: + raise ValueError("cascade_steps must be >= 1") + if not 0.0 <= impact_per_step < 1.0: + raise ValueError("impact_per_step must be in [0, 1)") + self._maintenance_margin = float(maintenance_margin) + self._cascade_steps = int(cascade_steps) + self._impact_per_step = float(impact_per_step) + self._peak: Optional[float] = None + self._liquidating = False + + def reset(self, **kwargs: Any): + self._peak = None + self._liquidating = False + return self.env.reset(**kwargs) + + def step(self, action): + executed = np.zeros_like(np.asarray(action)) if self._liquidating else action + obs, reward, terminated, truncated, info = self.env.step(executed) + nav = info.get("nav") + if nav is None: + return obs, reward, bool(terminated), bool(truncated), info + nav = float(nav) + if self._peak is None or nav > self._peak: + self._peak = nav + + info = dict(info) + events = list(info.get("events") or []) + cascade: dict[str, Any] = {"triggered": False, "events": [], "final_nav": nav} + + if self._peak > 0.0 and nav <= self._maintenance_margin * self._peak: + self._liquidating = True + chain, final_nav = _run_cascade( + nav, + self._peak, + maintenance_margin=self._maintenance_margin, + cascade_steps=self._cascade_steps, + impact_per_step=self._impact_per_step, + ) + events.extend(chain) + cascade = {"triggered": True, "events": chain, "final_nav": final_nav} + if final_nav <= 0.0: + terminated = True + else: + truncated = True + else: + self._liquidating = False + + info["events"] = events + info["cascade"] = cascade + return obs, reward, bool(terminated), bool(truncated), info + + +def cascade_survived(info: dict[str, Any]) -> bool: + """True if no cascade fired this step, or one fired but equity was not wiped. + + The basis for a "survive the cascade" mandate-style evaluation: a survived stop-out is + acceptable, a wiped one is not. + """ + cascade = info.get("cascade") + if not cascade or not cascade.get("triggered"): + return True + return float(cascade.get("final_nav", 0.0)) > 0.0 + + +def cascade_summary(info: dict[str, Any]) -> dict[str, Any]: + """A process-check-friendly digest of a step's cascade, composable with the verifiers + event consumers. Counts each chain event-type and reports survival.""" + cascade = info.get("cascade") or {} + events = cascade.get("events", []) if cascade.get("triggered") else [] + counts = {"margin_call": 0, "forced_reduce": 0, "cascade_impact": 0} + for e in events: + name = str(e.get("event", "")) + if name in counts: + counts[name] += 1 + return { + "triggered": bool(cascade.get("triggered", False)), + "margin_calls": counts["margin_call"], + "forced_reduces": counts["forced_reduce"], + "cascade_impacts": counts["cascade_impact"], + "final_nav": float(cascade.get("final_nav", info.get("nav", 0.0))), + "survived": cascade_survived(info), + } + + +__all__ = ["LiquidationCascadeEnv", "cascade_survived", "cascade_summary"] diff --git a/crates/openoutcry-py/python/openoutcry/reward_misspecification.py b/crates/openoutcry-py/python/openoutcry/reward_misspecification.py new file mode 100644 index 0000000..39e8402 --- /dev/null +++ b/crates/openoutcry-py/python/openoutcry/reward_misspecification.py @@ -0,0 +1,376 @@ +"""Reward-misspecification **negative controls** for OpenOutcry. + +OpenOutcry's thesis is that the SharpeBench kernel (deflated Sharpe / ``pass^k`` / +process checks) PUNISHES naive, over-fit, churn-heavy strategies. This module makes that +thesis *falsifiable*: it ships a registry of deliberately-FLAWED reward functions and a +demonstration that proxy agents optimized for them score BELOW a clean baseline on the +scorer — high in-sample raw return, near-zero deflated Sharpe out-of-sample. + +CRITICAL INVARIANT — these rewards are NEGATIVE CONTROLS for research only. They are NOT +valid scoring options. They MUST NEVER be registered into ``rewards.REWARD_SCHEMES`` and +MUST NEVER feed the scorer or the rank key. They re-introduce over-leverage / overfit / +churn / myopia *by design*. Importing this module does not mutate any production registry. + +The proxy policies (:data:`MISSPECIFIED_PROXY_POLICIES`) STAND IN for trained agents: we +cannot run GRPO here, so each proxy is the greedy maximizer of its flawed reward (a +max-leverage book for ``raw_pnl``, a tiny-position book for ``win_rate``, a momentum +chaser for ``indicator_shaped``, a last-bar chaser for ``recency_biased``). That is an +honest stand-in — the wedge is structural (deflated Sharpe is scale-invariant, so leverage +inflates raw return without moving Sharpe; deflation then floors it). +""" + +from __future__ import annotations + +import json +from typing import Any, Callable, Optional, Sequence + +import numpy as np + +from .openoutcry_py import score_run + +Policy = Callable[[dict], np.ndarray] +MakeEnv = Callable[[int], object] + + +# --------------------------------------------------------------------------- +# Misspecified reward functions — NEGATIVE CONTROLS. Bounded, pure over ``state``. +# Signature mirrors the production rewards so the rubric *could* route them — which is +# exactly why they must never be registered. +# --------------------------------------------------------------------------- + +_RAW_PNL_GAIN = 50.0 +_INDICATOR_WINDOW = 3 +_RECENCY_DECAY = 0.6 + + +def _returns_from_state(state: Optional[dict]) -> list[float]: + return [float(r) for r in (state or {}).get("returns", []) or []] + + +def _net_weights_per_bar(state: Optional[dict]) -> list[float]: + """Net signed weight per ``target_weights`` event (the agent's directional bet).""" + out: list[float] = [] + for e in (state or {}).get("events", []) or []: + if isinstance(e, dict) and e.get("event") == "target_weights": + w = e.get("weights") + if isinstance(w, (list, tuple)): + out.append(float(sum(float(x) for x in w))) + return out + + +def raw_pnl_unpenalized( + completion: Any = None, + state: Optional[dict] = None, + **kwargs: Any, +) -> float: + """NEGATIVE CONTROL — gross PnL with no risk, cost, or drawdown penalty. + + ``tanh(GAIN * sum(returns))``: a high-gain reward on raw cumulative return that ignores + volatility, turnover, and drawdown entirely, so it pays an agent to over-leverage and + churn. Bounded in ``[-1, 1]``. NOT a valid scoring option. + """ + rets = _returns_from_state(state) + if not rets: + return 0.0 + return float(np.tanh(_RAW_PNL_GAIN * float(np.sum(rets)))) + + +def win_rate( + completion: Any = None, + state: Optional[dict] = None, + **kwargs: Any, +) -> float: + """NEGATIVE CONTROL — fraction of positive bars, magnitude-blind. + + Rewards *how often* a bar is green, never *by how much*, so it pays an agent to harvest + many tiny wins and accept rare catastrophic losses (the classic blow-up reward). Bounded + in ``[0, 1]``. NOT a valid scoring option. + """ + rets = _returns_from_state(state) + if not rets: + return 0.0 + return float(np.mean([1.0 if r > 0.0 else 0.0 for r in rets])) + + +def indicator_shaped( + completion: Any = None, + state: Optional[dict] = None, + *, + window: int = _INDICATOR_WINDOW, + **kwargs: Any, +) -> float: + """NEGATIVE CONTROL — alignment with a naive short-window momentum signal. + + For each bar past ``window``, rewards the agent when its net position sign agrees with the + sign of the trailing ``window``-bar return momentum — a classic in-sample-overfit reward + that fits the agent to a single indicator instead of risk-adjusted edge. Reads the recorded + ``target_weights`` events; vacuously ``0.0`` if the agent never declared a direction. + Bounded in ``[0, 1]``. NOT a valid scoring option. + """ + rets = _returns_from_state(state) + nets = _net_weights_per_bar(state) + if len(rets) <= window or not nets: + return 0.0 + hits = 0.0 + count = 0 + for i in range(window, min(len(rets), len(nets))): + mom = float(np.sign(np.sum(rets[i - window : i]))) + pos = float(np.sign(nets[i])) + if mom != 0.0 and pos != 0.0: + hits += 1.0 if mom == pos else 0.0 + count += 1 + return float(hits / count) if count else 0.0 + + +def recency_biased( + completion: Any = None, + state: Optional[dict] = None, + *, + decay: float = _RECENCY_DECAY, + **kwargs: Any, +) -> float: + """NEGATIVE CONTROL — exponentially over-weights the most recent bars (myopic). + + Geometric decay places almost all weight on the last few bars, so an agent is rewarded for + chasing the latest move and discounting the path that produced it. Bounded in ``[-1, 1]`` + via ``tanh`` of the recency-weighted mean return. NOT a valid scoring option. + """ + rets = _returns_from_state(state) + if not rets: + return 0.0 + n = len(rets) + w = np.array([decay ** (n - 1 - i) for i in range(n)], dtype=float) + weighted = float(np.dot(w, np.asarray(rets, dtype=float)) / w.sum()) + return float(np.tanh(_RAW_PNL_GAIN * weighted)) + + +MISSPECIFIED_REWARDS: dict[str, Callable[..., float]] = { + "raw_pnl_unpenalized": raw_pnl_unpenalized, + "win_rate": win_rate, + "indicator_shaped": indicator_shaped, + "recency_biased": recency_biased, +} + + +# --------------------------------------------------------------------------- +# Proxy policies — greedy maximizers of each flawed reward. STAND-INS for trained agents. +# --------------------------------------------------------------------------- + + +class MaxLeveragePolicy: + """Greedy maximizer of ``raw_pnl_unpenalized``: full long on every symbol. + + Gross exposure ``= n`` (max per-symbol weight), so it harvests the most raw PnL the action + space allows. Because deflated Sharpe is scale-invariant, the leverage that inflates raw + return does NOT move the Sharpe — the wedge.""" + + name = "max_leverage" + + def __init__(self, max_weight: float = 1.0) -> None: + self._w = float(max_weight) + + def __call__(self, obs: dict) -> np.ndarray: + n = int(np.asarray(obs["closes"]).reshape(-1).shape[0]) + return np.full((n,), self._w, dtype=np.float32) + + +class TinyPositionPolicy: + """Greedy maximizer of ``win_rate``: a tiny constant long. + + Minimal exposure maximizes the fraction of green bars (slight drift wins often) while each + win is negligible — high win_rate, near-zero risk-adjusted edge.""" + + name = "tiny_position" + + def __init__(self, eps: float = 0.02) -> None: + self._eps = float(eps) + + def __call__(self, obs: dict) -> np.ndarray: + n = int(np.asarray(obs["closes"]).reshape(-1).shape[0]) + return np.full((n,), self._eps, dtype=np.float32) + + +class MomentumChasePolicy: + """Greedy maximizer of ``indicator_shaped``: full-size last-move sign chase. + + Bets the full per-symbol weight in the direction of the last close change, so its position + sign agrees with short-window momentum by construction. Warms up full long.""" + + name = "momentum_chase" + + def __init__(self, max_weight: float = 1.0) -> None: + self._w = float(max_weight) + self._prev: Optional[np.ndarray] = None + + def __call__(self, obs: dict) -> np.ndarray: + closes = np.asarray(obs["closes"], dtype=np.float64).reshape(-1) + n = closes.shape[0] + if self._prev is None: + self._prev = closes + return np.full((n,), self._w, dtype=np.float32) + sign = np.sign(closes - self._prev) + self._prev = closes + return (sign * self._w).astype(np.float32) + + +class RecencyChasePolicy: + """Greedy maximizer of ``recency_biased``: bet on the single most recent move. + + Sizes each symbol by the sign of its last one-bar change at full weight and ignores all + earlier history — maximally myopic. Warms up full long.""" + + name = "recency_chase" + + def __init__(self, max_weight: float = 1.0) -> None: + self._w = float(max_weight) + self._prev: Optional[np.ndarray] = None + + def __call__(self, obs: dict) -> np.ndarray: + closes = np.asarray(obs["closes"], dtype=np.float64).reshape(-1) + n = closes.shape[0] + if self._prev is None: + self._prev = closes + return np.full((n,), self._w, dtype=np.float32) + sign = np.sign(closes - self._prev) + self._prev = closes + return (sign * self._w).astype(np.float32) + + +def _clean_reference_policy() -> Policy: + """An equal-weight-long book — buy-and-hold analog standing in for a clean + (differential-Sharpe-trained) agent.""" + + def _policy(obs: dict) -> np.ndarray: + n = int(np.asarray(obs["closes"]).reshape(-1).shape[0]) + return np.full((n,), 1.0 / n, dtype=np.float32) + + return _policy + + +MISSPECIFIED_PROXY_POLICIES: dict[str, Callable[[], Policy]] = { + "raw_pnl_unpenalized": MaxLeveragePolicy, + "win_rate": TinyPositionPolicy, + "indicator_shaped": MomentumChasePolicy, + "recency_biased": RecencyChasePolicy, +} + + +# --------------------------------------------------------------------------- +# Scoring the proxies through the REAL SharpeBench kernel +# --------------------------------------------------------------------------- + + +def _rollout_returns(env, policy: Policy, max_steps: int) -> list[float]: + obs, _ = env.reset() + out: list[float] = [] + for _ in range(max_steps): + obs, reward, terminated, truncated, _info = env.step(policy(obs)) + out.append(float(reward)) + if bool(terminated) or bool(truncated): + break + return out + + +def _score_policy( + make_env_for_seed: MakeEnv, + seeds: Sequence[int], + factory: Callable[[], Policy], + max_steps: int, + n_trials: int, +) -> dict: + """Roll a fresh policy per seed, pool the return series, and score with ``score_run``.""" + pooled: list[float] = [] + passed: list[float] = [] + for s in seeds: + returns = _rollout_returns(make_env_for_seed(s), factory(), max_steps) + pooled.extend(returns) + if len(returns) >= 2: + comp = json.loads(score_run(returns, n_trials)) + passed.append(1.0 if comp.get("passed_k", False) else 0.0) + composite = json.loads(score_run(pooled, n_trials)) if len(pooled) >= 2 else {} + return { + "deflated_sharpe": float(composite.get("deflated_sharpe", 0.0)), + "passed_k": float(np.mean(passed)) if passed else 0.0, + "mean_return": float(np.mean(pooled)) if pooled else 0.0, + } + + +def misspecification_gap( + make_env_for_seed: MakeEnv, + seeds: Sequence[int], + *, + clean_reward: str = "differential_sharpe", + flawed_reward: str, + policy: Optional[Callable[[], Policy]] = None, + max_steps: int = 512, + n_trials: int = 2, +) -> dict: + """Score a clean reference vs a flawed-optimized proxy over the SAME seeds. + + ``policy`` is the clean reference policy factory (defaults to an equal-weight-long book + standing in for a ``clean_reward``-trained agent); the flawed side is the greedy proxy for + ``flawed_reward`` from :data:`MISSPECIFIED_PROXY_POLICIES`. Both are scored by the real + ``score_run`` kernel and the deflated-Sharpe / mean-return gaps are reported. The proxies + STAND IN for trained agents — we cannot run GRPO here, so we evaluate the reward's greedy + maximizer instead. + """ + if flawed_reward not in MISSPECIFIED_PROXY_POLICIES: + raise ValueError( + f"unknown flawed_reward {flawed_reward!r}; choose from " + f"{sorted(MISSPECIFIED_PROXY_POLICIES)}" + ) + seeds = list(seeds) + clean_factory = policy or _clean_reference_policy + flawed_factory = MISSPECIFIED_PROXY_POLICIES[flawed_reward] + clean = _score_policy(make_env_for_seed, seeds, clean_factory, max_steps, n_trials) + flawed = _score_policy(make_env_for_seed, seeds, flawed_factory, max_steps, n_trials) + return { + "clean_reward": clean_reward, + "flawed_reward": flawed_reward, + "clean": clean, + "flawed": flawed, + "gap_deflated_sharpe": clean["deflated_sharpe"] - flawed["deflated_sharpe"], + "gap_mean_return": clean["mean_return"] - flawed["mean_return"], + "proxy_is_stand_in": True, + } + + +def demonstrate_punishment( + make_env_for_seed: MakeEnv, + seeds: Sequence[int], + *, + max_steps: int = 512, + n_trials: Optional[int] = None, +) -> dict: + """Run every flawed-reward proxy over ``seeds`` and score it with SharpeBench. + + Returns ``{reward_name: {deflated_sharpe, passed_k, mean_return}}`` — the falsifiable + demonstration that proxies optimized for the misspecified rewards score poorly on deflated + Sharpe (and fail ``pass^k``) despite, where applicable, a healthy raw mean return. The + proxies STAND IN for trained agents. ``n_trials`` defaults to the proxy count — the honest + declared in-sample search breadth, which deflates Sharpe for multiple-comparison luck. + """ + seeds = list(seeds) + trials = len(MISSPECIFIED_PROXY_POLICIES) if n_trials is None else int(n_trials) + table: dict[str, dict] = {} + for reward_name, factory in MISSPECIFIED_PROXY_POLICIES.items(): + table[reward_name] = _score_policy( + make_env_for_seed, seeds, factory, max_steps, trials + ) + return table + + +__all__ = [ + "MISSPECIFIED_REWARDS", + "MISSPECIFIED_PROXY_POLICIES", + "raw_pnl_unpenalized", + "win_rate", + "indicator_shaped", + "recency_biased", + "MaxLeveragePolicy", + "TinyPositionPolicy", + "MomentumChasePolicy", + "RecencyChasePolicy", + "misspecification_gap", + "demonstrate_punishment", +] diff --git a/crates/openoutcry-py/tests/test_cascade.py b/crates/openoutcry-py/tests/test_cascade.py new file mode 100644 index 0000000..e582f29 --- /dev/null +++ b/crates/openoutcry-py/tests/test_cascade.py @@ -0,0 +1,260 @@ +"""Tests for the forced-liquidation cascade wrapper (Stream W3-CASCADE). + +Run from the crate dir:: + + python -m pytest crates/openoutcry-py/tests/test_cascade.py -q + +The logic tests drive a deterministic NAV-scripted stub env so the cascade chain is an exact +function of a known NAV path (no binding needed). The live-binding test is skipped when the +native ``openoutcry`` module is unavailable. +""" + +from __future__ import annotations + +import numpy as np +import pytest +import gymnasium as gym +from gymnasium import spaces + +from openoutcry.cascade import ( + LiquidationCascadeEnv, + cascade_survived, + cascade_summary, +) + + +class _NavEnv(gym.Env): + """Stub env replaying a scripted NAV path, surfacing ``info["nav"]`` and recording the + action actually executed so an action override can be observed.""" + + def __init__(self, navs, n: int = 2) -> None: + super().__init__() + self._navs = [float(x) for x in navs] + self.action_space = spaces.Box(-1.0, 1.0, shape=(n,), dtype=np.float32) + self.observation_space = spaces.Box(-np.inf, np.inf, shape=(1,), dtype=np.float64) + self._i = 0 + self.executed: np.ndarray | None = None + + def _obs(self): + return np.zeros((1,), dtype=np.float64) + + def reset(self, *, seed=None, options=None): + super().reset(seed=seed) + self._i = 0 + self.executed = None + return self._obs(), {} + + def step(self, action): + self.executed = np.asarray(action).copy() + nav = self._navs[self._i] + self._i += 1 + out_of_bars = self._i >= len(self._navs) + terminated = nav <= 0.0 + truncated = out_of_bars and not terminated + return self._obs(), 0.0, terminated, truncated, {"nav": nav} + + +def _act(env) -> np.ndarray: + n = env.action_space.shape[0] + return np.full((n,), 0.5, dtype=np.float32) + + +def _events_of(info, name): + return [e for e in info.get("events", []) if e.get("event") == name] + + +# -- breach fires a bounded, ordered cascade -------------------------------- + +def test_cascade_fires_and_emits_ordered_bounded_chain(): + # peak=1.0; maintenance_margin=0.4 -> breach when nav <= 0.4. nav drops to 0.3. + env = LiquidationCascadeEnv( + _NavEnv([1.0, 1.0, 0.3]), + maintenance_margin=0.4, + cascade_steps=3, + impact_per_step=0.02, + ) + env.reset() + a = _act(env) + env.step(a) + env.step(a) + _o, _r, terminated, truncated, info = env.step(a) + + chain = info["cascade"]["events"] + names = [e["event"] for e in chain] + # margin_call once, then (forced_reduce, cascade_impact) per step, bounded by 3. + assert names == [ + "margin_call", + "forced_reduce", "cascade_impact", + "forced_reduce", "cascade_impact", + "forced_reduce", "cascade_impact", + ] + assert info["cascade"]["triggered"] is True + assert len(_events_of(info, "cascade_impact")) == 3 + assert [e["step"] for e in _events_of(info, "cascade_impact")] == [1, 2, 3] + # chain is also merged into info["events"] for the verifiers consumers. + assert _events_of(info, "margin_call")[0]["deficit"] == pytest.approx(0.4 - 0.3) + # survived (equity not wiped) -> truncated, not terminated. + assert truncated is True and terminated is False + assert cascade_survived(info) is True + + +def test_cascade_summary_counts(): + env = LiquidationCascadeEnv(_NavEnv([1.0, 0.2]), maintenance_margin=0.5, cascade_steps=2) + env.reset() + a = _act(env) + env.step(a) + _o, _r, _t, _tr, info = env.step(a) + s = cascade_summary(info) + assert s["triggered"] is True + assert s["margin_calls"] == 1 + assert s["forced_reduces"] == 2 + assert s["cascade_impacts"] == 2 + assert s["survived"] is True + + +# -- no breach -> pass-through ---------------------------------------------- + +def test_no_breach_no_cascade_events(): + # nav dips to 0.7 of a peak of 1.0; margin 0.4 must NOT fire (0.7 > 0.4). + env = LiquidationCascadeEnv(_NavEnv([1.0, 0.7, 0.6]), maintenance_margin=0.4) + env.reset() + a = _act(env) + rows = [env.step(a) for _ in range(2)] + for _o, _r, terminated, truncated, info in rows: + assert info["cascade"]["triggered"] is False + assert _events_of(info, "margin_call") == [] + assert terminated is False and truncated is False + assert cascade_survived(info) is True + + +def test_passthrough_action_until_breach(): + base = _NavEnv([1.0, 0.9, 0.3, 0.3]) + env = LiquidationCascadeEnv(base, maintenance_margin=0.4) + env.reset() + a = _act(env) + env.step(a) # nav 1.0 + env.step(a) # nav 0.9, no breach + assert np.allclose(base.executed, a) # still passing the agent's action through + + +# -- forced-flat override during the cascade -------------------------------- + +def test_position_forced_toward_flat_while_liquidating(): + # nav breaches at step 3 (latch set), step 4 stays breached -> executed forced flat. + base = _NavEnv([1.0, 1.0, 0.3, 0.3]) + env = LiquidationCascadeEnv(base, maintenance_margin=0.4) + env.reset() + a = _act(env) + env.step(a) + env.step(a) + env.step(a) # breach -> latch + assert np.allclose(base.executed, a) # the breach step itself executed the agent action + env.step(a) # liquidating -> override flat + assert np.all(base.executed == 0.0) + + +def test_latch_releases_on_recovery(): + base = _NavEnv([1.0, 0.3, 0.9, 0.9]) + env = LiquidationCascadeEnv(base, maintenance_margin=0.4) + env.reset() + a = _act(env) + env.step(a) # nav 1.0 + env.step(a) # nav 0.3 -> breach, latch + env.step(a) # forced flat; nav 0.9 recovers above 0.4 -> latch clears + assert np.all(base.executed == 0.0) + env.step(a) # recovered -> agent action restored + assert np.allclose(base.executed, a) + + +# -- wipe -> terminated, not truncated -------------------------------------- + +def test_cascade_wipe_sets_terminated(): + # large impact so the chain drives equity below zero -> terminated. + env = LiquidationCascadeEnv( + _NavEnv([1.0, 0.01]), + maintenance_margin=0.5, + cascade_steps=5, + impact_per_step=0.9, + ) + env.reset() + a = _act(env) + env.step(a) + _o, _r, terminated, truncated, info = env.step(a) + assert info["cascade"]["final_nav"] <= 0.0 + assert terminated is True + assert cascade_survived(info) is False + # bounded: at most cascade_steps impacts even when wiping. + assert len(_events_of(info, "cascade_impact")) <= 5 + + +def test_base_termination_preserved(): + # base env goes bankrupt (nav <= 0) -> terminated stays true regardless of cascade. + env = LiquidationCascadeEnv(_NavEnv([1.0, -0.1]), maintenance_margin=0.4) + env.reset() + a = _act(env) + env.step(a) + _o, _r, terminated, _tr, _info = env.step(a) + assert terminated is True + + +# -- determinism ------------------------------------------------------------ + +def test_cascade_deterministic_event_sequence(): + def run(): + env = LiquidationCascadeEnv( + _NavEnv([1.0, 1.2, 1.0, 0.35, 0.3, 0.25]), + maintenance_margin=0.4, + cascade_steps=3, + impact_per_step=0.05, + ) + env.reset() + a = _act(env) + seq = [] + for _ in range(5): + _o, _r, _t, _tr, info = env.step(a) + seq.append( + [ + (e["event"], round(float(e.get("mark_drop", e.get("deficit", e.get("fraction", 0.0)))), 10)) + for e in info["cascade"]["events"] + ] + ) + return seq + + assert run() == run() + + +def test_five_tuple_shape_preserved(): + env = LiquidationCascadeEnv(_NavEnv([1.0, 1.0]), maintenance_margin=0.4) + env.reset() + out = env.step(_act(env)) + assert len(out) == 5 + _o, reward, terminated, truncated, info = out + assert np.isfinite(reward) + assert isinstance(terminated, bool) and isinstance(truncated, bool) + assert isinstance(info, dict) + + +# -- live binding (skipped when the native module is absent) ----------------- + +openoutcry = pytest.importorskip("openoutcry") + + +def test_live_cascade_extreme_scenario(): + # Drive a live env, then wrap with an extreme maintenance margin so any drawdown + # breaches and fires a cascade; assert the 5-tuple and event plumbing hold up. + base = openoutcry.OpenOutcryEnv(n_symbols=3, n_days=50, seed=1, distribution_mode="extreme") + env = LiquidationCascadeEnv(base, maintenance_margin=0.999, cascade_steps=3) + env.reset(seed=1) + n = env.action_space.shape[0] + a = np.full((n,), 1.0 / n, dtype=np.float32) + fired = False + for _ in range(40): + _o, _r, terminated, truncated, info = env.step(a) + assert "cascade" in info and "events" in info + if info["cascade"]["triggered"]: + fired = True + assert any(e["event"] == "margin_call" for e in info["cascade"]["events"]) + assert isinstance(cascade_survived(info), bool) + if terminated or truncated: + break + assert fired diff --git a/crates/openoutcry-py/tests/test_reward_misspecification.py b/crates/openoutcry-py/tests/test_reward_misspecification.py new file mode 100644 index 0000000..e466fc5 --- /dev/null +++ b/crates/openoutcry-py/tests/test_reward_misspecification.py @@ -0,0 +1,157 @@ +"""Tests for the reward-misspecification negative-control track. + +When the native binding and numpy are importable these exercise a real (small) proxy sweep +against the SharpeBench kernel and the falsifiable punishment demonstration. The module +imports ``score_run`` at top, so the whole file is binding-gated like ``test_baselines``. +""" + +import importlib + +import pytest + +try: + import numpy as np + + from openoutcry.reward_misspecification import ( + MISSPECIFIED_PROXY_POLICIES, + MISSPECIFIED_REWARDS, + demonstrate_punishment, + indicator_shaped, + misspecification_gap, + raw_pnl_unpenalized, + recency_biased, + win_rate, + ) + from openoutcry.gym import OpenOutcryEnv + + _HAVE_BINDING = importlib.util.find_spec("openoutcry.openoutcry_py") is not None +except Exception: # pragma: no cover - exercised only without the binding/numpy + _HAVE_BINDING = False + + +requires_binding = pytest.mark.skipif( + not _HAVE_BINDING, reason="native openoutcry binding not built" +) + + +def _make_env(seed: int): + return OpenOutcryEnv(n_symbols=3, n_days=60, seed=seed) + + +def _hand_state() -> dict: + return { + "returns": [0.01, -0.02, 0.03, -0.005, 0.012, -0.001, 0.004], + "events": [ + {"event": "target_weights", "weights": [0.3, 0.3, 0.3]} for _ in range(7) + ], + } + + +# -- rewards: bounded + computable from state ------------------------------- + + +@requires_binding +def test_misspecified_rewards_are_bounded_and_computable(): + state = _hand_state() + for name, fn in MISSPECIFIED_REWARDS.items(): + r = fn(state=state) + assert isinstance(r, float) + assert np.isfinite(r) + assert -1.0 <= r <= 1.0, name + # win_rate / indicator_shaped live in [0, 1]. + assert 0.0 <= win_rate(state=state) <= 1.0 + assert 0.0 <= indicator_shaped(state=state) <= 1.0 + + +@requires_binding +def test_rewards_handle_empty_state(): + for fn in MISSPECIFIED_REWARDS.values(): + assert fn(state={}) == 0.0 + assert fn(state=None) == 0.0 + + +@requires_binding +def test_raw_pnl_rewards_gross_return_unpenalized(): + # No risk/cost term: a bigger summed return strictly scores higher. + low = raw_pnl_unpenalized(state={"returns": [0.001, 0.001]}) + high = raw_pnl_unpenalized(state={"returns": [0.05, 0.05]}) + assert high > low + + +@requires_binding +def test_win_rate_ignores_magnitude(): + # Many tiny wins + one huge loss still scores high (the blow-up reward). + blowup = win_rate(state={"returns": [0.001, 0.001, 0.001, -0.5]}) + assert blowup == pytest.approx(0.75) + + +# -- proxy policies produce valid actions ----------------------------------- + + +@requires_binding +def test_proxy_policies_produce_valid_actions(): + obs = {"closes": np.array([100.0, 101.0, 99.0])} + for factory in MISSPECIFIED_PROXY_POLICIES.values(): + policy = factory() + first = policy(obs) + second = policy({"closes": np.array([101.0, 100.0, 100.0])}) + for action in (first, second): + assert action.shape == (3,) + assert np.all(np.isfinite(action)) + assert np.all(np.abs(action) <= 1.0 + 1e-6) + + +# -- the falsifiable wedge -------------------------------------------------- + + +@requires_binding +def test_demonstrate_punishment_scorer_punishes_flawed_proxies(): + table = demonstrate_punishment(_make_env, range(6), max_steps=128) + assert set(table) == set(MISSPECIFIED_REWARDS) + for row in table.values(): + assert set(row) == {"deflated_sharpe", "passed_k", "mean_return"} + assert np.isfinite(row["deflated_sharpe"]) + assert 0.0 <= row["passed_k"] <= 1.0 + + # No flawed proxy wins on the real metric: deflated Sharpe stays ~0. + assert max(r["deflated_sharpe"] for r in table.values()) <= 0.5 + + # The wedge: the proxy with the best raw mean return looks profitable in-sample yet + # earns ~0 deflated Sharpe — naive reward -> high raw return, scorer does not reward it. + best = max(table.values(), key=lambda r: r["mean_return"]) + assert best["mean_return"] > 0.0 + assert best["deflated_sharpe"] <= 0.5 + + +@requires_binding +def test_misspecification_gap_reports_clean_vs_flawed(): + g = misspecification_gap(_make_env, range(6), flawed_reward="raw_pnl_unpenalized", max_steps=128) + assert g["clean_reward"] == "differential_sharpe" + assert g["flawed_reward"] == "raw_pnl_unpenalized" + assert g["proxy_is_stand_in"] is True + assert np.isfinite(g["gap_deflated_sharpe"]) + assert np.isfinite(g["gap_mean_return"]) + for side in ("clean", "flawed"): + assert set(g[side]) == {"deflated_sharpe", "passed_k", "mean_return"} + + +@requires_binding +def test_misspecification_gap_rejects_unknown_reward(): + with pytest.raises(ValueError): + misspecification_gap(_make_env, range(2), flawed_reward="not_a_reward") + + +# -- the critical invariant: never registered into production ---------------- + + +@requires_binding +def test_negative_controls_are_not_in_production_registry(): + from openoutcry import rewards as production_rewards + + for name in ( + "raw_pnl_unpenalized", + "win_rate", + "indicator_shaped", + "recency_biased", + ): + assert name not in production_rewards.REWARD_SCHEMES