Polish comparison defaults, trust notes, and reporting metadata

Author: Yurashku

README.md

@@ -10,6 +10,10 @@
   - `p_value` (двусторонний centered paired bootstrap test для `H0: delta = 0`),
   - `is_significant` + `significance_rule="centered_paired_bootstrap_p_value_lt_alpha"`.
 - В comparison output также включены trust/stability diagnostics (`ESS`, `ESS/N`, replay overlap, weight tails, clip/switch share, warning flags).
+- В high-level summary добавлены explicit recommended-default metadata и структурированные note-группы:
+  - `recommended_defaults` (рекомендуемые режимы по умолчанию),
+  - `info_notes`, `diagnostic_warnings`, `inference_warnings`, `trust_notes`,
+  - итоговый `trust_level` и короткая `recommendation`.
 - Все основные OPE‑оценщики снабжены подробными docstring на русском (аргументы, возвращаемые значения, интерпретация).
 
 ## Установка
@@ -57,6 +61,15 @@ pip install -e .
 
 В `compare_policies(...).to_dict()` и `diagnostics` возвращаются `propensity_source` и `propensity_column` (если применимо).
 
+### Recommended defaults (safe-by-default guidance)
+
+Официальные defaults для общего сценария:
+- preferred estimator: `dr`;
+- если logged propensity доступна и валидна: `propensity_source="auto"` (предпочтёт logged path);
+- если logged propensity недоступна/невалидна: fallback в estimated propensity path;
+- `use_crossfit=True` обычно рекомендуется для `dm/dr/sndr/switch_dr`, когда важна bias-hardening устойчивость;
+- `trust_level in {"caution", "elevated_concern"}` — сигнал поднимать требования к интерпретации результата.
+
 ### Nuisance model diagnostics
 
 В high-level summary добавлен блок `nuisance_diagnostics`:
@@ -304,3 +317,8 @@ jupyter nbconvert --to notebook --execute examples/tutorial.ipynb --inplace
   - `p_value` (centered paired bootstrap approximation для `H0: delta = 0`),
   - `inference_method`,
   - `alpha`.
+- Дополнительно для API-polish:
+  - структурированные note/warning-поля: `info_notes`, `diagnostic_warnings`, `inference_warnings`, `trust_notes`;
+  - `trust_level` + `recommendation`;
+  - `recommended_defaults` для явного safe-by-default workflow.
+  - Поле `notes` сохранено для backward compatibility как объединение структурированных групп.

docs/architecture.md

@@ -36,6 +36,8 @@
 7. **Comparison result**  
    Сводка по `V_A`, `V_B`, `delta`, CI, `p-value` и диагностике для сравнения A vs B.
    Официальный orchestration path: `policyscope.comparison.compare_policies(...)`.
+   В summary дополнительно нормализованы группы заметок/предупреждений:
+   `info_notes`, `diagnostic_warnings`, `inference_warnings`, `trust_notes`, и агрегированный `trust_level`.
 
 8. **Scalar target metric (core abstraction)**  
    Базовая единица оценки — одна скалярная метрика награды. Несколько метрик поддерживаются как повторные запуски оценки для разных target-колонок, а не как native vector-valued reward.
@@ -135,3 +137,14 @@ Harness поддерживает сравнение методов (`replay`, `i
 - cross-fit mode: diagnostics отмечаются как OOF (fold-aware provenance).
 
 Этот слой не меняет формулы estimators и служит для trust-quality интерпретации результатов.
+
+
+## 11) Recommended defaults (API-polish)
+
+Чтобы high-level API был opinionated и безопаснее по умолчанию, в comparison metadata фиксируются рекомендации:
+- `preferred_estimator_general_use = "dr"`;
+- `preferred_propensity_mode_when_logged_available = "auto"`;
+- `preferred_propensity_fallback_when_logged_unavailable = "estimated"`;
+- рекомендация cross-fit для `dm/dr/sndr/switch_dr`.
+
+Это guidance-слой и metadata; математика реализованных estimators не меняется.

docs/validation_harness.md

@@ -11,6 +11,7 @@
 - `Delta_CI` coverage (если CI рассчитан);
 - частота significance decision (`is_significant`);
 - diagnostics-поля (например, `weight_ess_ratio`, `weight_p99`);
+- trust metadata (`trust_level`, structured warnings/notes), чтобы видеть когда выводы стоит считать менее надёжными;
 - provenance (`propensity_source_used`, `propensity_column_used`);
 - nuisance-quality summaries (например behavior log-loss, outcome log-loss/RMSE) для сравнения режимов.
 

src/policyscope/__init__.py

@@ -10,8 +10,21 @@
 
 import logging
 
+from policyscope.comparison import (
+    RECOMMENDED_CROSSFIT_ESTIMATORS,
+    RECOMMENDED_ESTIMATOR,
+    RECOMMENDED_PROPENSITY_SOURCE_FALLBACK,
+    RECOMMENDED_PROPENSITY_SOURCE_WITH_LOGGED,
+)
+
 logging.basicConfig(level=logging.INFO, format="%(message)s")
 
-__all__ = ["__version__"]
+__all__ = [
+    "__version__",
+    "RECOMMENDED_ESTIMATOR",
+    "RECOMMENDED_PROPENSITY_SOURCE_WITH_LOGGED",
+    "RECOMMENDED_PROPENSITY_SOURCE_FALLBACK",
+    "RECOMMENDED_CROSSFIT_ESTIMATORS",
+]
 
 __version__ = "0.1.0"

src/policyscope/comparison.py

@@ -20,6 +20,11 @@
     resolve_behavior_predictions,
 )
 
+RECOMMENDED_ESTIMATOR = "dr"
+RECOMMENDED_PROPENSITY_SOURCE_WITH_LOGGED = "auto"
+RECOMMENDED_PROPENSITY_SOURCE_FALLBACK = "estimated"
+RECOMMENDED_CROSSFIT_ESTIMATORS = frozenset({"dm", "dr", "sndr", "switch_dr"})
+
 
 @dataclass(frozen=True)
 class PolicyValueResult:
@@ -47,6 +52,12 @@ class PolicyComparisonSummary:
     inference_warnings: tuple[str, ...] = field(default_factory=tuple)
     diagnostics: PolicyDiagnostics | None = None
     notes: tuple[str, ...] = field(default_factory=tuple)
+    info_notes: tuple[str, ...] = field(default_factory=tuple)
+    diagnostic_warnings: tuple[str, ...] = field(default_factory=tuple)
+    trust_notes: tuple[str, ...] = field(default_factory=tuple)
+    trust_level: str = "ok"
+    recommendation: Optional[str] = None
+    recommended_defaults: dict[str, object] = field(default_factory=dict)
     propensity_source: Optional[str] = None
     propensity_column: Optional[str] = None
     nuisance_diagnostics: Optional[NuisanceDiagnostics] = None
@@ -60,7 +71,15 @@ def to_dict(self) -> dict:
             "Delta": self.delta,
             "diagnostics": self.diagnostics.to_dict() if self.diagnostics is not None else {},
             "notes": list(self.notes),
+            "info_notes": list(self.info_notes),
+            "diagnostic_warnings": list(self.diagnostic_warnings),
+            "trust_notes": list(self.trust_notes),
+            "trust_level": self.trust_level,
         }
+        if self.recommendation is not None:
+            out["recommendation"] = self.recommendation
+        if self.recommended_defaults:
+            out["recommended_defaults"] = self.recommended_defaults
         if self.v_a_ci is not None:
             out["V_A_CI"] = self.v_a_ci
         if self.v_b_ci is not None:
@@ -90,6 +109,49 @@ def to_dict(self) -> dict:
         return out
 
 
+def _recommended_defaults(estimator: str) -> dict[str, object]:
+    return {
+        "preferred_estimator_general_use": RECOMMENDED_ESTIMATOR,
+        "preferred_propensity_mode_when_logged_available": RECOMMENDED_PROPENSITY_SOURCE_WITH_LOGGED,
+        "preferred_propensity_fallback_when_logged_unavailable": RECOMMENDED_PROPENSITY_SOURCE_FALLBACK,
+        "crossfit_recommended_for_estimator": estimator in RECOMMENDED_CROSSFIT_ESTIMATORS,
+    }
+
+
+def _build_trust_metadata(
+    *,
+    estimator: str,
+    use_crossfit: bool,
+    propensity_notes: tuple[str, ...],
+    diagnostic_warnings: tuple[str, ...],
+    inference_warnings: tuple[str, ...],
+) -> tuple[tuple[str, ...], tuple[str, ...], str, Optional[str]]:
+    info_notes = list(dict.fromkeys(propensity_notes))
+    trust_notes: list[str] = []
+    risk_score = 0
+    if diagnostic_warnings:
+        risk_score += len(diagnostic_warnings)
+        trust_notes.append("diagnostics_warnings_present_review_weight_overlap_metrics")
+    if inference_warnings:
+        risk_score += len(inference_warnings)
+        trust_notes.append("inference_warnings_present_ci_and_p_value_less_stable")
+    if estimator in RECOMMENDED_CROSSFIT_ESTIMATORS and not use_crossfit:
+        info_notes.append("crossfit_optional_recommendation_for_bias_hardening")
+    if any(w in {"low_ess_ratio", "heavy_weight_tail", "extreme_max_weight"} for w in diagnostic_warnings):
+        risk_score += 1
+        trust_notes.append("trust_elevated_concern_unstable_importance_weights")
+
+    trust_level = "ok"
+    recommendation = None
+    if risk_score >= 3:
+        trust_level = "elevated_concern"
+        recommendation = "Treat comparison as directional; improve overlap/weights or collect more representative logs."
+    elif risk_score > 0:
+        trust_level = "caution"
+        recommendation = "Review diagnostics and inference warnings before making product decisions."
+    return tuple(info_notes), tuple(trust_notes), trust_level, recommendation
+
+
 @dataclass(frozen=True)
 class MultiMetricComparisonResult:
     estimator: str
@@ -263,14 +325,29 @@ def point_on(part: pd.DataFrame) -> float:
     )
 
     if not with_ci:
+        diag_warnings = tuple(diag.warnings)
+        info_notes, trust_notes, trust_level, recommendation = _build_trust_metadata(
+            estimator=estimator,
+            use_crossfit=use_crossfit,
+            propensity_notes=propensity_notes,
+            diagnostic_warnings=diag_warnings,
+            inference_warnings=tuple(),
+        )
+        notes = tuple(dict.fromkeys(info_notes + diag_warnings + trust_notes))
         return PolicyComparisonSummary(
             estimator=estimator,
             target=target,
             v_a=float(v_a),
             v_b=float(v_b),
             delta=float(v_b - v_a),
             diagnostics=diag,
-            notes=propensity_notes + tuple(diag.warnings),
+            notes=notes,
+            info_notes=info_notes,
+            diagnostic_warnings=diag_warnings,
+            trust_notes=trust_notes,
+            trust_level=trust_level,
+            recommendation=recommendation,
+            recommended_defaults=_recommended_defaults(estimator),
             propensity_source=diag.propensity_source or resolved_source,
             propensity_column=diag.propensity_column or resolved_propensity_col,
             nuisance_diagnostics=nuisance_diag,
@@ -291,7 +368,15 @@ def estimator_pair(part: pd.DataFrame):
     inference_warnings = tuple(inf.get("inference_warnings", []))
     if fallback_triggered["value"]:
         inference_warnings = inference_warnings + (external_nuisance_bootstrap_warning,)
-    notes = propensity_notes + tuple(diag.warnings) + inference_warnings
+    diag_warnings = tuple(diag.warnings)
+    info_notes, trust_notes, trust_level, recommendation = _build_trust_metadata(
+        estimator=estimator,
+        use_crossfit=use_crossfit,
+        propensity_notes=propensity_notes,
+        diagnostic_warnings=diag_warnings,
+        inference_warnings=inference_warnings,
+    )
+    notes = tuple(dict.fromkeys(info_notes + diag_warnings + inference_warnings + trust_notes))
     return PolicyComparisonSummary(
         estimator=estimator,
         target=target,
@@ -310,6 +395,12 @@ def estimator_pair(part: pd.DataFrame):
         inference_warnings=inference_warnings,
         diagnostics=diag,
         notes=notes,
+        info_notes=info_notes,
+        diagnostic_warnings=diag_warnings,
+        trust_notes=trust_notes,
+        trust_level=trust_level,
+        recommendation=recommendation,
+        recommended_defaults=_recommended_defaults(estimator),
         propensity_source=diag.propensity_source or resolved_source,
         propensity_column=diag.propensity_column or resolved_propensity_col,
         nuisance_diagnostics=nuisance_diag,

src/policyscope/report.py

@@ -48,22 +48,30 @@ def decision_summary(res: Dict, metric_name: str, business_threshold: float = 0.
     V_A = res["V_A"]
     V_B = res["V_B"]
     D = res["Delta"]
+    alpha = float(res.get("alpha", 0.05))
+    ci_level = int(round((1.0 - alpha) * 100))
     A_lo, A_hi = res["V_A_CI"]
     B_lo, B_hi = res["V_B_CI"]
     D_lo, D_hi = res["Delta_CI"]
 
     lines = []
     lines.append(f"Метрика: {metric_name}")
-    lines.append(f"V(A) = {V_A:.6f} (95% CI: {A_lo:.6f} .. {A_hi:.6f})")
-    lines.append(f"V(B) = {V_B:.6f} (95% CI: {B_lo:.6f} .. {B_hi:.6f})")
-    lines.append(f"Delta (B−A) = {D:.6f} (95% CI: {D_lo:.6f} .. {D_hi:.6f})")
+    lines.append(f"V(A) = {V_A:.6f} ({ci_level}% CI: {A_lo:.6f} .. {A_hi:.6f})")
+    lines.append(f"V(B) = {V_B:.6f} ({ci_level}% CI: {B_lo:.6f} .. {B_hi:.6f})")
+    lines.append(f"Delta (B−A) = {D:.6f} ({ci_level}% CI: {D_lo:.6f} .. {D_hi:.6f})")
 
     if D_lo > business_threshold:
         lines.append(f"Решение: модель B лучше A, поскольку нижняя граница CI превышает порог {business_threshold}.")
     elif D_hi < -business_threshold:
         lines.append(f"Решение: модель A лучше B, поскольку верхняя граница CI ниже -{business_threshold}.")
     else:
         lines.append("Решение: статистически значимого отличия не обнаружено или эффект слишком мал.")
+    recommendation = res.get("recommendation")
+    trust_level = res.get("trust_level")
+    if trust_level is not None:
+        lines.append(f"Уровень доверия к оценке: {trust_level}.")
+    if recommendation:
+        lines.append(f"Рекомендация: {recommendation}")
     return "\n".join(lines)
 
 

src/policyscope/validation.py

@@ -44,6 +44,7 @@ class ValidationRunRow:
     p_value: Optional[float]
     propensity_source_used: Optional[str]
     propensity_column_used: Optional[str]
+    trust_level: Optional[str]
     ess_ratio: Optional[float]
     weight_p99: Optional[float]
     behavior_log_loss: Optional[float]
@@ -191,6 +192,7 @@ def run_simulation_validation(
                         p_value=summary.p_value,
                         propensity_source_used=summary.propensity_source,
                         propensity_column_used=summary.propensity_column,
+                        trust_level=summary.trust_level,
                         ess_ratio=diag.get("weight_ess_ratio"),
                         weight_p99=diag.get("weight_p99"),
                         behavior_log_loss=(

tests/test_bootstrap_report.py

@@ -160,3 +160,20 @@ def test_decision_summary_outcomes():
     res_neu = {**base, "Delta": 0.0, "Delta_CI": (-0.03, 0.04)}
     txt_neu = decision_summary(res_neu, "metric", business_threshold=0.01)
     assert "статистически значимого отличия" in txt_neu
+
+
+def test_decision_summary_uses_alpha_from_result():
+    res = {
+        "V_A": 0.2,
+        "V_B": 0.25,
+        "Delta": 0.05,
+        "V_A_CI": (0.18, 0.22),
+        "V_B_CI": (0.20, 0.30),
+        "Delta_CI": (0.01, 0.09),
+        "alpha": 0.1,
+        "trust_level": "caution",
+        "recommendation": "check diagnostics",
+    }
+    txt = decision_summary(res, "metric", business_threshold=0.0)
+    assert "90% CI" in txt
+    assert "Уровень доверия к оценке: caution." in txt

tests/test_comparison.py

@@ -46,6 +46,9 @@ def test_official_comparison_entrypoint_shape():
     assert "V_A_CI" in d and "Delta_CI" in d
     assert "diagnostics" in d and "weight_ess_ratio" in d["diagnostics"]
     assert 0.0 <= d["p_value"] <= 1.0
+    assert d["recommended_defaults"]["preferred_estimator_general_use"] == "dr"
+    assert "info_notes" in d and "diagnostic_warnings" in d and "trust_notes" in d
+    assert d["trust_level"] in {"ok", "caution", "elevated_concern"}
 
 
 def test_multi_target_repeated_scalar_evaluation():
@@ -223,10 +226,30 @@ def test_propensity_source_auto_fallback_and_metadata():
         propensity_col="missing_propensity",
     )
     assert summary.propensity_source == "estimated"
-    assert any("fallback" in n for n in summary.notes)
+    assert any("fallback" in n for n in summary.info_notes)
     assert summary.to_dict()["diagnostics"]["propensity_source"] == "estimated"
 
 
+def test_notes_are_structured_and_legacy_notes_remain_compatible():
+    logs, policyB = _prepare_env(114)
+    summary = compare_policies(
+        logs,
+        policyB,
+        estimator="dr",
+        target="accept",
+        feature_cols=["loyal", "age", "risk", "income"],
+        action_col="a_A",
+        with_ci=True,
+        n_boot=10,
+    )
+    assert isinstance(summary.info_notes, tuple)
+    assert isinstance(summary.diagnostic_warnings, tuple)
+    assert isinstance(summary.inference_warnings, tuple)
+    assert isinstance(summary.trust_notes, tuple)
+    # Legacy combined notes stays available for backward-compatible consumers.
+    assert set(summary.info_notes).issubset(set(summary.notes))
+
+
 def test_propensity_source_logged_requires_valid_column():
     logs, policyB = _prepare_env(110)
     logs = logs.copy()