[FEATURE] CUPED

[tikhomirovd]

🚀 Feature Proposal

Implement CUPED in AB Tests

Motivation

Currently, AB tests in HypEx do not leverage variance reduction techniques like CUPED (Controlled Experiment Using Pre-Experiment Data). CUPED can significantly reduce variance in experimental results, improving statistical power and enabling more reliable conclusions. Adding CUPED will enhance the robustness of AB tests in HypEx.

Feature Description

• Implement CUPED variance reduction in the AB testing framework.
• Ensure CUPED is an optional feature that can be toggled by users.
• Integrate the method into the existing experimental pipeline while maintaining backward compatibility.
• Provide clear documentation on how and when to use CUPED.
• Add example notebooks demonstrating CUPED’s impact on variance reduction.

Potential Impacts

• Improved statistical power in AB tests.
• Reduced sample size requirements for detecting significant effects.
• Increased reliability of experiment results.

Alternatives

• Keep the current AB testing approach without variance reduction (less efficient).
• Implement alternative variance reduction techniques such as stratification or post-stratification.

Additional Context

• Follow best practices in implementing CUPED (pre-experiment covariates, linear adjustment).
• Ensure integration is compatible with existing experiment structures.
• Add unit tests to verify the correctness of CUPED’s implementation.

Checklist

• Implement CUPED in AB testing framework.
• Ensure seamless integration with existing experiments.
• Write unit tests covering CUPED logic and correctness.
• Provide clear documentation and usage examples.
• Create a tutorial notebook demonstrating CUPED’s impact.
• Review and refine before merging.