expectation: A modern take on statistical A/B testing with e-values and martingales

Wednesday 12:25 in Helium3

Modern data science demands flexible statistical methods that can handle sequential data analysis and continuous monitoring. Traditional p-values, while widely used, have limitations when dealing with sequential testing scenarios. This talk introduces a Python library that implements e-values and e-processes, offering a more natural approach to measuring statistical evidence and enabling true sequential testing.

Outline:

Statistical toolkit

Current tools
Purpose and fundamental concepts
Challenges in modern statistics
Type 1 error concerns
Optional stopping problems

Sequential testing

Origins
The concept of sequential testing
Peeking

e-values

What are e-values?
Definitions and concepts
Betting interpretation
Wealth process
Ville's inequality
Anytime valid inference
p-value vs. e-value differences

Python library

Architecture
Core components
Installation and basic setup

Demo 1: A/B testing
Beyond A/B testing

Broader applications
Conformal e-testing
Confidence sequences

Demo 2: It is a versatile library
Acknowledgments

Q&A Session

Jako Rostami

I am a Machine Learning Engineer at H&M Group, former Data Scientist at Lidl Sweden, as a professional I am designing Machine Learning services, extracting insights and arranging meaningful stories for my clients by conducting high-quality modeling, engineering, data mining and analytics.

I have a Bachelor degree in Statistics and Probability theory from Uppsala University of Sweden. Because I am a Statistician at core I have good experience with Data Sciencr, Python, R, time series modeling, simulations, machine learning algorithms, SQL, Excel, Spark and database technologies, as well as good communication skills.

You’ll find two comprehensive Python libraries I have open-sourced. One is based on an emerging modern statistical hypothesis testing framework using e-values and martingales based on game-theoretic statistics. The other is for computational Supply Chain and Logistics. The first one is called ’expectation’ and the second one is called ’supplyseer’ and you can find both on my GitHub.