License statement

The code is provided under a Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Under the license, the code is provided royalty free for non-commercial purposes only. The code may be covered by patents and if you want to use the code for commercial purposes, please contact us for a different license.

A collection of sequential statistical hypothesis testing methods for two-by-two contingency tables.

[Paper]  [Website]

Installation Instructions [Standard]

The basic environmental setup is shown below. A virtual / conda environment may be constructed; however, the requirements are quite lightweight and this is probably not needed.

$ pip install sequentialized_barnard_tests

Installation Instructions [Dev]

For potential contributors and developers, we recommend a virtualenv:

$ cd <some_directory>
$ git clone git@github.com:TRI-ML/sequentialized_barnard_tests.git
$ virtualenv --python=python3.10 <env_name>
$ source <env_name>/bin/activate
$ cd sequentialized_barnard_tests
$ pip install -r requirements.txt
$ pip install -e .
$ pre-commit install

We assume that any specified virtual / conda environment has been activated for all subsequent code snippets.

Quick Start Guides

Convenience: Automatic Test Instantiation

For convenience, you can automatically select between STEP and Lai (a baseline method) depending on the value of n_max using the factory function in auto.py:

from sequentialized_barnard_tests import get_mirrored_test
test = get_mirrored_test(n_max, alternative, alpha, verbose=True, ...)

If n_max > 500, this will instantiate a MirroredLaiTest, which is a computationally efficient baseline with comparable performance to MirroredStepTest for a large-enough sample size; otherwise, it will use the more powerful MirroredStepTest which can take longer to synthesize the decision rule. All shared and class-specific arguments can be passed as keyword arguments.

Example Usage

Below is a minimum example code with different policy evaluation data, leading to three distinct evaluation results.

Case 1: Test yields AcceptAlternative

from sequentialized_barnard_tests import get_mirrored_test, Hypothesis
n_max = 100  # maximum sample size is 100 (per policy)
alternative = Hypothesis.P0LessThanP1  # we want to test if "success rate of the first policy < success rate of the second policy"
alpha = 0.05  # false positive rate is 5%
test = get_mirrored_test(n_max=n_max, alternative=alternative, alpha=alpha)
success_array_policy_0 = [False] * 10  # the first policy failed 10 times
success_array_policy_1 = [True] * 10  # the second policy succeeded 10 times 
result = test.run_on_sequence(success_array_policy_0, success_array_policy_1)
decision = result.decision
print(decision)  # AcceptAlternative: success rate of the first policy < success rate of the second policy with 95% confidence

Case 2: Test yields AcceptNull

from sequentialized_barnard_tests import get_mirrored_test, Hypothesis
n_max = 100  # maximum sample size is 100 (per policy)
alternative = Hypothesis.P0LessThanP1  # we want to test if "success rate of the first policy < success rate of the second policy"
alpha = 0.05  # false positive rate is 5%
test = get_mirrored_test(n_max=n_max, alternative=alternative, alpha=alpha)
success_array_policy_0 = [True] * 10  # the first policy succeeded 10 times
success_array_policy_1 = [False] * 10  # the second policy failed 10 times 
result = test.run_on_sequence(success_array_policy_0, success_array_policy_1)
decision = result.decision
print(decision)  # AcceptNull: success rate of the first policy > success rate of the second policy with 95% confidence

Note: AcceptNull is a valid decision only for "mirrored" tests. In our terminology, a mirrored test is one that runs two one-sided tests simultaneously, with the null and the alternaive flipped from each other. (Because of the monotonicity of the test statistic, mirrored tests suffer no penalty for running two tests simultaneously, and therefore essentially dominate one-sided tests.) In the example above, the alternative is Hypothesis.P0LessThanP1 and the decision is Decision.AcceptNull, which should be interpreted as accepting Hypothesis.P0MoreThanP1. If you rather want a more conventional one-sided test, you can instantiate one by calling get_test instead of get_mirrored_test.

Case 3: Test yields FailToDecide

from sequentialized_barnard_tests import get_mirrored_test, Hypothesis
n_max = 100  # maximum sample size is 100 (per policy)
alternative = Hypothesis.P0LessThanP1  # we want to test if "success rate of the first policy < success rate of the second policy"
alpha = 0.05  # false positive rate is 5%
test = get_mirrored_test(n_max=n_max, alternative=alternative, alpha=alpha)
success_array_policy_0 = [True, False, False, True]  # the first policy succeeded 2 out of 4 times
success_array_policy_1 = [False, True, True, True]  # the second policy succeeded 3 out of 4 times
result = test.run_on_sequence(success_array_policy_0, success_array_policy_1)
decision = result.decision
print(decision)  # FailToDecide: difference was not statistically separable; user can collect 100 - 4 = 96 more rollouts for each policy to re-run the test.

More Working Examples

• quick_start.ipynb presents a single-task policy comparison example using actual hardware evaluation results.
• multi_policy_comparison_example.ipynb describes how to compare multiple policies at once, both on single task and multiple tasks.

Key Notes for Understanding the Core Ideas of STEP Code

We include key notes for understanding the core ideas of the STEP code. Quick-start resources are included in both shell script and notebook form.

(1A) Understanding the Accepted Shape Parameters

In order to synthesize a STEP Policy for specific values of n_max and alpha, one additional set of parametric decisions will be required. The user will need to set the risk budget shape, which is specified by choice of function family (p-norm vs zeta-function) and particular shape parameter. The shape parameter is real-valued; it is used directly for zeta functions and is exponentiated for p-norms.

For p-norms