{"id":20482817,"url":"https://github.com/oliverhennhoefer/online-fdr","last_synced_at":"2026-06-06T19:31:34.023Z","repository":{"id":249667466,"uuid":"832007334","full_name":"OliverHennhoefer/online-fdr","owner":"OliverHennhoefer","description":"Online Multiple Hypothesis Testing.","archived":false,"fork":false,"pushed_at":"2024-12-13T16:04:13.000Z","size":134,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-16T04:16:58.867Z","etag":null,"topics":["addis","alpha-investing","alpha-spending","anomaly-detection","batching","benjamini-hochberg","bonferroni","false-discovery-rate","false-positive","false-positive-control","fdr","fwer","hypothesis-testing","interim-analysis","lond","lord","online-hypothesis-testing","p-value","saffron","uncertainty-quantification"],"latest_commit_sha":null,"homepage":"https://projecteuclid.org/journals/statistical-science/volume-38/issue-4/Online-Multiple-Hypothesis-Testing/10.1214/23-STS901.full","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OliverHennhoefer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-22T07:03:40.000Z","updated_at":"2024-12-23T16:17:50.000Z","dependencies_parsed_at":"2024-08-26T09:35:18.000Z","dependency_job_id":"e9b11efa-71ce-4a16-994b-9717dd916951","html_url":"https://github.com/OliverHennhoefer/online-fdr","commit_stats":null,"previous_names":["oliverhennhoefer/online-fdr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliverHennhoefer%2Fonline-fdr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliverHennhoefer%2Fonline-fdr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliverHennhoefer%2Fonline-fdr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OliverHennhoefer%2Fonline-fdr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OliverHennhoefer","download_url":"https://codeload.github.com/OliverHennhoefer/online-fdr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242058087,"owners_count":20065062,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["addis","alpha-investing","alpha-spending","anomaly-detection","batching","benjamini-hochberg","bonferroni","false-discovery-rate","false-positive","false-positive-control","fdr","fwer","hypothesis-testing","interim-analysis","lond","lord","online-hypothesis-testing","p-value","saffron","uncertainty-quantification"],"created_at":"2024-11-15T16:14:42.061Z","updated_at":"2026-06-06T19:31:34.016Z","avatar_url":"https://github.com/OliverHennhoefer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Online FDR: Online False Discovery Rate Control Algorithms\n\n[![python](https://img.shields.io/badge/Python-3.10+-3776AB.svg?style=flat\u0026logo=python\u0026logoColor=white)](https://www.python.org)\n[![License](https://img.shields.io/badge/License-BSD_3--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)\n[![Code style: black](https://img.shields.io/badge/code_style-black-black)](https://github.com/psf/black)\n\n## Overview\n\n**online-fdr** is a Python library for controlling False Discovery Rate (FDR) and Family-Wise Error Rate (FWER) in online multiple hypothesis testing scenarios. Unlike traditional methods that require all p-values upfront, this library provides truly online algorithms that make decisions sequentially as data arrives.\n\n### Why Online FDR Control?\n\nIn many applications, hypotheses arrive sequentially:\n- **Clinical Trials**: Interim analyses as patient data accumulates\n- **A/B Testing**: Continuous experimentation in tech companies  \n- **Genomics**: Sequential gene discovery studies\n- **Finance**: Real-time anomaly detection in trading\n- **Web Analytics**: Ongoing feature testing and optimization\n\nThis library implements state-of-the-art online algorithms that:\n- Make immediate decisions without waiting for future data\n- Maintain rigorous statistical guarantees\n- Support both independent and dependent p-values\n- Provide a unified API for sequential and batch testing\n\n## Installation\n\n```bash\npip install online-fdr\n```\n\n## Quick Start\n\n```python\nfrom online_fdr.investing.addis.addis import Addis\nfrom online_fdr.utils.generation import DataGenerator, GaussianLocationModel\n\n# Initialize a data generator for demonstration\ndgp = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\ngenerator = DataGenerator(n=1000, pi0=0.9, dgp=dgp)  # 10% alternatives\n\n# Create an online FDR procedure  \naddis = Addis(alpha=0.05, wealth=0.025, lambda_=0.25, tau=0.5)\n\n# Test hypotheses sequentially\ndiscoveries = []\nfor i in range(100):\n    p_value, label = generator.sample_one()\n    is_discovery = addis.test_one(p_value)\n    \n    if is_discovery:\n        discoveries.append(i)\n        print(f\"Discovery at test {i}: p-value = {p_value:.4f}\")\n\nprint(f\"Made {len(discoveries)} discoveries\")\n```\n\n## Implemented Methods\n\n### Sequential Testing Methods\n\nMethods that test one hypothesis at a time:\n\n#### **Alpha Investing Family**\n- **Generalized Alpha Investing (GAI)**: `from online_fdr.investing.alpha.alpha import Gai`\n- **SAFFRON**: `from online_fdr.investing.saffron.saffron import Saffron`  \n- **ADDIS**: `from online_fdr.investing.addis.addis import Addis`\n\n#### **LORD Family**\n- **LORD3**: `from online_fdr.investing.lord.three import LordThree`\n- **LORD++**: `from online_fdr.investing.lord.plus_plus import LordPlusPlus`\n- **D-LORD**: `from online_fdr.investing.lord.dependent import LordDependent`\n- **LORD with Discard**: `from online_fdr.investing.lord.discard import LordDiscard`\n- **LORD with Memory Decay**: `from online_fdr.investing.lord.mem_decay import LORDMemoryDecay`\n\n#### **LOND Family**\n- **LOND**: `from online_fdr.investing.lond.lond import Lond`\n\n#### **Alpha Spending**\n- **Alpha Spending**: `from online_fdr.spending.alpha_spending import AlphaSpending`\n- **Online Fallback**: `from online_fdr.spending.online_fallback import OnlineFallback`\n\n### Batch Testing Methods\n\nMethods that test hypotheses in batches:\n\n- **BatchBH**: `from online_fdr.batching.bh import BatchBH`\n- **BatchStoreyBH**: `from online_fdr.batching.storey_bh import BatchStoreyBH`\n- **BatchPRDS**: `from online_fdr.batching.prds import BatchPRDS`\n- **BatchBY**: `from online_fdr.batching.by import BatchBY`\n\n## Usage Examples\n\n### 1. **Alpha Investing (GAI)**\n\n```python\nfrom online_fdr.investing.alpha.alpha import Gai\nfrom online_fdr.utils.generation import DataGenerator, GaussianLocationModel\n\n# Note: GAI requires a wealth parameter\ngai = Gai(alpha=0.05, wealth=0.025)\n\n# Generate test data\ndgp = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\ngenerator = DataGenerator(n=100, pi0=0.9, dgp=dgp)\n\n# Test sequentially\nfor i in range(100):\n    p_value, true_label = generator.sample_one()\n    is_discovery = gai.test_one(p_value)\n    print(f\"Test {i}: p={p_value:.4f}, Discovery={is_discovery}\")\n```\n\n### 2. **LOND for Independent and Dependent P-values**\n\n```python\nfrom online_fdr.investing.lond.lond import Lond\nfrom online_fdr.utils.generation import DataGenerator, GaussianLocationModel\n\n# For independent p-values\nlond_indep = Lond(alpha=0.05)\n\n# For dependent p-values  \nlond_dep = Lond(alpha=0.05, dependent=True)\n\n# Generate test data\ndgp = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\ngenerator = DataGenerator(n=100, pi0=0.85, dgp=dgp)\n\nprint(\"LOND Independent:\")\ndiscoveries_indep = []\nfor i in range(50):\n    p_value, true_label = generator.sample_one()\n    result = lond_indep.test_one(p_value)\n    if result:\n        discoveries_indep.append(i)\n        print(f\"  Discovery at test {i}: p={p_value:.4f}\")\n\nprint(f\"\\nIndependent LOND made {len(discoveries_indep)} discoveries\")\n\n# Reset generator for dependent test\ngenerator = DataGenerator(n=100, pi0=0.85, dgp=dgp)\nprint(\"\\nLOND Dependent:\")\ndiscoveries_dep = []\nfor i in range(50):\n    p_value, true_label = generator.sample_one()\n    result = lond_dep.test_one(p_value)\n    if result:\n        discoveries_dep.append(i)\n        print(f\"  Discovery at test {i}: p={p_value:.4f}\")\n\nprint(f\"\\nDependent LOND made {len(discoveries_dep)} discoveries\")\n```\n\n### 3. **LORD with Memory Decay for Time Series**\n\n```python\nfrom online_fdr.investing.lord.mem_decay import LORDMemoryDecay\nfrom online_fdr.utils.evaluation import MemoryDecayFDR\nfrom online_fdr.utils.generation import GaussianLocationModel, DataGenerator\n\n# For non-stationary time series with decay\nlord_decay = LORDMemoryDecay(alpha=0.1, delta=0.99, eta=0.5)\n\n# Track memory-decay FDR  \nmem_fdr = MemoryDecayFDR(delta=0.99, offset=0)\n\n# Generate test data with higher alternative proportion for more discoveries\ndgp = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\ngenerator = DataGenerator(n=200, pi0=0.90, dgp=dgp)\n\ndiscoveries = []\nfdr_values = []\n\nprint(\"LORD Memory Decay Testing:\")\nfor i in range(100):\n    p_value, true_label = generator.sample_one()\n    is_discovery = lord_decay.test_one(p_value)\n    fdr = mem_fdr.score_one(is_discovery, true_label)\n    \n    if is_discovery:\n        discoveries.append(i)\n        print(f\"  Discovery at test {i}: p={p_value:.4f}, FDR={fdr:.4f}\")\n    \n    fdr_values.append(fdr)\n\nprint(f\"\\nTotal discoveries: {len(discoveries)}\")\nprint(f\"Final memory-decay FDR: {fdr_values[-1]:.4f}\")\nprint(f\"Average FDR over sequence: {sum(fdr_values)/len(fdr_values):.4f}\")\n```\n\n### 4. **Batch Testing**\n\n```python\nfrom online_fdr.batching.storey_bh import BatchStoreyBH\nfrom online_fdr.utils.generation import GaussianLocationModel, DataGenerator\n\nbatch_proc = BatchStoreyBH(alpha=0.1, lambda_=0.5)\n\n# Generate test data with higher alternative proportion for more discoveries\ndgp = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\ngenerator = DataGenerator(n=200, pi0=0.85, dgp=dgp)\n\n# Process multiple batches to demonstrate batch testing\nbatch_size = 25\ntotal_discoveries = 0\ntotal_false_discoveries = 0\n\nprint(\"Batch Storey-BH Testing:\")\nfor batch_num in range(3):\n    p_values, labels = [], []\n    \n    # Generate one batch\n    for _ in range(batch_size):\n        p_value, label = generator.sample_one()\n        p_values.append(p_value)\n        labels.append(label)\n    \n    # Test entire batch at once\n    results = batch_proc.test_batch(p_values)\n    discoveries = sum(results)\n    \n    # Calculate false discoveries\n    false_discoveries = sum(1 for r, l in zip(results, labels) if r and not l)\n    batch_fdr = false_discoveries / discoveries if discoveries \u003e 0 else 0.0\n    \n    print(f\"  Batch {batch_num + 1}: {discoveries} discoveries, FDR = {batch_fdr:.4f}\")\n    print(f\"    Significant p-values: {[f'{p:.4f}' for p, r in zip(p_values, results) if r]}\")\n    \n    total_discoveries += discoveries\n    total_false_discoveries += false_discoveries\n\noverall_fdr = total_false_discoveries / total_discoveries if total_discoveries \u003e 0 else 0.0\nprint(f\"\\nOverall: {total_discoveries} discoveries, FDR = {overall_fdr:.4f}\")\n```\n\n## Evaluation and Utilities\n\nThe library provides evaluation utilities to assess performance:\n\n```python\nfrom online_fdr.utils.evaluation import calculate_sfdr, calculate_power\nfrom online_fdr.utils.format import format_result\n\n# Example: Evaluate ADDIS performance\nfrom online_fdr.investing.addis.addis import Addis\nfrom online_fdr.utils.generation import DataGenerator, GaussianLocationModel\n\ndgp = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\ngenerator = DataGenerator(n=100, pi0=0.9, dgp=dgp)\naddis = Addis(alpha=0.05, wealth=0.025, lambda_=0.25, tau=0.5)\n\ntrue_positive = 0\nfalse_positive = 0\nfalse_negatives = 0\n\nfor i in range(100):\n    p_value, true_label = generator.sample_one()\n    result = addis.test_one(p_value)\n    \n    # Update counters\n    true_positive += true_label and result\n    false_positive += not true_label and result\n    false_negatives += true_label and not result\n    \n    # Optional: Format output\n    format_result(i, result, p_value, addis.alpha)\n\n# Calculate performance metrics\nsfdr = calculate_sfdr(tp=true_positive, fp=false_positive)\npower = calculate_power(tp=true_positive, fn=false_negatives)\n\nprint(f\"Empirical sFDR: {sfdr:.4f}\")\nprint(f\"Empirical Power: {power:.4f}\")\n```\n\n## Available Data Generation Models\n\nThe library includes several data generation models for testing:\n\n```python\nfrom online_fdr.utils.generation import (\n    DataGenerator, \n    GaussianLocationModel,\n    BetaMixtureModel, \n    ChiSquaredModel,\n    SparseGaussianModel\n)\n\n# Gaussian location model (most common for power analysis)\ndgp1 = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\n\n# Beta mixture model (common in genomics)\ndgp2 = BetaMixtureModel(alt_alpha=0.5, alt_beta=10.0)\n\n# Chi-squared model (for variance/goodness-of-fit testing)\ndgp3 = ChiSquaredModel(df=1, alt_scale=3.0)\n\n# Sparse Gaussian model (for screening applications)\ndgp4 = SparseGaussianModel(effect_dist=\"uniform\", min_effect=2.0, max_effect=5.0)\n\n# Example usage with different models\nprint(\"Testing different data generation models:\")\n\nfor i, (name, dgp) in enumerate([\n    (\"Gaussian Location\", dgp1),\n    (\"Beta Mixture\", dgp2), \n    (\"Chi-squared\", dgp3),\n    (\"Sparse Gaussian\", dgp4)\n]):\n    generator = DataGenerator(n=100, pi0=0.9, dgp=dgp)\n    # Sample a few p-values to demonstrate\n    sample_p_values = [generator.sample_one()[0] for _ in range(5)]\n    print(f\"  {name}: {[f'{p:.4f}' for p in sample_p_values]}\")\n```\n\n## Advanced Usage\n\n### Alpha Spending with Custom Functions\n\n```python\nfrom online_fdr.spending.alpha_spending import AlphaSpending\nfrom online_fdr.spending.functions.bonferroni import Bonferroni\nfrom online_fdr.investing.lord.three import LordThree\nfrom online_fdr.utils.generation import DataGenerator, GaussianLocationModel\n\n# Generate test data\ndgp = GaussianLocationModel(alt_mean=3.0, alt_std=1.0, one_sided=True)\ngenerator = DataGenerator(n=100, pi0=0.9, dgp=dgp)\n\n# Compare Bonferroni spending vs. adaptive LORD3\nk = 50  # Expected number of tests\nalpha = 0.05\n\n# Bonferroni spending: equal alpha allocation\nbonf_spending = AlphaSpending(alpha=alpha, spend_func=Bonferroni(k))\n\n# LORD3 investing: adaptive thresholds (this is the proper LORD3)\nlord3_adaptive = LordThree(alpha=alpha, wealth=0.025, reward=0.025)\n\nprint(\"Alpha Spending vs. Adaptive LORD3 Comparison:\")\nprint(f\"Overall alpha level: {alpha}\")\nprint(f\"Expected tests: {k}\")\n\nbonf_discoveries = []\nlord3_discoveries = []\n\n# Reset generator for fair comparison\ngenerator = DataGenerator(n=100, pi0=0.9, dgp=dgp)\n\nfor i in range(20):\n    p_value, true_label = generator.sample_one()\n    \n    # Test with both methods\n    bonf_result = bonf_spending.test_one(p_value)\n    lord3_result = lord3_adaptive.test_one(p_value)\n    \n    if bonf_result:\n        bonf_discoveries.append(i)\n    if lord3_result:\n        lord3_discoveries.append(i)\n    \n    # Show alpha thresholds for first few tests\n    if i \u003c 5:\n        bonf_threshold = alpha / k\n        lord3_threshold = lord3_adaptive.alpha\n        print(f\"  Test {i+1}: p={p_value:.4f}\")\n        print(f\"    Bonferroni ={bonf_threshold:.6f}, reject={bonf_result}\")\n        print(f\"    LORD3 ={lord3_threshold:.6f}, reject={lord3_result}\")\n\nprint(f\"\\nBonferroni discoveries: {bonf_discoveries}\")\nprint(f\"LORD3 adaptive discoveries: {lord3_discoveries}\")\nprint(f\"LORD3 typically shows higher power, especially early in the sequence\")\n```\n\n## Key Features\n\n- **True Online API**: Make decisions sequentially as p-values arrive\n- **Unified Interface**: All methods use `test_one()` for sequential testing\n- **Batch Support**: Batch methods use `test_batch()` for multiple p-values\n- **Rich Data Generation**: Multiple data generation models for testing\n- **Performance Evaluation**: Built-in utilities for calculating sFDR and power\n- **Light-weight**: Minimal external dependencies\n\n## Mathematical Guarantees\n\nGuarantees are method-specific and assumption-specific.\n\n- **Proven FDR/FWER guarantees** are provided where the algorithm and parameter regime match published theory.\n- **Parity methods** align behavior with the `onlineFDR` reference implementation for overlapping scope.\n- **Extension/experimental methods** are documented explicitly and should not be interpreted as universally guaranteed.\n\nSee the full matrix: `docs/theory/guarantee_matrix.md`.\n\n## onlineFDR Parity and Differences\n\nThis package is grounded against Bioconductor `onlineFDR` release semantics for\noverlapping procedures (see `docs/user_guide/onlinefdr_parity.md`).\n\n- **Parity**: ADDIS, SAFFRON, LORD family variants, LOND, Alpha-investing,\n  Alpha-spending, online-fallback, BatchBH, BatchPRDS, BatchStoreyBH.\n- **Intentional API divergence**: true stateful `test_one`/`test_batch`\n  interface instead of wrapper-style dataset reprocessing.\n- **Not currently mirrored**: internal same-date randomization and\n  asynchronous `*star` wrappers from the R ecosystem.\n\n### Mandatory Live R Parity Checks\n\nThe parity suite compares Python outputs directly against the live R\n`onlineFDR` package via `rpy2` (`tests/test_onlinefdr_parity.py`).\n\nDevelopment and CI require all of the following:\n\n- `rpy2` (installed through `uv sync --group dev`)\n- A system R installation on PATH\n- Bioconductor `onlineFDR` pinned to `2.18.0`\n- R `4.5.x` (required for Bioconductor `3.22`)\n\nIf any requirement is missing, parity tests fail with setup instructions.\nOn Ubuntu/WSL, if `rpy2` build fails with `cannot find -ltirpc`, install\n`libtirpc-dev`.\n\nIn CI, the full suite can run in a containerized R+Python environment through\n`.github/workflows/tests-container.yml`, which installs pinned `onlineFDR`\nautomatically before running `pytest`.\n## Acknowledgements\n\nThis library is inspired by and validated against the R package [onlineFDR](https://dsrobertson.github.io/onlineFDR/). \n\n**Key differentiator**: This implementation provides a truly online API with `test_one()` method calls, enabling real-time sequential applications (the R onlineFDR package requires pre-collected data).\n\n## License\n\nThis project is licensed under the BSD 3-Clause License - see the [LICENSE](LICENSE) file for details.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliverhennhoefer%2Fonline-fdr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foliverhennhoefer%2Fonline-fdr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliverhennhoefer%2Fonline-fdr/lists"}