{"id":34085575,"url":"https://github.com/shyaginuma/casual_inference","last_synced_at":"2025-12-14T13:06:43.277Z","repository":{"id":59649350,"uuid":"527752476","full_name":"shyaginuma/casual_inference","owner":"shyaginuma","description":"Do causal inference more casually.","archived":false,"fork":false,"pushed_at":"2025-02-01T06:27:48.000Z","size":668,"stargazers_count":26,"open_issues_count":9,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-01T07:22:11.535Z","etag":null,"topics":["ab-testing","causal-inference","data-science","python","statistics"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shyaginuma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-22T22:34:13.000Z","updated_at":"2025-01-16T02:51:32.000Z","dependencies_parsed_at":"2024-11-23T10:01:34.849Z","dependency_job_id":null,"html_url":"https://github.com/shyaginuma/casual_inference","commit_stats":{"total_commits":47,"total_committers":3,"mean_commits":"15.666666666666666","dds":0.4893617021276596,"last_synced_commit":"906159160db4ebfc980789ed7c0fe24593f802dd"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/shyaginuma/casual_inference","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyaginuma%2Fcasual_inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyaginuma%2Fcasual_inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyaginuma%2Fcasual_inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyaginuma%2Fcasual_inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shyaginuma","download_url":"https://codeload.github.com/shyaginuma/casual_inference/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyaginuma%2Fcasual_inference/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27728829,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-14T02:00:11.348Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","causal-inference","data-science","python","statistics"],"created_at":"2025-12-14T13:06:38.689Z","updated_at":"2025-12-14T13:06:43.264Z","avatar_url":"https://github.com/shyaginuma.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![ci](https://github.com/shyaginuma/casual_inference/actions/workflows/lint_and_test.yml/badge.svg)](https://github.com/shyaginuma/casual_inference/actions/workflows/lint_and_test.yml)\n[![PyPI version](https://badge.fury.io/py/casual_inference.svg)](https://badge.fury.io/py/casual_inference)\n[![Downloads](https://static.pepy.tech/badge/casual-inference/month)](https://pepy.tech/project/casual-inference)\n\n# casual_inference\n\nThe `casual_inference` is a Python package provides a simple interface to do causal inference.\nDoing causal analyses is a complicated stuff. We have to pay attention to many things to do such analyses properly.\nThe `casual_inference` is developed aiming to reduce such effort.\n\n## Installation\n\n```shell\npip install casual-inference\n```\n\n## Overview\n\nThis package will provide several types of **`evaluator`**. They have `evaluate()` and some `summary_xxx()` methods. The `evaluate()` method evaluates treatment impact by calculating several statistics in it, and the `summary_xxx()` methods summarize such statistics in some ways. (e.g., table style, bar chart style, ...)\n\nThe `evaluate()` method expects that the data which has a schema like as follows will be passed.\n\n|unit|variant|metric_A|metric_B|...|\n|----|-------|--------|--------|---|\n|1   |1      |0       |0.01    |...|\n|2   |1      |1       |0.05    |...|\n|3   |2      |0       |0.02    |...|\n|... |...    |...     |...     |...|\n\nThe table has been already aggregated by the `unit` column. (i.e. The `unit` column should be the primary key)\n\n### Columns\n\n- `unit`: The unit you want to conduct analysis on. Typically it will be user_id, session_id, ... in the web service domain.\n- `variant`: The group of intervention. This package always assumes `1` is a variant of control group.\n- `metrics`: metrics you want to evaluate. e.g., The number of purchases, conversion rate, ...\n\n## Quick Start\n\nThe `casual_inference` supports not only the evaluation of normal A/B testing and A/A testing, but also advanced causal inference techniques.\n\n### A/B test evaluation\n\n```python\nfrom casual_inference.dataset import create_sample_ab_result\nfrom casual_inference.evaluator import ABTestEvaluator\n\ndata = create_sample_ab_result(n_variant=3, sample_size=1000000, simulated_lift=[-0.01, 0.01])\n\nevaluator = ABTestEvaluator()\nevaluator.evaluate(\n    data=data,\n    unit_col=\"rand_unit\",\n    variant_col=\"variant\",\n    metrics=[\"metric_bin\", \"metric_cont\"]\n)\n\nevaluator.summary_plot()\n```\n\n![eval_result](examples/images/plot_abtestevaluator_result.png)\n\nIt diagnoses [Sample Ratio Mismatch](https://dl.acm.org/doi/10.1145/3292500.3330722) (SRM) automatically. When it detects the SRM, it'll display a warning on the output so that the Analyst can interpret the result carefully.\n\nYou can also see the [example notebook](https://github.com/shyaginuma/casual_inference/blob/main/examples/ab_test_evaluator.ipynb) to see more detailed example.\n\n### A/A test evaluation\n\n```python\nfrom casual_inference.dataset import create_sample_ab_result\nfrom casual_inference.evaluator import AATestEvaluator\n\ndata = create_sample_ab_result(n_variant=2, sample_size=1000000, simulated_lift=[0.0])\n\nevaluator = AATestEvaluator()\nevaluator.evaluate(\n    data=data,\n    unit_col=\"rand_unit\",\n    metrics=[\"metric_bin\", \"metric_cont\"]\n)\n\nevaluator.summary_plot()\n```\n\n![eval_result](examples/images/plot_aatestevaluator_result.png)\n\nYou can also see the [example notebook](https://github.com/shyaginuma/casual_inference/blob/main/examples/aa_test_evaluator.ipynb) to see more detailed example.\n\n### Sample Size evaluation\n\n```python\nfrom casual_inference.dataset import create_sample_ab_result\nfrom casual_inference.evaluator import SampleSizeEvaluator\n\ndata = create_sample_ab_result(n_variant=2, sample_size=1000000)\n\nevaluator = SampleSizeEvaluator()\nevaluator.evaluate(\n    data=data,\n    unit_col=\"rand_unit\",\n    metrics=[\"metric_bin\", \"metric_cont\"]\n)\n\nevaluator.summary_plot()\n```\n\n![eval_result](examples/images/plot_samplesizeevaluator_result.png)\n\nYou can also see the [example notebook](https://github.com/shyaginuma/casual_inference/blob/main/examples/sample_size_evaluator.ipynb) to see more detailed example.\n\n### Advanced causal inference techniques\n\nIt also supports advanced causal inference techniques.\n\n- Linear Regression\n\nAnother evaluation methods like Propensity Score Matching are planed to implement in the future.\n\n## References\n\n- Kohavi, Ron, Diane Tang, and Ya Xu. 2020. ​Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. https://experimentguide.com/\n  - A Great book covering comprehensive topics around practical A/B testing. I do recommend to read this book for all people who works on A/B testing.\n- Alex Deng, Ulf Knoblich, and Jiannan Lu. 2018. Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery \u0026 Data Mining (KDD '18). Association for Computing Machinery, New York, NY, USA, 233–242. https://doi.org/10.1145/3219819.3219919\n  - Describing how to approximate variance of relative difference, and when the analysis unit was more granular than the randomization unit.\n- Lucile Lu. 2016. Power, minimal detectable effect, and bucket size estimation in A/B tests. Twitter Engineering Blog. [link](https://blog.twitter.com/engineering/en_us/a/2016/power-minimal-detectable-effect-and-bucket-size-estimation-in-ab-tests)\n  - Describing Concept around Type I error and Type II error, Power Analysis. (Sample size calculation)\n- Aleksander Fabijan, Jayant Gupchup, Somit Gupta, Jeff Omhover, Wen Qin, Lukas Vermeer, and Pavel Dmitriev. 2019. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery \u0026 Data Mining (KDD '19). Association for Computing Machinery, New York, NY, USA, 2156–2164. https://doi.org/10.1145/3292500.3330722\n  - Introduce Sample Ratio Mismatch (SRM) and describe various example of SRM happening, and provide taxonomy that help debugging when the SRM happened.\n- Shota Yasui. 2020. 効果検証入門. 技術評論社. https://gihyo.jp/book/2020/978-4-297-11117-5\n  - A Great introduction book about practical causal inference technique written in Japanese.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshyaginuma%2Fcasual_inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshyaginuma%2Fcasual_inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshyaginuma%2Fcasual_inference/lists"}