{"id":33251845,"url":"https://github.com/OpenMined/PipelineDP","last_synced_at":"2025-11-21T17:01:56.105Z","repository":{"id":37400258,"uuid":"337809546","full_name":"OpenMined/PipelineDP","owner":"OpenMined","description":"PipelineDP is a Python framework for applying differentially private aggregations to large datasets using batch processing systems such as Apache Spark, Apache Beam, and more.","archived":false,"fork":false,"pushed_at":"2025-11-09T14:53:33.000Z","size":3002,"stargazers_count":281,"open_issues_count":32,"forks_count":84,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-11-09T16:22:45.899Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pipelinedp.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenMined.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"contributing/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"openmined"}},"created_at":"2021-02-10T18:04:22.000Z","updated_at":"2025-11-09T14:53:37.000Z","dependencies_parsed_at":"2024-01-15T17:40:22.878Z","dependency_job_id":"897f4258-51b7-4fa6-b183-323d03e07a63","html_url":"https://github.com/OpenMined/PipelineDP","commit_stats":{"total_commits":394,"total_committers":36,"mean_commits":"10.944444444444445","dds":0.5,"last_synced_commit":"52e210bbce0504d062ae3cd64e4145a1237a309b"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/OpenMined/PipelineDP","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2FPipelineDP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2FPipelineDP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2FPipelineDP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2FPipelineDP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenMined","download_url":"https://codeload.github.com/OpenMined/PipelineDP/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2FPipelineDP/sbom","scorecard":{"id":105425,"data":{"date":"2025-08-11","repo":{"name":"github.com/OpenMined/PipelineDP","commit":"0856dfd9624b0e9af93ffccad5bc853f7ca2d957"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.5,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":7,"reason":"9 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 7","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/publish.yml:1","Warn: no topLevel permission defined: .github/workflows/tests.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/publish.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/publish.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/publish.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/publish.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish.yml:60: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/publish.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/tests.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/tests.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:46: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/tests.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:50: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenMined/PipelineDP/tests.yml/main?enable=pin","Warn: containerImage not pinned by hash: contributing/Dockerfile:1: pin your Docker image by updating python:3.11-slim to python:3.11-slim@sha256:0ce77749ac83174a31d5e107ce0cfa6b28a2fd6b0615e029d9d84b39c48976ee","Warn: pipCommand not pinned by hash: contributing/Dockerfile:16","Warn: pipCommand not pinned by hash: contributing/Dockerfile:19","Warn: pipCommand not pinned by hash: Makefile:26","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:48","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:52","Warn: pipCommand not pinned by hash: .github/workflows/publish.yml:78","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:28","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:29","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:56","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:57","Info:   0 out of   7 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of  10 pipCommand dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":0,"reason":"17 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2018-34 / GHSA-2fc2-6r4j-p65h","Warn: Project is vulnerable to: PYSEC-2021-856 / GHSA-5545-2q6w-2gh6","Warn: Project is vulnerable to: PYSEC-2019-108 / GHSA-9fq2-x9r6-wfmf","Warn: Project is vulnerable to: PYSEC-2018-33 / GHSA-cw6w-4rcx-xphc","Warn: Project is vulnerable to: PYSEC-2021-857 / GHSA-f7c7-j99h-c22f","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: PYSEC-2017-1 / GHSA-frgw-fgh6-9g52","Warn: Project is vulnerable to: PYSEC-2020-73","Warn: Project is vulnerable to: PYSEC-2023-44 / GHSA-329j-jfvr-rhr6","Warn: Project is vulnerable to: PYSEC-2022-42976 / GHSA-43xg-8wmj-cw8h","Warn: Project is vulnerable to: PYSEC-2022-236 / GHSA-4x9r-j582-cgr8","Warn: Project is vulnerable to: PYSEC-2018-25 / GHSA-6mqq-8r44-vmjc","Warn: Project is vulnerable to: PYSEC-2017-147 / GHSA-8rhc-48pp-52gr","Warn: Project is vulnerable to: PYSEC-2022-186 / GHSA-9rr6-jpg7-9jg6","Warn: Project is vulnerable to: PYSEC-2019-114 / GHSA-fp5j-3fpf-mhj5","Warn: Project is vulnerable to: PYSEC-2020-95 / GHSA-wgx7-jwwm-cgjv","Warn: Project is vulnerable to: PYSEC-2023-72"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-15T11:00:03.731Z","repository_id":37400258,"created_at":"2025-08-15T11:00:03.732Z","updated_at":"2025-08-15T11:00:03.732Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285655011,"owners_count":27209077,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-21T02:00:06.175Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-17T00:00:37.460Z","updated_at":"2025-11-21T17:01:56.099Z","avatar_url":"https://github.com/OpenMined.png","language":"Python","funding_links":["https://github.com/sponsors/openmined"],"categories":["5.4 DP Libraries","Code and Projects"],"sub_categories":[],"readme":"# PipelineDP\n\nPipelineDP is a framework for applying differentially private aggregations to large\ndatasets using batch processing systems such as Apache Spark, Apache Beam,\nand more.\n\nTo make differential privacy accessible to non-experts, PipelineDP:\n\n* Provides a convenient API familiar to Spark or Beam developers.\n* Encapsulates the complexities of differential privacy, such as:\n  * protecting outliers and rare categories,\n  * generating safe noise,\n  * privacy budget accounting.\n* Supports many standard computations, such as count, sum, and average. \n\nAdditional information can be found at [pipelinedp.io](https://pipelinedp.io).\n\n*Note* that this project is still experimental and is subject to change.\nAt the moment we don't recommend its usage in production systems as it's not\nthoroughly tested yet. You can learn more in the\n[Roadmap section](https://pipelinedp.io/overview/#roadmap).\n\nThe project is a collaboration between OpenMined and Google in an effort \nto bring Differential Privacy to production.\n\n## Getting started\n\nHere are some examples of how to use PipelineDP:\n\n* [Apache Spark example](https://github.com/OpenMined/PipelineDP/blob/main/examples/movie_view_ratings/run_on_spark.py)\n* [Apache Beam example](https://github.com/OpenMined/PipelineDP/blob/main/examples/movie_view_ratings/run_on_beam.py)\n* [Framework-free example](https://github.com/OpenMined/PipelineDP/blob/main/examples/movie_view_ratings/run_without_frameworks.py)\n* [Example with all frameworks](https://github.com/OpenMined/PipelineDP/blob/main/examples/movie_view_ratings/run_all_frameworks.py)\n\nPlease check out the [codelab](https://github.com/OpenMined/PipelineDP/blob/main/examples/restaurant_visits.ipynb) for a more detailed demonstration of the API functionality and usage.\n\nCode sample showing private processing on Spark:\n```python\n# Define the privacy budget available for our computation.\nbudget_accountant = pipeline_dp.NaiveBudgetAccountant(total_epsilon=1,\n                                                      total_delta=1e-6)\n\n# Wrap Spark's RDD into its private version. You will use this private wrapper\n# for all further processing instead of the Spark's RDD. Using the wrapper ensures\n# that only private statistics can be released.\nprivate_movie_views = \\\n    make_private(movie_views, budget_accountant, lambda mv: mv.user_id)\n\n# Calculate the private sum of ratings per movie\ndp_result = private_movie_views.sum(\n    SumParams(\n              # The aggregation key: we're grouping data by movies\n              partition_extractor=lambda mv: mv.movie_id,\n              # The value we're aggregating: we're summing up ratings\n              value_extractor=lambda mv: mv.rating,\n\n              # Limits to how much one user can contribute:\n              # .. at most two movies rated per user\n              #    (if there's more, randomly choose two)\n              max_partitions_contributed=2,\n              # .. at most one ratings for each movie\n              max_contributions_per_partition=1,\n              # .. with minimal rating of \"1\"\n              #    (automatically clip the lesser values to \"1\")\n              min_value=1,\n              # .. and maximum rating of \"5\"\n              #    (automatically clip the greater values to \"5\")\n              max_value=5)\n              )\nbudget_accountant.compute_budgets()\n\n# Save the results\ndp_result.saveAsTextFile(FLAGS.output_file)\n```\n\n## Installation\n\nPipelineDP without any frameworks:\n\n`pip install pipeline-dp`\n\nIf you like to run PipelineDP on Apache Spark:\n\n`pip install pipeline-dp pyspark`\n\non Apache Beam:\n\n`pip install pipeline-dp apache-beam`.\n\nSupported Python version \u003e= 3.8.\n\n**Note for Apple Silicon users:** PipelineDP pip package is currently available only \nfor x86 architecture. The reason is that [PyDP](https://github.com/OpenMined/PyDP) does not\nhave pip pacakge. It might be possible to compile it from sources for Apple Silicon.\n\n## Attack Model\n\nPipelineDP has the same [attack model](https://github.com/google/differential-privacy/blob/main/common_docs/attack_model.md)\nas the Google Differential Privacy Library.\n \n## Development\n\nTo setup a local environment and contribute with the development of PipelineDP, please see our guidelines in [CONTRIBUTING](https://github.com/OpenMined/PipelineDP/blob/main/contributing/CONTRIBUTING.md).\n\n## Support and Community on Slack\n\nIf you have questions about the PipelineDP, join\n[OpenMined's Slack](https://slack.openmined.org) and check the\n**#differential-privacy** channel.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenMined%2FPipelineDP","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenMined%2FPipelineDP","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenMined%2FPipelineDP/lists"}