{"id":13704522,"url":"https://github.com/opendp/smartnoise-core","last_synced_at":"2025-05-05T10:30:26.769Z","repository":{"id":45128340,"uuid":"206400520","full_name":"opendp/smartnoise-core","owner":"opendp","description":"Differential privacy validator and runtime","archived":true,"fork":false,"pushed_at":"2022-01-06T17:26:12.000Z","size":111499,"stargazers_count":290,"open_issues_count":42,"forks_count":33,"subscribers_count":30,"default_branch":"develop","last_synced_at":"2024-11-08T18:05:16.242Z","etag":null,"topics":["differential-privacy","opendp","privacy","smartnoise"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opendp.png","metadata":{"files":{"readme":"README.md","changelog":"History.md","contributing":"contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-04T19:46:55.000Z","updated_at":"2024-08-22T14:00:42.000Z","dependencies_parsed_at":"2022-08-30T16:00:22.888Z","dependency_job_id":null,"html_url":"https://github.com/opendp/smartnoise-core","commit_stats":null,"previous_names":["opendifferentialprivacy/whitenoise-core"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendp%2Fsmartnoise-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendp%2Fsmartnoise-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendp%2Fsmartnoise-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opendp%2Fsmartnoise-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opendp","download_url":"https://codeload.github.com/opendp/smartnoise-core/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224175773,"owners_count":17268389,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["differential-privacy","opendp","privacy","smartnoise"],"created_at":"2024-08-02T21:01:11.391Z","updated_at":"2024-11-13T11:31:29.709Z","avatar_url":"https://github.com/opendp.png","language":"Rust","funding_links":[],"categories":["Tools"],"sub_categories":["Privacy"],"readme":"**Notice**: SmartNoise-Core is deprecated. Please migrate to the OpenDP library:\n- [OpenDP Rust Crate](https://crates.io/crates/opendp)\n- [OpenDP GitHub Repo](https://github.com/opendp/opendp/)\n\n----------------------------------------------------\n\n[![.github/workflows/rust.yml](https://github.com/opendp/smartnoise-core/actions/workflows/rust.yml/badge.svg)](https://github.com/opendp/smartnoise-core/actions/workflows/rust.yml) \n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n\n\n\n\u003ca href=\"http://opendp.org\"\u003e\u003cimg src=\"images/SmartNoise_Logos/SVG/LogoMark_color.svg\" align=\"left\" height=\"70\" vspace=\"8\" hspace=\"18\"\u003e\u003c/a\u003e\n\n\n## \u003cbr/\u003eSmartNoise Core Differential Privacy Library \u003cbr/\u003e\n\nPlease see the accompanying [SmartNoise Documentation](https://docs.smartnoise.org), [SmartNoise SDK repository](https://github.com/opendp/smartnoise-sdk), and [SmartNoise Samples repository](https://github.com/opendp/smartnoise-samples) accompanying repositories for this system.\n\n---\n\nDifferential privacy is the gold standard definition of privacy protection.  The SmartNoise project, in collaboration with OpenDP, aims to connect theoretical solutions from the academic community with the practical lessons learned from real-world deployments, to make differential privacy broadly accessible to future deployments.  Specifically, we provide several basic building blocks that can be used by people involved with sensitive data, with implementations based on vetted and mature differential privacy research.  Here in the Core, we provide a pluggable open source library of differentially private algorithms and mechanisms for releasing privacy preserving queries and statistics, as well as APIs for defining an analysis and a validator for evaluating these analyses and composing the total privacy loss on a dataset.\n\nThe mechanisms library provides a fast, memory-safe native runtime for validating and running differentially private analyses.  The runtime and validator are built in Rust, while Python support is available and R support is forthcoming.\n\nDifferentially private computations are specified as an analysis graph that can be validated and executed to produce differentially private releases of data.  Releases include metadata about accuracy of outputs and the complete privacy cost of the analysis.\n\n\n- [More about SmartNoise Core](#more-about-smartnoise-core)\n  - [Components](#components)\n  - [Architecture](#architecture)\n- [Installation](#installation)\n  - [Crates.io](#cratesio)\n  - [From Source](#from-source)\n- [Getting Started](#getting-started)\n  - [Jupyter Notebook Examples](#jupyter-notebook-examples)\n  - [SmartNoise Core Rust Documentation](#smartnoise-core-rust-documentation)\n- [Communication](#communication)\n- [Releases and Contributing](#releases-and-contributing)\n- [Contributing Team](#contributing-team)\n\n---\n\n## More about SmartNoise Core\n\n### Components\n\nThe primary releases available in the library, and the mechanisms for generating these releases, are enumerated below.\nFor a full listing of the extensive set of components available in the library [see this documentation.](https://opendp.github.io/smartnoise-core/doc/smartnoise_validator/docs/components/index.html)\n\n| Statistics    | Mechanisms | Utilities  |\n| ------------- |------------|------------|\n| Count         | Gaussian   | Cast       |\n| Histogram     | Geometric  | Clamping   |\n| Mean          | Laplace    | Digitize   |\n| Quantiles     |            | Filter     |\n| Sum           |            | Imputation |\n| Variance/Covariance |      | Transform  |\n\n\n\u003c!-- (forthcoming: links to minisite/research papers) --\u003e\n\n### Architecture\n\nThere are three sub-projects that address individual architectural concerns. These sub-projects communicate via protobuf messages that encode a graph description of an arbitrary computation, called an `analysis`.\n\n#### 1. Validator\n  - Location: `/validator-rust`\n\nThe core library, is the `validator`, which provides a suite of utilities for checking and deriving sufficient conditions for an analysis to be differentially private. This includes checking if specific properties have been met for each component, deriving sensitivities, noise scales and accuracies for various definitions of privacy, building reports and dynamically validating individual components. This library is written in Rust.\n\n#### 2. Runtime\n  - Location: `/runtime-rust`\n\nThere must also be a medium to execute the analysis, called a `runtime`. There is a reference runtime written in Rust, but runtimes may be written using any computation framework--be it SQL, Spark or Dask--to address your individual data needs.\n\n#### 3. Bindings\n  - Python Bindings: [core-python](https://github.com/opendp/smartnoise-core-python)\n  - Rust Bindings (in progress): [core-Rust](https://opendp.github.io/smartnoise-core/doc/smartnoise_validator/bindings/index.html)\n\nFinally, there are helper libraries for building analyses, called `bindings`. Bindings may be written for any language, and are thin wrappers over the validator and/or runtime(s). Language bindings are currently available for Python, with support for at minimum R, Rust and SQL forthcoming.\n\n\n#### Note on Protocol Buffers\n\n  - Location: `/validator-rust/prototypes`\n\nCommunication among projects is handled via [Protocol Buffer definitions](https://developers.google.com/protocol-buffers/) in the `/validator-rust/prototypes` directory. All three sub-projects implement:\n  - Protobuf code generation\n  - Protobuf serialization/deserialization\n  - Communication over [FFI](https://en.wikipedia.org/wiki/Foreign_function_interface)\n  - Handling of distributable packaging\n\nAt some point the projects have compiled cross-platform (more testing needed). The validator and reference runtime compile to standalone libraries that may be linked into your project, allowing communication over C foreign function interfaces.\n\n## Installation\nRefer to [troubleshooting.md](troubleshooting.md) for install problems.\n\n#### PyPi packages\nRefer to [core-python](https://github.com/opendp/smartnoise-core-python) which contains python bindings, including links to PyPi packages.\n\n#### Crates.io\nThe crates are intended for library consumers.\n\nThe Rust Validator and Runtime are available as crates:\n- Validator: [validator](https://crates.io/crates/smartnoise_validator) on crates.io\n- Runtime: [runtime](https://crates.io/crates/smartnoise_runtime) on crates.io\n\n### From Source\nThe source install is intended for library developers.\n\nYou may find it easier to use the library with this repository set up as a submodule of some set of language bindings.\nIn this case, switch to the language bindings setup.\nYou can still push commits and branches from the core submodule of whatever bindings language you prefer.\n- [Python](https://github.com/opendp/smartnoise-core-python#from-source)\n- [Rust (WIP)](https://opendp.github.io/smartnoise-core/doc/smartnoise_validator/bindings/index.html)\n\n1. Clone the repository\n\n        git clone git@github.com:opendp/smartnoise-core.git\n\n2. Install system dependencies (rust, gcc)   \n    Mac:\n    ```shell script\n    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n    xcode-select --install\n    ```\n\n    Linux:\n    ```shell script\n    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n    sudo apt-get install diffutils gcc make m4\n    ```\n\n    Windows:\n    Install WSL and refer to the linux instructions.\n\n3. In a new terminal:  \n    Build crate\n\n        cargo build\n\n    Test crate\n\n        cargo test\n\n    Document crate\n\n        cargo rustdoc --open\n\n    Build production docs\n\n        ./build_docs.sh\n\nThere are crates in `validator-rust` and `runtime-rust`, and a virtual crate in root that runs commands on both.\nSwitch between crates via `cd`, or by setting the manifest path `--manifest-path=validator-rust/Cargo.toml`.\n\n\n---\n## Getting Started\n\n### Jupyter Notebook Examples\n\nWe have [numerous Jupyter notebooks](https://github.com/opendp/smartnoise-samples/tree/master/analysis) demonstrating the use of the Core library and validator through our Python bindings.  These are in our accompanying [samples repository](https://github.com/opendp/smartnoise-samples) which has exemplars, notebooks and sample code demonstrating most facets of this project.\n\n[\u003cimg src=\"images/figs/plugin_mean_comparison.png\" alt=\"Relative error distributions\" height=\"100\"\u003e](https://github.com/opendp/smartnoise-samples/tree/master/analysis)\n[\u003cimg src=\"images/figs/example_size.png\" alt=\"Release box plots\" height=\"100\"\u003e](https://github.com/opendp/smartnoise-samples/tree/master/analysis)\n[\u003cimg src=\"images/figs/example_education.png\" alt=\"Histogram releases\" height=\"100\"\u003e](https://github.com/opendp/smartnoise-samples/tree/master/analysis)\n[\u003cimg src=\"images/figs/example_utility.png\" alt=\"Utility simulations\" height=\"100\"\u003e](https://github.com/opendp/smartnoise-samples/tree/master/analysis)\n[\u003cimg src=\"images/figs/example_simulations.png\" alt=\"Bias simulations\" height=\"100\"\u003e](https://github.com/opendp/smartnoise-samples/tree/master/analysis)\n\n### SmartNoise Core Rust Documentation\n\nThe [Rust documentation](https://opendp.github.io/smartnoise-core/) includes full documentation on all pieces of the library and validator, including extensive [component by component descriptions with examples](https://opendp.github.io/smartnoise-core/doc/smartnoise_runtime/components/index.html).\n\n## Communication\n\n- You are very welcome to join us on [GitHub Discussions](https://github.com/opendp/opendp/discussions/categories/smartnoise)!\n- Please use [GitHub Issues](https://github.com/opendp/smartnoise-core/issues) for bug reports and feature requests.\n- For other requests, including security issues, please contact us at [smartnoise@opendp.org](mailto:smartnoise@opendp.org).\n\n## Releases and Contributing\n\nPlease let us know if you encounter a bug by [creating an issue](https://github.com/opendp/smartnoise-core/issues).\n\nWe appreciate all contributions. We welcome pull requests with bug-fixes without prior discussion.\n\nIf you plan to contribute new features, utility functions or extensions to the core, please first open an issue and discuss the feature with us.\n  - Sending a PR without discussion might end up resulting in a rejected PR, because we may be taking the core in a different direction than you might be aware of.\n\n## Contributing Team\n\nJoshua Allen, Christian Covington, Ethan Cowan, Eduardo de Leon, Ira Globus-Harris, James Honaker, Jason Huang, Michael Phelan, Raman Prasad, Michael Shoemate, Saniya Vahedian Movahed, You?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendp%2Fsmartnoise-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendp%2Fsmartnoise-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendp%2Fsmartnoise-core/lists"}