{"id":13487528,"url":"https://github.com/trailofbits/PrivacyRaven","last_synced_at":"2025-03-27T22:31:31.161Z","repository":{"id":38142662,"uuid":"266141010","full_name":"trailofbits/PrivacyRaven","owner":"trailofbits","description":"Privacy Testing for Deep Learning","archived":false,"fork":false,"pushed_at":"2023-07-20T15:30:24.000Z","size":108872,"stargazers_count":183,"open_issues_count":37,"forks_count":17,"subscribers_count":32,"default_branch":"master","last_synced_at":"2024-08-01T18:30:40.000Z","etag":null,"topics":["deep-learning","machine-learning","membership-inference","model-extraction","model-inversion","privacy","privacy-enhancing-technologies","privacy-preserving-machine-learning","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trailofbits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-22T15:13:26.000Z","updated_at":"2024-07-31T12:36:01.000Z","dependencies_parsed_at":"2023-01-25T09:15:35.062Z","dependency_job_id":null,"html_url":"https://github.com/trailofbits/PrivacyRaven","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2FPrivacyRaven","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2FPrivacyRaven/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2FPrivacyRaven/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2FPrivacyRaven/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trailofbits","download_url":"https://codeload.github.com/trailofbits/PrivacyRaven/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222322034,"owners_count":16966433,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-learning","membership-inference","model-extraction","model-inversion","privacy","privacy-enhancing-technologies","privacy-preserving-machine-learning","python"],"created_at":"2024-07-31T18:01:00.330Z","updated_at":"2024-10-30T22:31:13.720Z","avatar_url":"https://github.com/trailofbits.png","language":"Python","funding_links":[],"categories":["Privacy Toolboxes","Security tools and techniques","Libraries","Tools","LLM SECURITY / AI SECURITY","Uncategorized","Securing AI SaaS","Defense \u0026 Security Controls"],"sub_categories":["API probing","**Outlines of this repo:**","AI Model Security \u0026 Privacy","Uncategorized","Privacy Tools","Privacy \u0026 Confidential Computing"],"readme":"![PrivacyRaven Logo](https://github.com/trailofbits/PrivacyRaven/blob/master/images/privacyraven_logo.jpeg) \n\u003chr style=\"height:5px\"/\u003e\n\nNote: This project is on hiatus. \n\n**PrivacyRaven** is a privacy testing library for deep learning systems.\nYou can use it to determine the susceptibility of a model to different privacy attacks; evaluate privacy preserving machine learning techniques; develop novel privacy metrics and attacks; and repurpose attacks for data provenance and other use cases.\n\nPrivacyRaven supports label-only black-box model extraction, membership inference, and (soon) model inversion attacks.\nWe also plan to include differential privacy verification, automated hyperparameter optimization, more classes of attacks, and other features; see the [GitHub issues](https://github.com/trailofbits/PrivacyRaven/issues) for more information.\nPrivacyRaven has been featured at the [OpenMined Privacy Conference](https://www.youtube.com/watch?v=F46lX5VIoas\u0026list=PLUNOsx6Az_ZGKQd_p4StdZRFQkCBwnaY6\u0026t=2h21m50s), [Empire Hacking](https://www.empirehacking.nyc/), and [Trail of Bits blog](https://blog.trailofbits.com/2020/10/08/privacyraven-has-left-the-nest/).\n\n## Why use PrivacyRaven?\n\nDeep learning systems, particularly neural networks, have proliferated in a wide range of applications, including privacy-sensitive use cases such as facial recognition and medical diagnoses.\nHowever, these models are vulnerable to privacy attacks that target both the intellectual property of the model and the confidentiality of the training data.\nRecent literature has seen an arms race between privacy attacks and defenses on various systems.\nAnd until now, engineers and researchers have not had the privacy analysis tools they need to rival this trend.\nHence, we developed PrivacyRaven- a machine learning assurance tool that aims to be:\n+ **Usable**: Multiple levels of abstraction allow users to either automate much of the internal mechanics or directly control them, depending on their use case and familiarity with the domain. \n+ **Flexible**: A modular design makes the attack configurations customizable and interoperable. It also allows new privacy metrics and attacks to be incorporated straightforwardly.\n+ **Efficient**: PrivacyRaven reduces the boilerplate, affording quick prototyping and fast experimentation. Each attack can be launched in fewer than 15 lines of code.\n\n## How does it work?\n\nPrivacyRaven partitions each attack into multiple customizable and optimizable phases.\nDifferent interfaces are also provided for each attack.\nThe interface shown below is known as the core interface.\nPrivacyRaven also provides wrappers around specific attack configurations found in the literature and a run-all-attacks feature.\n\nHere is how you would launch a model extraction attack in PrivacyRaven:\n\n```python\n#examples/extract_mnist_gpu.py\nimport privacyraven as pr\nfrom privacyraven.utils.data import get_emnist_data\nfrom privacyraven.extraction.core import ModelExtractionAttack\nfrom privacyraven.utils.query import get_target\nfrom privacyraven.models.victim import train_four_layer_mnist_victim\nfrom privacyraven.models.four_layer import FourLayerClassifier\n\n# Create a query function for a target PyTorch Lightning model\nmodel = train_four_layer_mnist_victim()\n\n\ndef query_mnist(input_data):\n    # PrivacyRaven provides built-in query functions\n    return get_target(model, input_data, (1, 28, 28, 1))\n\n\n# Obtain seed (or public) data to be used in extraction\nemnist_train, emnist_test = get_emnist_data()\n\n# Run a model extraction attack\nattack = ModelExtractionAttack(\n    query_mnist, # query function\n    200,  # query limit\n    (1, 28, 28, 1), # victim input shape\n    10, # number of targets\n    (3, 1, 28, 28),  # substitute input shape\n    \"copycat\", # synthesizer name\n    FourLayerClassifier, # substitute model architecture\n    784,  # substitute input size\n    emnist_train, # seed train data\n    emnist_test, # seed test data\n)\n```\nSince the only main requirement from the victim model is a query function, PrivacyRaven can be used to attack a wide range of models regardless of the framework and distribution method.\nThe other classes of attacks can be launched in a similar fashion. See the `examples` folder for more information.\n\n## Want to use PrivacyRaven?\n\n1. Install [poetry](https://python-poetry.org/docs/).\n2. Git clone this repository.\n3. Run `poetry update`\n4. Run `poetry install`.\n\nIf you'd like to use a Jupyter Notebook environment, run `poetry shell` followed by `jupyter notebook`.\n\nAdditionally, if you'd like to run PrivacyRaven in a Docker container, run `chmod +x build.sh` followed by `./build.sh`.  Note that depending on the amount of resources you allocate to Docker, PrivacyRaven's performance may be drastically impacted.\n\nFeel free to join our #privacyraven channel in [Empire Hacking](https://slack.empirehacking.nyc/) if you need help using or extending PrivacyRaven.\nThe official pip release will arrive soon.\n\nPlease note that PrivacyRaven is still early in development and is undergoing rapid changes. Users are advised to update frequently and avoid applying PrivacyRaven to critical use cases. \n\n## Want to contribute to PrivacyRaven?\n\nPrivacyRaven is still a work-in-progress.\nWe invite you to contribute however you can whether you want to incorporate a new synthesis technique or make an attack function more readable.\nPlease visit [CONTRIBUTING.md](https://github.com/trailofbits/PrivacyRaven/blob/master/CONTRIBUTING.md) to get started.\n\n## Why is it called PrivacyRaven?\n\nThe raven has been associated with a variety of concepts in different cultures through time.\nAmong these, the raven is commonly associated with prophecy and insight.\nNaturally, we named the tool PrivacyRaven because it is designed to provide insights into the privacy of deep learning.\n\n## Who maintains PrivacyRaven?\n\nThe core maintainers are:\n+ [Suha S. Hussain](https://github.com/suhacker1)\n+ [Philip Wang](https://github.com/pwang00) \n+ [Jim Miller](https://github.com/james-miller-93)\n\n## License\n\nThis library is available under the [Apache License 2.0](https://github.com/trailofbits/PrivacyRaven/blob/master/LICENSE).\nFor an exception to the terms, please [contact us](mailto:opensource@trailofbits.com).\n\n## References\n\nWhile PrivacyRaven was built upon a [plethora of research](https://github.com/stratosphereips/awesome-ml-privacy-attacks) on attacking machine learning privacy, the research most critical to the development of PrivacyRaven are:\n\n+ [A Survey of Privacy Attacks in Machine Learning](https://arxiv.org/abs/2007.07646)\n+ [Membership Inference Attacks on Machine Learning: A Survey](https://arxiv.org/abs/2103.07853)\n+ [Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment](https://dl.acm.org/doi/pdf/10.1145/3319535.3354261?casa_token=lDNQ40-4Wa4AAAAA%3Ap9olQ3qMdDZ0n2sl-nNIgk4sOuLRMBTGVTxycZ5wjGpnFPf5lTz-MYw0e8ISggSseHC9T46it5yX)\n+ [Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data](https://ieeexplore.ieee.org/document/8489592)\n+ [Knockoff Nets: Stealing Functionality of Black-Box Models](https://arxiv.org/abs/1812.02766)\n\n## Appearances \n\nThis is a list of publications, presentations, blog posts, and other public-facing media discussing PrivacyRaven. \n\n+ [PrivacyRaven: Implementing a proof of concept for model inversion. Trail of Bits Blog](https://www.trailofbits.com/post/privacyraven-implementing-proof-of-concept-model-inversion)\n+ [PrivacyRaven: Comprehensive Privacy Testing for Deep Learning. OpenMined Privacy Conference](https://www.youtube.com/watch?v=F46lX5VIoas\u0026list=PLUNOsx6Az_ZGKQd_p4StdZRFQkCBwnaY6\u0026t=8510s)\n+ [PrivacyRaven: Comprehensive Privacy Testing for Deep Learning. Empire Hacking](https://www.youtube.com/watch?v=tKFc0ZsWNX4)\n+ [PrivacyRaven: Comprehensive Privacy Testing for Deep Learning. Trail of Bits Blog](https://blog.trailofbits.com/2020/10/08/privacyraven-has-left-the-nest/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailofbits%2FPrivacyRaven","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrailofbits%2FPrivacyRaven","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailofbits%2FPrivacyRaven/lists"}