Privacy Testing for Deep Learning

trailofbits, updated 🕥 2023-03-08 17:20:42

PrivacyRaven Logo

Note: This project is on hiatus.

PrivacyRaven is a privacy testing library for deep learning systems. You can use it to determine the susceptibility of a model to different privacy attacks; evaluate privacy preserving machine learning techniques; develop novel privacy metrics and attacks; and repurpose attacks for data provenance and other use cases.

PrivacyRaven supports label-only black-box model extraction, membership inference, and (soon) model inversion attacks. We also plan to include differential privacy verification, automated hyperparameter optimization, more classes of attacks, and other features; see the GitHub issues for more information. PrivacyRaven has been featured at the OpenMined Privacy Conference, Empire Hacking, and Trail of Bits blog.

Why use PrivacyRaven?

Deep learning systems, particularly neural networks, have proliferated in a wide range of applications, including privacy-sensitive use cases such as facial recognition and medical diagnoses. However, these models are vulnerable to privacy attacks that target both the intellectual property of the model and the confidentiality of the training data. Recent literature has seen an arms race between privacy attacks and defenses on various systems. And until now, engineers and researchers have not had the privacy analysis tools they need to rival this trend. Hence, we developed PrivacyRaven- a machine learning assurance tool that aims to be: + Usable: Multiple levels of abstraction allow users to either automate much of the internal mechanics or directly control them, depending on their use case and familiarity with the domain. + Flexible: A modular design makes the attack configurations customizable and interoperable. It also allows new privacy metrics and attacks to be incorporated straightforwardly. + Efficient: PrivacyRaven reduces the boilerplate, affording quick prototyping and fast experimentation. Each attack can be launched in fewer than 15 lines of code.

How does it work?

PrivacyRaven partitions each attack into multiple customizable and optimizable phases. Different interfaces are also provided for each attack. The interface shown below is known as the core interface. PrivacyRaven also provides wrappers around specific attack configurations found in the literature and a run-all-attacks feature.

Here is how you would launch a model extraction attack in PrivacyRaven:

```python

examples/extract_mnist_gpu.py

import privacyraven as pr from privacyraven.utils.data import get_emnist_data from privacyraven.extraction.core import ModelExtractionAttack from privacyraven.utils.query import get_target from privacyraven.models.victim import train_four_layer_mnist_victim from privacyraven.models.four_layer import FourLayerClassifier

Create a query function for a target PyTorch Lightning model

model = train_four_layer_mnist_victim()

def query_mnist(input_data): # PrivacyRaven provides built-in query functions return get_target(model, input_data, (1, 28, 28, 1))

Obtain seed (or public) data to be used in extraction

emnist_train, emnist_test = get_emnist_data()

Run a model extraction attack

attack = ModelExtractionAttack( query_mnist, # query function 200, # query limit (1, 28, 28, 1), # victim input shape 10, # number of targets (3, 1, 28, 28), # substitute input shape "copycat", # synthesizer name FourLayerClassifier, # substitute model architecture 784, # substitute input size emnist_train, # seed train data emnist_test, # seed test data ) `` Since the only main requirement from the victim model is a query function, PrivacyRaven can be used to attack a wide range of models regardless of the framework and distribution method. The other classes of attacks can be launched in a similar fashion. See theexamples` folder for more information.

Want to use PrivacyRaven?

Install poetry.
Git clone this repository.
Run poetry update
Run poetry install.

If you'd like to use a Jupyter Notebook environment, run poetry shell followed by jupyter notebook.

Additionally, if you'd like to run PrivacyRaven in a Docker container, run chmod +x build.sh followed by ./build.sh. Note that depending on the amount of resources you allocate to Docker, PrivacyRaven's performance may be drastically impacted.

Feel free to join our #privacyraven channel in Empire Hacking if you need help using or extending PrivacyRaven. The official pip release will arrive soon.

Please note that PrivacyRaven is still early in development and is undergoing rapid changes. Users are advised to update frequently and avoid applying PrivacyRaven to critical use cases.

Want to contribute to PrivacyRaven?

PrivacyRaven is still a work-in-progress. We invite you to contribute however you can whether you want to incorporate a new synthesis technique or make an attack function more readable. Please visit CONTRIBUTING.md to get started.

Why is it called PrivacyRaven?

The raven has been associated with a variety of concepts in different cultures through time. Among these, the raven is commonly associated with prophecy and insight. Naturally, we named the tool PrivacyRaven because it is designed to provide insights into the privacy of deep learning.

Who maintains PrivacyRaven?

The core maintainers are: + Suha S. Hussain + Philip Wang + Jim Miller

License

This library is available under the Apache License 2.0. For an exception to the terms, please contact us.

References

While PrivacyRaven was built upon a plethora of research on attacking machine learning privacy, the research most critical to the development of PrivacyRaven are:

Appearances

This is a list of publications, presentations, blog posts, and other public-facing media discussing PrivacyRaven.

Issues

Bump future from 0.18.2 to 0.18.3

opened on 2023-03-08 17:20:41 by dependabot[bot]

Bumps future from 0.18.2 to 0.18.3.

Release notes

Sourced from future's releases.

v0.18.3

This is a minor bug-fix release containing a number of fixes:

Backport fix for bpo-38804 (c91d70b)

Fix bug in fix_print.py fixer (dffc579)

Fix bug in fix_raise.py fixer (3401099)

Fix newint bool in py3 (fe645ba)

Fix bug in super() with metaclasses (6e27aac)

docs: fix simple typo, reqest -> request (974eb1f)

Correct eq (c780bf5)

Pass if lint fails (2abe00d)

Update docker image and parcel out to constant variable. Add comment to update version constant (45cf382)

fix order (f96a219)

Add flake8 to image (046ff18)

Make lint.sh executable (58cc984)

Add docker push to optimize CI (01e8440)

Build System (42b3025)

Add docs build status badge to README.md (3f40bd7)

Use same docs requirements in tox (18ecc5a)

Add docs/requirements.txt (5f9893f)

Add PY37_PLUS, PY38_PLUS, and PY39_PLUS (bee0247)

fix 2.6 test, better comment (ddedcb9)

fix 2.6 test (3f1ff7e)

remove nan test (4dbded1)

include list test values (e3f1a12)

fix other python2 test issues (c051026)

fix missing subTest (f006cad)

import from old imp library on older python versions (fc84fa8)

replace fstrings with format for python 3.4,3.5 (4a687ea)

minor style/spelling fixes (8302d8c)

improve cmp function, add unittest (0d95a40)

Pin typing==3.7.4.1 for Python 3.3 compatiblity (1a48f1b)

Fix various py26 unit test failures (9ca5a14)

Add initial contributing guide with docs build instruction (e55f915)

Add docs building to tox.ini (3ee9e7f)

Support NumPy's specialized int types in builtins.round (b4b54f0)

Added r""" to the docstring to avoid warnings in python3 (5f94572)

Add subclasscheck for past.types.basestring (c9bc0ff)

Correct example in README (681e78c)

Add simple documentation (6c6e3ae)

Add pre-commit hooks (a9c6a37)

Handling of next and next by future.utils.get_next was reversed (52b0ff9)

Add a test for our fix (461d77e)

Compare headers to correct definition of str (3eaa8fd)

#322 Add support for negative ndigits in round; additionally, fixing a bug so that it handles passing in Decimal properly (a4911b9)

Add tkFileDialog to future.movers.tkinter (f6a6549)

Sort before comparing dicts in TestChainMap (6126997)

Fix typo (4dfa099)

Fix formatting in "What's new" (1663dfa)

Fix typo (4236061)

... (truncated)

Commits

af1db97 Merge pull request #613 from PythonCharmers/lwan/0.18.3-release
079ee9b Prepare for 0.18.3 release
02f7a81 Merge pull request #610 from wshanks/wshanks-patch-1
c91d70b Backport fix for bpo-38804
80523f3 Merge pull request #569 from jmadler/master
5e5af71 Merge pull request #582 from r3m0t/patch-6
17e4bbd Merge pull request #596 from abjonnes/fix-print-trailing-comma
1b427ba Merge branch 'xZise-official-count' into master
c8eb497 Merge branch 'official-count' of https://github.com/xZise/python-future into ...
dffc579 Fix bug in fix_print.py fixer
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/trailofbits/PrivacyRaven/network/alerts).

Bump certifi from 2020.12.5 to 2022.12.7

opened on 2022-12-08 08:52:42 by dependabot[bot]

Bumps certifi from 2020.12.5 to 2022.12.7.

Commits

9e9e840 2022.12.07
b81bdb2 2022.09.24
939a28f 2022.09.14
aca828a 2022.06.15.2
de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...
b8eb5e9 2022.06.15.1
47fb7ab Fix deprecation warning on Python 3.11 (#199)
b0b48e0 fixes #198 -- update link in license
9d514b4 2022.06.15
4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump dparse from 0.5.1 to 0.5.2

opened on 2022-10-06 17:22:34 by dependabot[bot]

Bumps dparse from 0.5.1 to 0.5.2.

Changelog

Sourced from dparse's changelog.

0.5.2 (2022-08-09)

Install pyyaml only when asked for with extras (conda extra)

Add support for piptools requirements.in

Use ConfigParser directly

Removed a regex used in the index server validation, fixing a possible ReDos security issue

Commits

2bcf15b Fixing rst issue
aa53997 Fixing travis file
13934e1 Release 0.5.2
125beef Adding pyyaml to tests, now pyyaml is an extra
b18762f Adding conda extra, pyyaml won't be installed by default.
69ba6dc Updapting travis importlib_metadata
8c99017 Merge pull request #57 from pyupio/security/remove-intensive-regex
d87364f Removing index server validation
3290bb5 Merge pull request #53 from oz123/local-import-yaml
5a3ad57 Local import YAML instead of top level
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump protobuf from 3.16.0 to 3.18.3

opened on 2022-09-23 22:58:49 by dependabot[bot]

Bumps protobuf from 3.16.0 to 3.18.3.

Release notes

Sourced from protobuf's releases.

Protocol Buffers v3.18.3

C++

Reduce memory consumption of MessageSet parsing

This release addresses a Security Advisory for C++ and Python users

Protocol Buffers v3.16.1

Java

Improve performance characteristics of UnknownFieldSet parsing (#9371)

Protocol Buffers v3.18.2

Java

Improve performance characteristics of UnknownFieldSet parsing (#9371)

Protocol Buffers v3.18.1

Python

Update setup.py to reflect that we now require at least Python 3.5 (#8989)

Performance fix for DynamicMessage: force GetRaw() to be inlined (#9023)

Ruby

Update ruby_generator.cc to allow proto2 imports in proto3 (#9003)

Protocol Buffers v3.18.0

C++

Fix warnings raised by clang 11 (#8664)

Make StringPiece constructible from std::string_view (#8707)

Add missing capability attributes for LLVM 12 (#8714)

Stop using std::iterator (deprecated in C++17). (#8741)

Move field_access_listener from libprotobuf-lite to libprotobuf (#8775)

Fix #7047 Safely handle setlocale (#8735)

Remove deprecated version of SetTotalBytesLimit() (#8794)

Support arena allocation of google::protobuf::AnyMetadata (#8758)

Fix undefined symbol error around SharedCtor() (#8827)

Fix default value of enum(int) in json_util with proto2 (#8835)

Better Smaller ByteSizeLong

Introduce event filters for inject_field_listener_events

Reduce memory usage of DescriptorPool

For lazy fields copy serialized form when allowed.

Re-introduce the InlinedStringField class

v2 access listener

Reduce padding in the proto's ExtensionRegistry map.

GetExtension performance optimizations

Make tracker a static variable rather than call static functions

Support extensions in field access listener

Annotate MergeFrom for field access listener

Fix incomplete types for field access listener

Add map_entry/new_map_entry to SpecificField in MessageDifferencer. They record the map items which are different in MessageDifferencer's reporter.

Reduce binary size due to fieldless proto messages

TextFormat: ParseInfoTree supports getting field end location in addition to start.

... (truncated)

Commits

a902b39 No-op whitespace change
ae62acd Updating version.json and repo version numbers to: 18.3
f43ac49 Merge pull request #10542 from deannagarcia/3.18.x
9efdf55 Add missing includes
d1635e1 Apply patch
5b37c91 Update version.json with "lts": true (#10534)
c39d622 Merge pull request #10529 from protocolbuffers/deannagarcia-patch-5
f77d3b6 Update version.json
8178b06 Merge pull request #10503 from deannagarcia/3.18.x
24ca839 Add version file
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Fix HopSkipJump extraction

opened on 2022-02-11 20:41:25 by suhacker1

Is your feature request related to a problem? Please describe. With the new security updates and other changes, HopSkipJump-based extraction no longer works. We need to fix it so that using HopSkipJump to populate the synthetic dataset for model extraction is possible.

Retroactively prove if PrivacyRaven was used on a model

opened on 2021-06-04 22:34:24 by suhacker1

Comment with a design document explaining your solution. One potential solution is to watermark the models/"hide" the PrivacyRaven logo or some key inside the mode.

Trail of Bits

More code: binary lifters @lifting-bits, blockchain @crytic, forks @trail-of-forks

GitHub Repository

machine-learning deep-learning python privacy privacy-preserving-machine-learning privacy-enhancing-technologies membership-inference model-extraction model-inversion