Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/elastic/anonymize-it

a general utility for anonymizing data
https://github.com/elastic/anonymize-it

Last synced: 27 days ago
JSON representation

a general utility for anonymizing data

Awesome Lists containing this project

README 

        
# anonymize-it

A general utility for anonymizing data



`anonymize-it` can be run as a script that accepts a config file specifying the type source, anonymization mappings, and destination and an anonymizer pipeline. Individual pipeline components can also be imported into any python program that wishes to anonymize data. 



Currently, the `anonymize-it` supports two methods for anonymization: 

1) Faker-based: Relies on providers from [`Faker`](http://faker.readthedocs.io) to perform masking of fields. This method is suitable for one-off anonymization usecases, where correlation between data obtained from different sources (indices/clusters) is not necessary.



E.g.:



```

>>> from faker import Faker

>>> f = Faker()

>>> f.file_path()

'/break/Congress.json'

```

2) Hash-based: Uses a unique user/customer ID as a salt to anonymize fields. This method is suitable when anonymization of data needs to be performed regularly and/or if correlation of data from different sources is crucial. 



E.g.: A user wants to anonymize network events and process events stored in two separate indices but wants to correlate all activity for a particular host even after anonymization



# Disclaimer



`anonymize-it` is intended to serve as a tool to replace real data values with sensical artificial ones such that the semantics of the data are retained. It is not intended to be used for anonymization requirements of GDPR policies, but rather to aid pseudonymization efforts. There may also be some collisions in high cardinality datasets on using the Faker implementation.



# Instructions for use



## Installation



This must be run in a virtualenvironment with the correct dependencies installed. These are enumerated in `requirements.txt`



### Install `virtualenv` globally:



```

[sudo] pip install virtualenv

```



Create a virtualenv and install the dependencies of `anonymize-it`

```

virtualenv -p python3 venv

source venv/bin/activate

pip install -r requirements.txt

```



and run:



```

python anonymize.py configs/config.json

```



## Quick Start



`anonymize.py` is reproduced below to walk through a simple anonymization pipeline.



First load and parse the config file.

 

```python

config_file = sys.argv[1]

config = read_config(config_file) # opens json file and stores as python dict

config = utils.parse_config(config) # utility function for parsing configuration and setting variables

```



Then, create the reader as defined in the configuration. `reader_mapping` is used as a dispatcher that maps human reader reader types (e.g. elasticsearch) to reader classes (e.g. `ESReader()`).

```python

reader = reader_mapping[config.source['type']]

reader = reader(config.source['params'], config.masked_fields, config.suppressed_fields)

```



Next, create the writer in the same way.

```python

writer = writer_mapping[config.dest['type']]

writer = writer(config.dest['params'])

```



Finally, create an anonymizer by passing the reader and writer instances and run `anonymize()`.

```python

anon = Anonymizer(reader=reader, writer=writer)

anon.anonymize()

```



### Creating your own anonymizer pipeline



An anonymizer requires a `reader` and a `writer`. Currently, only an elasticsearch reader `readers.ESReader()` and a filesystem writer `writers.FSWriter()` are provided.



#### `readers`



Creating an instance of a reader requires the following:



* a `source` object, which contains parameters about the source. Please note that each reader class requires a different set of parameters. Please consult docstrings for specific parameters. 

* `masked_fields` which is a dictionary that contains field names that should be masked, along with the faker provider to be used for masking, if using the Faker-based anonymization. e.g.: `{"user.name": "user_name", "user.email": "email"}`

If using the hash-based implementation, `masked_fields` is simply a list of field names to be masked. e.g.: `["user.name", "user.email"]`

* `suppressed_fields` which is a list of fields that should NOT be included in anonymization.



`masked_fields` is required on the reader since the reader is responsible for enumerating the distinct values for each field to be used as a lookup for masking values in the faker-based anonymization.



`suppressed_fields` is required on the reader since we will explicitly exclude these from a search query.



Readers must implement the following methods:

* `get_data()`, which is responsible for returning data from the source and passing it to the anonymizer.

* (If using Faker-based anonymization), `create_mappings()`, which is responsible for generating a dictionary to be used by the anonymizer object. The dictionary is structured as so:

    ```python

    {

      "field.1": {

          "val1.1": None,

          "val1.2": None,

          ...,

          "val1.n": None

        },

      "field.2": {

          "val2.1": None,

          "val2.2": None,

          ...,

          "val2.m": None

        }

    }

    ``` 

where `field.1` and `field.2` are the fields to be anonymized and the `val1.1`, `val1.2` etc. are the distinct values for each field



#### `writers`



Creating an instance of a writer requires the following:



*  A `dest` object, which contains parameters about the destination. Please note that each writer class requires a different set of parameters. Please consult docstrings for specific parameters.



Writers must implement the following methods:



* `write_data()`, which send anonymized data to the destination.



## Run as Script



#### `anonymizers`



```

python anonymize.py configs/config.json

```



`config.json` defines the work to be done, please see template file at `configs/config.json` for guidance:



*  `source` defines the location of the original data to be anonymized along with the type of reader that should be invoked.

   *  `source.type`: a reader type. one of:

      * "elasticsearch"

      * "csv" (TBD)

      * "json" (TBD)

   * `source.params`: parameters allowing for access of data. specific to the reader type.

      * "elasticsearch":

         * `host`

         * `index`

         * `use_ssl`

         * `auth` (`native` optional)

* `dest` defines the location where the data should be written back to

    * `dest.type` a writer type. one of:

        * "filesystem"

        * "csv' (TBD)

        * "elasticsearch" (TBD)

    * `dest.params`: parameters allowing for writing of data. specific to writer types

       * "json":

          * `directory` : directory to write output json files

* `anonymization`: type of anonymization i.e. `faker` or `hash`

* `include`: the fields to mask along with the method for anonymization in case of faker-based anonymization. This is a dict with entries like `{"field.name":"faker.provider.mask"}`. Please see faker documentation for providers [here](http://faker.readthedocs.io/en/master/providers.html).

For hash-based anonymization, this can be a list of fields to be masked like `["field.name"]`.

* `exclude`: specific fields to exclude

* `sensitive`: included fields (apart from the masked fields) that should not be completely replaced by a faker/hash substitute, but should be searched for sensitive information

* `include_rest`: `{true|false}` if true, all fields except excluded fields will be written. if false, only fields specified in `masks` will be written.



### Important notes for Faker-based anonymization

1) Set the `provider_map` class attribute for the `Anonymizer` class, which is a dict with entries like `{"field.name":self.faker.provider.mask}`. Refer `anonymizers.py` for a test configuration of `provider_map`.

2) If the fields being anonymized have high cardinality, set the `high_cardinality_fields` class attribute for the `Anonymizer` class, which is a dict with entries like `{"field.name": [self.faker.provider.mask(10) for _ in range(10)]}`.



### Important notes for hash-based anonymization

1) The user should have `monitor` privilege for the Elastic environment in which to run the anonymization.

2) If you are a Cloud user and want to perform hash-based anonymization, you'll need to create an API key in the Elasticsearch Service Console and provide it as input when prompted. To create an API key, follow the instructions [here](https://www.elastic.co/guide/en/cloud/current/ec-api-authentication.html).



In addition to the above settings, for more fine-grained control over the anonymization, you can also set the following class attributes for `Anonymizer`:

1) `user_regexes`, which is a dict with entries like `{"regex.name": "regex"}`. These regexes are used to redact PII (apart from secrets, which is already taken care of) from the `sensitive` fields

2) `keywords`, which is a list like `["keyword1", "keyword2"]`. Documents containing any of the keywords in any of the `sensitive` fields are dropped.



# Adding Masks



For the faker-based anonymization, the anonymizer class only knows how to use providers that are enumerated in the `provider_map` class attribute. If you would like to add support for new faker providers, please add entries to this dict.



# Adding Readers



Readers can be added to `readers.py`, simply extend the base reader class and implement all abstract methods. Add a new entry to `reader_mapping`



# Adding Writers



Readers can be added to `writers.py`, simply extend the base writer class and implement all abstract methods. Add a new entry to `reader_mapping` 



# General Notes

https://stackoverflow.com/questions/17486578/how-can-you-bundle-all-your-python-code-into-a-single-zip-file



# Running Tests



To run the unit tests, 

1. Create a virtual environment and install dependencies in `requirements.txt`

2. Execute `py.test` from the top-level repository directory