https://github.com/fgmacedo/datanonymizer
Anonymizer tool for datasets such CSV files
https://github.com/fgmacedo/datanonymizer
data-anonymity data-anonymization data-anonymized fake-data fake-data-generator
Last synced: about 1 year ago
JSON representation
Anonymizer tool for datasets such CSV files
- Host: GitHub
- URL: https://github.com/fgmacedo/datanonymizer
- Owner: fgmacedo
- License: mit
- Created: 2021-05-13T15:47:08.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2024-10-07T06:31:02.000Z (over 1 year ago)
- Last Synced: 2025-04-23T00:37:40.468Z (about 1 year ago)
- Topics: data-anonymity, data-anonymization, data-anonymized, fake-data, fake-data-generator
- Language: Python
- Homepage:
- Size: 21.5 KB
- Stars: 9
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.rst
- License: LICENSE.txt
Awesome Lists containing this project
README
=============
datanonymizer
=============
Anonymizer tool for datasets such CSV files.
To generate fake data, you can choose between two excelent generators:
- `Faker `_ (default).
- `mimesis `_ via optional install.
Install
=======
Using pip:
.. code-block:: bash
pip install datanonymizer
Using mimesis instead of the default Faker:
.. code-block:: bash
pip install datanonymizer[mimesis]
Or from source:
.. code-block:: bash
git clone https://github.com/fgmacedo/datanonymizer
cd datanonymizer
python setup.py install
Usage
=====
Pass your data through ``stdin`` and get it back anonymized on ``stdout``.
.. note::
In this case, the output will be equal to the input as no conversions were applied.
.. code-block:: bash
cat input_file.csv | datanonymizer >output_file.csv
Using a config file to declare conversions and generators for the required fields:
.. code-block:: bash
cat input_file.csv | datanonymizer --config ./dataset_anon_config.yml >output_file.csv
Please see examples folder for a small demo:
.. code-block:: bash
cat examples/small.csv | python -m datanonymizer -i --config examples/small_faker.yml --seed my_seed >examples/small_anonymized_using_faker.csv
Optional arguments:
.. code-block::
-h, --help show this help message and exit
-l LANGUAGE, --language LANGUAGE
Language used by the Generator
-di DELIMITER_INPUT, --delimiter_input DELIMITER_INPUT
CSV delimiter
-do DELIMITER_OUTPUT, --delimiter_output DELIMITER_OUTPUT
CSV delimiter
-i, --ignore_errors Continue on errors
--head HEAD Outputs only the first lines
-g {faker,mimesis}, --generator {faker,mimesis}
Generator library to be used for fake data
--seed SEED Seed for the pseudo random generator providers
--config CONFIG Configuration file
Config file
===========
You'l need a configuration file to setup transformations for each dataset.
This file is a simple `yaml `_ where you can configure fields.
Field names should match the column name declared into the CSV input file.
.. code-block:: yaml
---
fields:
Task ID:
omit: true
Location:
conversions:
- fn: coords_to_h3
kwargs:
resolution: 8
Client Address:
conversions:
- fn: has_value
rename: has_address
Company Name:
generator:
provider: business.company
rename: company
Invoice ID:
generator:
provider: person.identifier
kwargs:
mask: "#######"
rename: invoice
Generators
----------
The generatos clause depends of the library you choose to provide fake data.
You can use any generator available at the generic API from Faker or mimesis.
For example, if you wanna mimic data with company names:
- Faker
.. code-block:: yaml
---
fields:
Company Name:
generator:
provider: company
- Mimesis
.. code-block:: yaml
---
fields:
Company Name:
generator:
provider: business.company
But you can replace the real names by names of fruits (using Mimesis) or any other provider:
.. code-block:: yaml
---
fields:
Company Name:
generator:
provider: food.fruit
Or generate random integers to replace real IDs:
- Faker
.. code-block:: yaml
---
fields:
ID:
generator:
provider: pyint
kwargs:
min_value: 1
max_value: 15_000_000
- Mimesis
.. code-block:: yaml
---
fields:
ID:
generator:
provider: person.identifier
kwargs:
mask: "#######"
Conversions
-----------
You can apply any pre-configured conversion functions available.
- coords_to_h3
- has_value