Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/levitation-opensource/dataanonymiser

Anonymises data inside text files and in sheet files. It recognises and removes various sorts of personally identifiable information (PII). Each removed part is replaced with a suitable generic text, depending on the type of removed data. Currently English and Russian languages are supported. Russian works both with Cyrillic and Latin characters.
https://github.com/levitation-opensource/dataanonymiser

anonymisation anonymity anonymity-enhancement anonymization anonymization-technique anonymize anonymize-strings anonymized-data anonymizer csv data-filtering named-entity-recognition ner personally-identifiable-information privacy privacy-enhancing-technologies privacy-protection privacy-tool privacy-tools text-filter

Last synced: about 1 month ago
JSON representation

Anonymises data inside text files and in sheet files. It recognises and removes various sorts of personally identifiable information (PII). Each removed part is replaced with a suitable generic text, depending on the type of removed data. Currently English and Russian languages are supported. Russian works both with Cyrillic and Latin characters.

Awesome Lists containing this project

README 

        
# Data Anonymiser



This software anonymises data inside text files and in sheet files. It recognises and removes various sorts of personally identifiable information (PII). Each removed part is replaced / obfuscated with a suitable generic text, depending on the type of removed data. 



Currently English and Russian languages are supported. Russian works both with Cyrillic and Latin characters. 



The language is automatically detected. In case of sheet files, the language of each cell is detected separately. Therefore multi-language sheet files are supported as well.



Currently supported sheet file formats are CSV-files, TSV-files, Excel files (XLSX only), and OpenDocument Sheet files (ODS).



## Example input and output files



Example input and output copied to an annotated PDF file: Anonymisation example 1.pdf



Example input and output file pairs for TXT and CSV file formats in English language, and TXT file format in Russian language with Cyrillic and Latin alphabet:

* data/test_input_en.txt - data/example_output_en.txt

* data/test_input_en.csv - data/example_output_en.csv

* data/test_input_ru_cyr.txt - data/example_output_ru_cyr.txt

* data/test_input_ru_lat.txt - data/example_output_ru_lat.txt



## How it works



This software uses a combination of Named Entity Recognition (NER) and regular expressions to perform its function.



## Usage



The configuration options can be found in the file Anonymiser.ini



`python Anonymiser.py "input_file.txt"|"input_file.csv"|"input_file.tsv"|"input_file.xlsx"|"input_file.ods" ["output_file.txt"|"output_file.csv"|"output_file.tsv"|"output_file.xlsx"|"output_file.ods"]`



The user provided files are expected to be in the same folder as the main Python script, unless an absolute path is provided. If run without arguments then sample files in the `data` folder are used. If the user provides input file name but no output file name then the output file name will be calculated as `input filename` + `_anonymised` + `.input filename extension`.



If the CSV file parsing fails or the CSV output seems to have wrong structure, please check and adjust the CSV parsing settings in Anonymiser.ini. More concretely, `CsvDelimiter`, `CsvQuoteChar`, `CsvDoubleQuote`, and `CsvEscapeChar` parameters may need adjustment.



## Current project state

Ready to use. Is actively developed further.