{"id":21685882,"url":"https://github.com/sodascience/metasyn","last_synced_at":"2025-04-07T07:15:29.936Z","repository":{"id":56719481,"uuid":"481256526","full_name":"sodascience/metasyn","owner":"sodascience","description":"Transparent and privacy-friendly synthetic data generation","archived":false,"fork":false,"pushed_at":"2025-03-20T12:55:05.000Z","size":7960,"stargazers_count":40,"open_issues_count":9,"forks_count":9,"subscribers_count":3,"default_branch":"develop","last_synced_at":"2025-03-31T06:06:05.197Z","etag":null,"topics":["metadata","open-data","privacy","synthetic-data"],"latest_commit_sha":null,"homepage":"https://metasyn.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sodascience.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-13T14:46:10.000Z","updated_at":"2025-03-04T12:35:40.000Z","dependencies_parsed_at":"2023-12-29T12:23:18.887Z","dependency_job_id":"b8049c7f-ae7e-4527-a613-986f4506abe8","html_url":"https://github.com/sodascience/metasyn","commit_stats":null,"previous_names":["sodascience/metasyn"],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sodascience%2Fmetasyn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sodascience%2Fmetasyn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sodascience%2Fmetasyn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sodascience%2Fmetasyn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sodascience","download_url":"https://codeload.github.com/sodascience/metasyn/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247608153,"owners_count":20965952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["metadata","open-data","privacy","synthetic-data"],"created_at":"2024-11-25T16:23:31.493Z","updated_at":"2025-04-07T07:15:29.912Z","avatar_url":"https://github.com/sodascience.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/sodascience/metasyn/blob/main/docs/source/images/logos/blue.svg\" width=\"600px\" alt=\"Metasyn logo\"\u003e\u003c/img\u003e\n  \u003ch3 align=\"center\"\u003eTransparent and privacy-friendly synthetic data generation\u003c/h3\u003e\n  \u003cp align=\"center\"\u003e\n    \u003cspan\u003e\n        \u003ca href=\"https://www.repostatus.org/#active\"\u003e\u003cimg src=\"https://www.repostatus.org/badges/latest/active.svg\" alt=\"Project Status: Active – The project has reached a stable, usable state and is being actively developed.\" /\u003e\u003c/a\u003e\n        \u003ca href=\"https://pypi.org/project/metasyn\"\u003e\u003cimg src=\"https://img.shields.io/pypi/pyversions/metasyn\" alt=\"metasyn on pypi\"\u003e\u003c/img\u003e\u003c/a\u003e\n        \u003ca href=\"https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"open getting started on colab\"\u003e\u003c/img\u003e\u003c/a\u003e\n        \u003ca href=\"https://metasyn.readthedocs.io/en/latest/index.html\"\u003e\u003cimg src=\"https://readthedocs.org/projects/metasyn/badge/?version=latest\" alt=\"Readthedocs\"\u003e\u003c/img\u003e\u003c/a\u003e\n        \u003ca href=\"https://hub.docker.com/r/sodateam/metasyn\"\u003e\u003cimg src=\"https://img.shields.io/docker/v/sodateam/metasyn?logo=docker\u0026label=docker\u0026color=blue\" alt=\"Docker image version\"\u003e\u003c/img\u003e\u003c/a\u003e\n        \u003ca href=\"https://zenodo.org/doi/10.5281/zenodo.7696031\"\u003e\u003cimg src=\"https://zenodo.org/badge/DOI/10.5281/zenodo.7696031.svg\" alt=\"DOI\"\u003e\u003c/a\u003e\n        \u003ca style=\"border-width:0\" href=\"https://doi.org/10.21105/joss.07099\"\u003e\n  \u003cimg src=\"https://joss.theoj.org/papers/10.21105/joss.07099/status.svg\" alt=\"DOI badge\" \u003e\n\u003c/a\u003e\n    \u003c/span\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\u003cbr/\u003e\n\n__Generate synthetic tabular data__ in a transparent, understandable, and privacy-friendly way. Metasyn makes it possible for owners of sensitive data to create test data, do open science, improve code reproducibility, encourage data reuse, and enhance accessibility of their datasets, without worrying about leaking private information. \n\nWith metasyn you can __fit__ a model to an existing dataframe, __save__ it to a transparent and auditable `.json` file, and __synthesize__ a dataframe that looks a lot like the real one. In contrast to most other synthetic data software, we make the explicit choice to strictly limit the statistical information in our model in order to adhere to the highest privacy standards.\n\n## Highlights\n- 👋 __Accessible__. Metasyn is designed to be easy to use and understand, and we do our best to be welcoming to newcomers and novice users. [Let us know](https://github.com/sodascience/metasyn/issues/new) if we can improve!\n- ✨ __Fully featured__. Out of the box, metasyn natively handles a wide range of data types, missing values, categorical data, key columns with unique values, and structured strings such as postcodes or identifiers.\n- 🔎 __Transparent__. With metasyn you share not only synthetic data, but also the model and settings used to create it through a traceable, auditable metadata format. Everyone can read and understand what the model does; it is crystal clear which information becomes public.\n- 🔐 __Private__. By default, metasyn does not incorporate multivariate information, meaning less risk of privacy issues such as identity, attribute, or group disclosure. On top of this, we support privacy plugins such as our own [disclosure control plugin](https://github.com/sodascience/metasyn-disclosure-control) to further enhance privacy in critically sensitive situations.\n- 🔗 __Integrated__. We integrate closely with popular, modern tools in the python ecosystem, building on the wonderful [polars](https://pola.rs/) dataframe library ([pandas](https://pandas.pydata.org/) is supported too), as well as [faker](https://faker.readthedocs.io/en/master/) to generate localized data for names, emails, and phone numbers, and more.\n- 📦 __Extensible__. Are you missing features? Do you have a different definition of privacy? Our plugin system allows you (or your organisation) to create their own extension to adjust metasyn to what you need. Or you can [contribute](#contributing) directly to the project.\n\n## Installation\nMetasyn can be installed directly from PyPI using the following command in the terminal:\n\n```sh\npip install metasyn\n```\n\nThe latest (possibly unstable) development version can be installed directly from GitHub like so:\n\n```sh\npip install git+https://github.com/sodascience/metasyn\n```\n\n## Usage\n\n![demo](https://github.com/user-attachments/assets/f3982077-4a02-4a41-b88c-d5145ef8bdd7)\n\nTo generate synthetic data, `metasyn` first needs to fit a `MetaFrame` to the data which can then be used to produce new synthetic rows:\n\n![Example input and output](https://github.com/sodascience/metasyn/blob/main/docs/source/images/example_input_output_concise.png)\n\nThe above image closely matches the Python code:\n\n```python\nimport polars as pl\nfrom metasyn import MetaFrame, demo_file\n\n# Get the csv file path for built-in demo dataset\ncsv_path = demo_file(\"fruit\")\n\n# Create a polars dataframe from the csv file.\n# It is important to ensure the data types are correct  \n# when creating your dataframe, especially categorical data!\ndf = pl.read_csv(csv_path, schema_overrides={\n  \"fruits\": pl.Categorical, \n  \"cars\": pl.Categorical\n})\n\n# Create a MetaFrame from the DataFrame.\nmf = MetaFrame.fit_dataframe(df)\n\n# Generate a new DataFrame with 5 rows from the MetaFrame.\ndf_synth = mf.synthesize(5)\n\n# This DataFrame can be exported to csv, parquet, excel and more.\ndf_synth.write_csv(\"output.csv\")\n```\n\nTo explore more options and try this out online, take a look at our interactive tutorial:\n\n[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb)\n\nFor more information on how to create dataframes with polars, refer to the [Polars documentation](https://pola.rs/). Metasyn also works with pandas dataframes!\n\n## Where to go next\n\n- To explore more options and try this out online, take a look at our interactive tutorial: [![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sodascience/metasyn/blob/main/examples/getting_started.ipynb)\n- As a next step to learn more about generating synthetic data with metasyn we recommend to check out the [user guide](https://metasyn.readthedocs.io/en/latest/usage/usage.html) and other [documentation](https://metasyn.readthedocs.io/en/latest).\n- For even more privacy, have a look at our [disclosure control plugin](https://github.com/sodascience/metasyn-disclosure-control).\n- Want to create programs that build on metasyn? Take a look at our versioned [Docker containers](https://hub.docker.com/r/sodateam/metasyn) and our [CLI](https://metasyn.readthedocs.io/en/latest/usage/cli.html).\n\n## Contributing\nMetasyn is an open-source project, and we welcome contributions from the community, from bug reports \u0026 feature requests to code contributions. Read our [contributing guidelines](.github/CONTRIBUTING.md) for more information and to get started!\n\n## Contact\n**Metasyn** is a project by the [ODISSEI Social Data Science (SoDa)](https://odissei-data.nl/nl/soda/) team.\nDo you have questions, suggestions, or remarks on the technical implementation? Create an issue in the [issue tracker](https://github.com/sodascience/metasyn/issues) or feel free to contact [Erik-Jan van Kesteren](https://github.com/vankesteren) or [Raoul Schram](https://github.com/qubixes).\n\n\u003cimg src=\"docs/source/images/logos/soda.png\" alt=\"SoDa logo\" width=\"250px\"/\u003e \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsodascience%2Fmetasyn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsodascience%2Fmetasyn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsodascience%2Fmetasyn/lists"}