{"id":35655085,"url":"https://github.com/tdspora/syngen","last_synced_at":"2026-06-09T14:00:56.619Z","repository":{"id":45718276,"uuid":"514217996","full_name":"tdspora/syngen","owner":"tdspora","description":"Open-source version of the TDspora synthetic data generation algorithm.","archived":false,"fork":false,"pushed_at":"2026-06-04T13:34:30.000Z","size":24337,"stargazers_count":18,"open_issues_count":26,"forks_count":12,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-04T14:09:02.394Z","etag":null,"topics":["generation","machine-learning","synthetic-data","tabular-data"],"latest_commit_sha":null,"homepage":"https://tdspora.ai/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tdspora.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-07-15T09:51:58.000Z","updated_at":"2026-06-03T09:55:57.000Z","dependencies_parsed_at":"2023-09-21T19:23:58.335Z","dependency_job_id":"c83285c7-1d6a-44ef-bb54-80def3a387f9","html_url":"https://github.com/tdspora/syngen","commit_stats":{"total_commits":999,"total_committers":19,"mean_commits":"52.578947368421055","dds":"0.38138138138138133","last_synced_commit":"29d7f68523e875037d57a066744ef17298b1221c"},"previous_names":[],"tags_count":329,"template":false,"template_full_name":null,"purl":"pkg:github/tdspora/syngen","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tdspora%2Fsyngen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tdspora%2Fsyngen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tdspora%2Fsyngen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tdspora%2Fsyngen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tdspora","download_url":"https://codeload.github.com/tdspora/syngen/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tdspora%2Fsyngen/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34110012,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generation","machine-learning","synthetic-data","tabular-data"],"created_at":"2026-01-05T15:14:50.178Z","updated_at":"2026-06-09T14:00:56.612Z","avatar_url":"https://github.com/tdspora.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI/CD](https://github.com/tdspora/syngen/actions/workflows/action-build-deploy.yml/badge.svg?branch=main)](https://github.com/tdspora/syngen/actions/workflows/action-build-deploy.yml)\n\n# EPAM Syngen\n\nEPAM Syngen is an unsupervised tabular data generation tool. It is useful for generation of test data with a given table as a template. Most datatypes including floats, integers, datetime, text, categorical, binary are supported. The linked tables i.e., tables sharing a key can also be generated using the simple statistical approach.\nThe source of data might be in CSV, Avro and Excel format and should be located locally and be in UTF-8 encoding.\n\nThe tool is based on the variational autoencoder model (VAE). The Bayesian Gaussian Mixture model is used to further detangle the latent space.\n\n## Prerequisites\n\nPython 3.10 or 3.11 is required to run the library. The library is tested on Linux and Windows operating systems.\nYou can download Python from [the official website](https://www.python.org/downloads/) and install manually, or you can install Python [from your terminal](https://docs.python-guide.org/starting/installation/). After the installation of Python, please, check whether [pip is installed](https://pip.pypa.io/en/stable/getting-started/).\n\n## Getting started\n\nBefore the installation of the library, you have to [set up the virtual environment](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/).\n\nYou can install the library with:\n\n```bash\npip install syngen\n```\n\n\nThe training and inference processes are separated with two CLI entry points. The training one receives paths to the original table, metadata json file or table name and used hyperparameters.\u003cbr\u003e\n\nTo start training with defaults parameters run:\n\n```bash\ntrain --source PATH_TO_ORIGINAL_CSV \\\n    --table_name TABLE_NAME\n```\n\nThis will train a model and save the model artifacts to disk.\n\nTo generate with defaults parameters data simply call:\n\n```bash\ninfer --table_name TABLE_NAME\n```\n\n\u003ci\u003ePlease notice that the name should match the one you used in the training process.\u003c/i\u003e\u003cbr\u003e\nThis will create a csv file with the synthetic table in \u003ci\u003e./model_artifacts/tmp_store/TABLE_NAME/merged_infer_TABLE_NAME.csv\u003c/i\u003e.\u003cbr\u003e\n\nHere is a quick example:\n\n```bash\ntrain --source ./examples/example-data/housing.csv –-table_name Housing\ninfer --table_name Housing\n```\n\nAs the example you can use the dataset \u003ci\u003e\"Housing\"\u003c/i\u003e in [examples/example-data/housing.csv](examples/example-data/housing.csv).\nIn this example, our real-world data is \u003ca href=\"https://www.kaggle.com/datasets/camnugent/california-housing-prices\" target=\"_blank\"\u003e\"Housing\"\u003c/a\u003e from Kaggle.\n\n## Features\n\n### Training\n\nYou can add flexibility to the training and inference processes using additional hyperparameters.\u003cbr\u003e\nFor training of single table call:\n\n```bash\ntrain --source PATH_TO_ORIGINAL_CSV \\\n    --table_name TABLE_NAME \\\n    --epochs INT \\\n    --row_limit INT \\\n    --drop_null BOOL \\\n    --reports STR \\\n    --batch_size INT \\\n    --log_level STR \\\n    --fernet_key STR\n```\n\n*Note:* To specify multiple options for the *--reports* parameter, you need to provide the *--reports* parameter multiple times.\nFor example:\n```bash\ntrain --source PATH_TO_ORIGINAL_CSV \\\n    --table_name TABLE_NAME \\\n    --reports accuracy \\\n    --reports sample\n```\nThe accepted values for the parameter \u003ci\u003e\"reports\"\u003c/i\u003e:\n - \u003ci\u003e\"none\"\u003c/i\u003e (default) - no reports will be generated\n - \u003ci\u003e\"accuracy\"\u003c/i\u003e - generates an accuracy report to measure the quality of synthetic data relative to the original dataset. This report is produced after the completion of the training process, during which a model learns to generate new data. The synthetic data generated for this report is of the same size as the original dataset to reach more accurate comparison.\n - \u003ci\u003e\"sample\"\u003c/i\u003e - generates a sample report (if original data is sampled, the comparison of distributions of original data and sampled data is provided in the report)\n - \u003ci\u003e\"metrics_only\"\u003c/i\u003e - outputs the metrics information only to standard output without generation of an accuracy report\n - \u003ci\u003e\"all\"\u003c/i\u003e - generates both accuracy and sample reports\u003cbr\u003e\nDefault value is \u003ci\u003e\"none\"\u003c/i\u003e.\n\nTo train one or more tables using a metadata file, you can use the following command:\n\n```bash\ntrain --metadata_path PATH_TO_METADATA_YAML\n```\n\nParameters that you can set up for training process:\n\n- \u003ci\u003esource\u003c/i\u003e – required parameter for training of single table, a path to the file that you want to use as a reference\n- \u003ci\u003etable_name\u003c/i\u003e – required parameter for training of single table, an arbitrary string to name the directories\n- \u003ci\u003eepochs\u003c/i\u003e – a number of training epochs. Since the early stopping mechanism is implemented the bigger value of epochs is the better\n- \u003ci\u003erow_limit\u003c/i\u003e – a number of rows to train over. A number less than the original table length will randomly subset the specified number of rows\n- \u003ci\u003edrop_null\u003c/i\u003e – whether to drop rows with at least one missing value\n- \u003ci\u003ebatch_size\u003c/i\u003e – if specified, the training is split into batches. This can save the RAM\n- \u003ci\u003ereports\u003c/i\u003e - controls the generation of quality reports, might require significant time for big tables (\u003e10000 rows)\n- \u003ci\u003emetadata_path\u003c/i\u003e – a path to the metadata file containing the metadata\n- \u003ci\u003ecolumn_types\u003c/i\u003e - might include the section \u003ci\u003ecategorical\u003c/i\u003e which contains columns explicitly defined as categorical by the user\n- \u003ci\u003elog_level\u003c/i\u003e - logging level for the process\n- \u003ci\u003efernet_key\u003c/i\u003e - the name of the environment variable that kept the value of the fernet key used to encrypt the sample data of the original data. If the fernet key is not set, the original data will be stored in '.pkl' format. If the fernet key is set, the original data will be encrypted and stored securely in '.dat' format. The same fernet key should be used for both training and inference processes to ensure that the original data can be decrypted correctly.\n\nRequirements for parameters of training process:\n* \u003ci\u003esource\u003c/i\u003e - data type - string\n* \u003ci\u003etable_name\u003c/i\u003e - data type - string\n* \u003ci\u003eepochs\u003c/i\u003e - data type - integer, must be equal to or more than 1, default value is 10\n* \u003ci\u003erow_limit\u003c/i\u003e - data type - integer\n* \u003ci\u003edrop_null\u003c/i\u003e - data type - boolean, default value - False\n* \u003ci\u003ebatch_size\u003c/i\u003e - data type - integer, must be equal to or more than 1, default value - 32\n* \u003ci\u003ereports\u003c/i\u003e - data type - if the value is passed through CLI - string, if the value is passed in the metadata file - string or list, accepted values: \u003ci\u003e\"none\"\u003c/i\u003e (default) - no reports will be generated, \u003ci\u003e\"all\"\u003c/i\u003e - generates both accuracy and sample reports, \u003ci\u003e\"accuracy\"\u003c/i\u003e - generates an accuracy report, \u003ci\u003e\"sample\"\u003c/i\u003e - generates a sample report, \u003ci\u003e\"metrics_only\"\u003c/i\u003e - outputs the metrics information only to standard output without generation of a report. Default value is \u003ci\u003e\"none\"\u003c/i\u003e. In the metadata file multiple values can be specified as a list of available options (\u003ci\u003e\"accuracy\"\u003c/i\u003e, \u003ci\u003e\"sample\"\u003c/i\u003e, \u003ci\u003e\"metrics_only\"\u003c/i\u003e) to generate multiple types of reports simultaneously, e.g. [\u003ci\u003e\"metrics_only\"\u003c/i\u003e, \u003ci\u003e\"sample\"\u003c/i\u003e]\n* \u003ci\u003emetadata_path\u003c/i\u003e - data type - string\n* \u003ci\u003ecolumn_types\u003c/i\u003e - data type - dictionary with the key \u003ci\u003ecategorical\u003c/i\u003e - the list of columns (data type - string)\n* \u003ci\u003elog_level\u003c/i\u003e - data type - string, must be one of the next values - *TRACE*, *\"DEBUG\"*, *\"INFO\"*, *\"WARNING\"*, *\"ERROR\"*, *\"CRITICAL\"*, default value is *\"INFO\"*\n* \u003ci\u003efernet_key\u003c/i\u003e - data type - string, the name of the environment variable that kept the value of the fernet key. It must be a 44-character URL-safe base64-encoded string, default value is None\n\n### Inference (generation)\n\nYou can customize the inference processes by calling for one table:\n\n```bash\ninfer --size INT \\\n    --table_name STR \\\n    --run_parallel BOOL \\\n    --batch_size INT \\\n    --random_seed INT \\\n    --reports STR \\\n    --log_level STR \\\n    --fernet_key STR\n```\n\n*Note:* To specify multiple options for the *--reports* parameter, you need to provide the *--reports* parameter multiple times.\nFor example:\n```bash\ninfer --table_name TABLE_NAME \\\n    --reports accuracy \\\n    --reports metrics_only\n```\nThe accepted values for the parameter \u003ci\u003e\"reports\"\u003c/i\u003e:\n - \u003ci\u003e\"none\"\u003c/i\u003e (default) - no reports will be generated\n - \u003ci\u003e\"accuracy\"\u003c/i\u003e - generates an accuracy report that compares original and synthetic data patterns to verify the quality of the generated data\n - \u003ci\u003e\"metrics_only\"\u003c/i\u003e - outputs the metrics information only to standard output without generation of an accuracy report\n - \u003ci\u003e\"all\"\u003c/i\u003e - generates an accuracy report\u003cbr\u003e\nDefault value is \u003ci\u003e\"none\"\u003c/i\u003e.\n\nTo generate one or more tables using a metadata file, you can use the following command:\n\n```bash\ninfer --metadata_path PATH_TO_METADATA\n```\n\nThe parameters which you can set up for generation process:\n\n- \u003ci\u003esize\u003c/i\u003e - the desired number of rows to generate\n- \u003ci\u003etable_name\u003c/i\u003e – required parameter for inference of single table, the name of the table, same as in training\n- \u003ci\u003erun_parallel\u003c/i\u003e – whether to use multiprocessing (feasible for tables \u003e 50000 rows)\n- \u003ci\u003ebatch_size\u003c/i\u003e – if specified, the generation is split into batches. This can save the RAM\n- \u003ci\u003erandom_seed\u003c/i\u003e – if specified, generates a reproducible result\n- \u003ci\u003ereports\u003c/i\u003e - controls the generation of quality reports, might require significant time for big generated tables (\u003e10000 rows)\n- \u003ci\u003emetadata_path\u003c/i\u003e – a path to metadata file\n- \u003ci\u003elog_level\u003c/i\u003e - logging level for the process\n- \u003ci\u003efernet_key\u003c/i\u003e - the name of the environment variable that kept the value of the fernet key used to encrypt the sample data of the original data. If the fernet key is not set, the original data will be stored in '.pkl' format. If the fernet key is set, the original data will be encrypted and stored securely in '.dat' format. The same fernet key should be used for both training and inference processes to ensure that the original data can be decrypted correctly.\n\nRequirements for parameters of generation process:\n* \u003ci\u003esize\u003c/i\u003e - data type - integer, must be equal to or more than 1, default value is 100\n* \u003ci\u003etable_name\u003c/i\u003e - data type - string\n* \u003ci\u003erun_parallel\u003c/i\u003e - data type - boolean, default value is False\n* \u003ci\u003ebatch_size\u003c/i\u003e - data type - integer, must be equal to or more than 1\n* \u003ci\u003erandom_seed\u003c/i\u003e - data type - integer, must be equal to or more than 0\n* \u003ci\u003ereports\u003c/i\u003e - data type - if the value is passed through CLI - string, if the value is passed in the metadata file - string or list, accepted values: \u003ci\u003e\"none\"\u003c/i\u003e (default) - no reports will be generated, \u003ci\u003e\"all\"\u003c/i\u003e - generates an accuracy report, \u003ci\u003e\"accuracy\"\u003c/i\u003e - generates an accuracy report, \u003ci\u003e\"metrics_only\"\u003c/i\u003e - outputs the metrics information only to standard output without generation of a report. Default value is \u003ci\u003e\"none\"\u003c/i\u003e. In the metadata file multiple values can be specified as a list of available options (\u003ci\u003e\"accuracy\"\u003c/i\u003e, \u003ci\u003e\"metrics_only\"\u003c/i\u003e) to generate multiple types of reports simultaneously\n* \u003ci\u003emetadata_path\u003c/i\u003e - data type - string\n* \u003ci\u003elog_level\u003c/i\u003e - data type - string, must be one of the next values - *TRACE*, *\"DEBUG\"*, *\"INFO\"*, *\"WARNING\"*, *\"ERROR\"*, *\"CRITICAL\"*, default value is *\"INFO\"*\n* \u003ci\u003efernet_key\u003c/i\u003e - data type - string, the name of the environment variable that kept the value of the fernet key. It must be a 44-character URL-safe base64-encoded string, default value is None\n\nThe metadata can contain any of the arguments above for each table. If so, the duplicated arguments from the CLI\nwill be ignored.\n\n### Linked tables generation\n\nTo generate one or more tables, you might provide metadata in yaml format. By providing information about the relationships\nbetween tables via metadata, it becomes possible to manage complex relationships across any number of tables.\nYou can also specify additional parameters needed for training and inference in the metadata file and in this case,\nthey will be ignored in the CLI call.\n\n*Note:* By using metadata file, you can also generate tables with absent relationships.\nIn this case, the tables will be generated independently.\n\nThe yaml metadata file should match the following template:\n\n```yaml\nglobal:                                     # Global settings. Optional parameter. In this section you can specify training and inference settings which will be set for all tables\n  train_settings:                           # Settings for training process. Optional parameter\n    epochs: 10                              # Number of epochs if different from the default in the command line options. Optional parameter\n    drop_null: False                        # Drop rows with NULL values. Optional parameter\n    row_limit: null                         # Number of rows to train over. A number less than the original table length will randomly subset the specified rows number. Optional parameter\n    batch_size: 32                          # If specified, the training is split into batches. This can save the RAM. Optional parameter\n    reports: none                           # Controls the generation of quality reports. Optional parameter. Accepted values: \"none\" (default) - no reports will be generated, \"all\" - generates both accuracy and sample reports, \"accuracy\" - generates an accuracy report, \"sample\" - generates a sample report, \"metrics_only\" - outputs the metrics information only to standard output without generation of a report. Multiple values can be specified as a list to generate multiple types of reports simultaneously, e.g. [\"metrics_only\", \"sample\"]. Might require significant time for big tables (\u003e10000 rows).\n\n  infer_settings:                           # Settings for infer process. Optional parameter\n    size: 100                               # Size for generated data. Optional parameter\n    run_parallel: False                     # Turn on or turn off parallel training process. Optional parameter\n    reports: none                           # Controls the generation of quality reports. Optional parameter. Accepted values: \"none\" (default) - no reports will be generated, \"all\" - generates an accuracy report, \"accuracy\" - generates an accuracy report, \"metrics_only\" - outputs the metrics information only to standard output without generation of a report. Multiple values can be specified as a list to generate multiple types of reports simultaneously. Might require significant time for big generated tables (\u003e10000 rows).\n    batch_size: null                        # If specified, the generation is split into batches. This can save the RAM. Optional parameter\n    random_seed: null                       # If specified, generates a reproducible result. Optional parameter\n\n  encryption:\n    fernet_key: null                       # The name of the environment variable that kept the value of the fernet key used to encrypt the sample data of the original data. If the fernet key is not set, the original data will be stored in '.pkl' format. If the fernet key is set, the original data will be encrypted and stored securely in '.dat' format. The same fernet key should be used for both training and inference processes to ensure that the original data can be decrypted correctly. Optional parameter\n\nCUSTOMER:                                   # Table name. Required parameter\n  train_settings:                           # Settings for training process. Required parameter\n    source: \"./files/customer.csv\"          # The path to the original data. Supported formats include local files in '.csv', '.avro' formats. Required parameter\n    epochs: 10                              # Number of epochs if different from the default in the command line options. Optional parameter\n    drop_null: False                        # Drop rows with NULL values. Optional parameter\n    row_limit: null                         # Number of rows to train over. A number less than the original table length will randomly subset the specified rows number. Optional parameter\n    batch_size: 32                          # If specified, the training is split into batches. This can save the RAM. Optional parameter\n    reports: none                           # Controls the generation of quality reports. Optional parameter. Accepted values: \"none\" (default) - no reports will be generated, \"all\" - generates both accuracy and sample reports, \"accuracy\" - generates an accuracy report, \"sample\" - generates a sample report, \"metrics_only\" - outputs the metrics information only to standard output without generation of a report. Multiple values can be specified as a list to generate multiple types of reports simultaneously, e.g. [\"metrics_only\", \"sample\"]. Might require significant time for big tables (\u003e10000 rows).\n    column_types:\n      categorical:                          # The list of columns explicitly defined as categorical by the user. Optional parameter\n        - gender\n        - marital_status\n\n  format:                                   # Settings for reading and writing data in '.csv', '.psv', '.tsv', '.txt', '.xls', '.xlsx' format. Optional parameter\n    sep: ','                                # Delimiter to use. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    quotechar: '\"'                          # The character used to denote the start and end of a quoted item. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    quoting: minimal                        # Control field quoting behavior per constants - [\"all\", \"minimal\", \"non-numeric\", \"none\"]. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    escapechar: '\"'                         # One-character string used to escape other characters. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    encoding: null                          # A string representing the encoding to use in the output file. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    header: infer                           # Row number(s) to use as the column names, and the start of the data. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    skiprows: null                          # Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    on_bad_lines: error                     # Specifies what to do upon encountering a bad line (a line with too many fields) - [\"error\", \"warn\", \"skip\"]. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    engine: null                            # Parser engine to use - [\"c\", \"python\"]. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    na_values: null                         # Additional strings to recognize as NA/NaN. The first value of the array will be used to replace NA/NaN values. Optional parameter. Applicable for '.csv', '.psv', '.tsv', '.txt' formats\n    sheet_name: 0                           # Name of the sheet in the Excel file. Optional parameter. Applicable for '.xls', '.xlsx' formats\n\n  infer_settings:                           # Settings for infer process. Optional parameter\n    destination: \"./files/generated_data_customer.csv\" # The path where the generated data will be stored. If the information about 'destination' isn't specified, by default the synthetic data will be stored locally in '.csv'. Supported formats include local files in '.csv', '.avro' formats. Optional parameter\n    size: 100                               # Size for generated data. Optional parameter\n    run_parallel: False                     # Turn on or turn off parallel training process. Optional parameter\n    reports: none                           # Controls the generation of quality reports. Optional parameter. Accepted values: \"none\" (default) - no reports will be generated, \"all\" - generates an accuracy report, \"accuracy\" - generates an accuracy report, \"metrics_only\" - outputs the metrics information only to standard output without generation of a report. Multiple values can be specified as a list to generate multiple types of reports simultaneously. Might require significant time for big generated tables (\u003e10000 rows).\n    batch_size: null                        # If specified, the generation is split into batches. This can save the RAM. Optional parameter\n    random_seed: null                       # If specified, generates a reproducible result. Optional parameter\n\n  encryption:\n    fernet_key: null                        # The name of the environment variable that kept the value of the fernet key used to encrypt the sample data of the original data. If the fernet key is not set, the original data will be stored in '.pkl' format. If the fernet key is set, the original data will be encrypted and stored securely in '.dat' format. The same fernet key should be used for both training and inference processes to ensure that the original data can be decrypted correctly. Optional parameter\n\n  keys:                                     # Keys of the table. Optional parameter\n    PK_CUSTOMER_ID:                         # Name of a key. Only one PK per table.\n      type: \"PK\"                            # The key type. Supported: PK - primary key, FK - foreign key, TKN - token key\n      columns:                              # Array of column names\n        - customer_id\n      regex_patterns:                       # Regex pattern for generating key values. Optional parameter. Applicable for PK and UQ key types only. Useful if the generation of the key column should follow specific rule(-s).\n        customer_id: \"CUST-[0-9]{6}\"        # Pattern to generate values for the specified column. E.g., \"CUST-123456\"\n\n    UQ1:                                    # Name of a key\n      type: \"UQ\"                            # One or many unique keys\n      columns:\n        - e_mail\n      regex_patterns:\n        e_mail: \"[a-z]{5,10}\\\\.[a-z]{3,7}@(gmail|yahoo|outlook)\\\\.com\"  # E.g., \"johnd.smith@gmail.com\"\n    FK1:                                    # One or many foreign keys\n      type: \"FK\"\n      columns:                              # Array of columns in the current table\n        - e_mail\n        - alias\n      references:\n        table: \"PROFILE\"                    # Name of the parent table\n        columns:                            # Array of columns in the parent table\n          - e_mail\n          - alias\n\n    FK2:\n      type: \"FK\"\n      columns:\n        - address_id\n      references:\n        table: \"ADDRESS\"\n        columns:\n          - address_id\n\n\nORDER:                                      # Table name. Required parameter\n  train_settings:                           # Settings for training process. Required parameter\n    source: \"./files/order.csv\"             # The path to the original data. Supported formats include local files in 'csv', '.avro' formats. Required parameter\n    epochs: 10                              # Number of epochs if different from the default in the command line options. Optional parameter\n    drop_null: False                        # Drop rows with NULL values. Optional parameter\n    row_limit: null                         # Number of rows to train over. A number less than the original table length will randomly subset the specified rows number. Optional parameter\n    batch_size: 32                          # If specified, the training is split into batches. This can save the RAM. Optional parameter\n    reports: none                           # Controls the generation of quality reports. Optional parameter. Accepted values: \"none\" (default) - no reports will be generated, \"all\" - generates both accuracy and sample reports, \"accuracy\" - generates an accuracy report, \"sample\" - generates a sample report, \"metrics_only\" - outputs the metrics information only to standard output without generation of a report, e.g. [\"metrics_only\", \"sample\"]. Might require significant time for big tables (\u003e10000 rows).\n    column_types:\n      categorical:                          # The list of columns explicitly defined as categorical by the user. Optional parameter\n        - gender\n        - marital_status\n\n  infer_settings:                           # Settings for infer process. Optional parameter\n    destination: \"./files/generated_data_order.csv\" # The path where the generated data will be stored. If the information about 'destination' isn't specified, by default the synthetic data will be stored locally in '.csv'. Supported formats include local files in 'csv', '.avro' formats. Required parameter\n    size: 100                               # Size for generated data. Optional parameter\n    run_parallel: False                     # Turn on or turn off parallel training process. Optional parameter\n    reports: none                           # Controls the generation of quality reports. Optional parameter. Accepted values: \"none\" (default) - no reports will be generated, \"all\" - generates an accuracy report, \"accuracy\" - generates an accuracy report, \"metrics_only\" - outputs the metrics information only to standard output without generation of a report. Multiple values can be specified as a list to generate multiple types of reports simultaneously.  Might require significant time for big generated tables (\u003e10000 rows).\n    batch_size: null                        # If specified, the generation is split into batches. This can save the RAM. Optional parameter\n    random_seed: null                       # If specified, generates a reproducible result. Optional parameter\n\n  format:                                   # Settings for reading and writing data in 'csv' format. Optional parameter\n    sep: ','                                # Delimiter to use. Optional parameter\n    quotechar: '\"'                          # The character used to denote the start and end of a quoted item. Optional parameter\n    quoting: minimal                        # Control field quoting behavior per constants - [\"all\", \"minimal\", \"non-numeric\", \"none\"]. Optional parameter\n    escapechar: '\"'                         # One-character string used to escape other characters. Optional parameter\n    encoding: null                          # A string representing the encoding to use in the output file. Optional parameter\n    header: infer                           # Row number(s) to use as the column names, and the start of the data. Optional parameter\n    skiprows: null                          # Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file. Optional parameter\n    on_bad_lines: error                     # Specifies what to do upon encountering a bad line (a line with too many fields) - [\"error\", \"warn\", \"skip\"]. Optional parameter\n    engine: null                            # Parser engine to use - [\"c\", \"python\"]. Optional parameter\n    sheet_name: 0                           # Name of the sheet in the Excel file. Optional parameter\n\n  encryption:\n    fernet_key: null                        # The name of the environment variable that kept the value of the fernet key used to encrypt the sample data of the original data. If the fernet key is not set, the original data will be stored in '.pkl' format. If the fernet key is set, the original data will be encrypted and stored securely in '.dat' format. The same fernet key should be used for both training and inference processes to ensure that the original data can be decrypted correctly. Optional parameter\n\n  keys:                                     # Keys of the table. Optional parameter\n    pk_order_id:\n      type: \"PK\"\n      columns:\n        - order_id\n\n    FK1:\n      type: \"FK\"\n      columns:\n        - customer_id\n      references:\n        table: \"CUSTOMER\"\n        columns:\n          - customer_id\n```\n\n*Note:*\n\u003cul\u003e\n\u003cli\u003eIn the section \u003ci\u003e\"global\"\u003c/i\u003e you can specify training and inference settings for all tables. If the same settings are specified for a specific table, they will override the global settings\u003c/li\u003e\n\u003cli\u003eIf the information about \u003ci\u003e\"destination\"\u003c/i\u003e isn't specified in \u003ci\u003e\"infer_settings\"\u003c/i\u003e, by default the synthetic data will be stored locally in \u003ci\u003e\".csv\"\u003c/i\u003e format\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003ci\u003eYou can find the example of metadata file in [examples/example-metadata/housing_metadata.yaml](examples/example-metadata/housing_metadata.yaml)\u003c/i\u003e\u003cbr\u003e\n\nBy providing the necessary information through a metadata file, you can initiate training and inference processes using the following commands:\n\n```bash\ntrain --metadata_path=PATH_TO_YAML_METADATA_FILE\ninfer --metadata_path=PATH_TO_YAML_METADATA_FILE\n```\nHere is a quick example:\n\n```bash\ntrain --metadata_path=\"./examples/example-metadata/housing_metadata.yaml\"\ninfer --metadata_path=\"./examples/example-metadata/housing_metadata.yaml\"\n```\n\nIf `--metadata_path` is present and the metadata contains the necessary parameters, other CLI parameters will be ignored.\u003cbr\u003e\n\n### Ways to set the value(s) in the section \"reports\" of the metadata file\n\nThe accepted values in the section \u003ci\u003e\"reports\"\u003c/i\u003e in \u003ci\u003e\"train_settings\"\u003c/i\u003e:\n - \u003ci\u003e\"none\"\u003c/i\u003e (default) - no reports will be generated\n - \u003ci\u003e\"accuracy\"\u003c/i\u003e - generates an accuracy report to measure the quality of synthetic data relative to the original dataset. This report is produced after the completion of the training process, during which a model learns to generate new data. The synthetic data generated for this report is of the same size as the original dataset to reach more accurate comparison.\n - \u003ci\u003e\"sample\"\u003c/i\u003e - generates a sample report (if original data is sampled, the comparison of distributions of original data and sampled data is provided in the report)\n - \u003ci\u003e\"metrics_only\"\u003c/i\u003e - outputs the metrics information only to standard output without generation of an accuracy report\n - \u003ci\u003e\"all\"\u003c/i\u003e - generates both accuracy and sample reports\u003cbr\u003e\nDefault value is \u003ci\u003e\"none\"\u003c/i\u003e.\n\nExamples how to set the value(s) in the section \u003ci\u003e\"reports\"\u003c/i\u003e in \u003ci\u003e\"train_settings\"\u003c/i\u003e:\n```yaml\n\nreports: none\n\nreports: all\n\nreports: accuracy\n\nreports: metrics_only\n\nreports: sample\n\nreports:\n  - accuracy\n  - metrics_only\n  - sample\n```\nThe accepted values for the parameter \u003ci\u003e\"reports\"\u003c/i\u003e in \u003ci\u003e\"infer_settings\"\u003c/i\u003e:\n - \u003ci\u003e\"none\"\u003c/i\u003e (default) - no reports will be generated\n - \u003ci\u003e\"accuracy\"\u003c/i\u003e - generates an accuracy report to verify the quality of the generated data\n - \u003ci\u003e\"metrics_only\"\u003c/i\u003e - outputs the metrics information only to standard output without generation of an accuracy report\n - \u003ci\u003e\"all\"\u003c/i\u003e - generates an accuracy report\u003cbr\u003e\nDefault value is \u003ci\u003e\"none\"\u003c/i\u003e.\n\nExamples how to set the value(s) in the section \u003ci\u003e\"reports\"\u003c/i\u003e in \u003ci\u003e\"infer_settings\"\u003c/i\u003e:\n```yaml\n\nreports: none\n\nreports: all\n\nreports: accuracy\n\nreports: metrics_only\n\nreports:\n  - accuracy\n  - metrics_only\n```\n\n\n### Docker images\n\nThe train and inference components of \u003ci\u003esyngen\u003c/i\u003e is available as public docker image:\n\n\u003chttps://hub.docker.com/r/tdspora/syngen\u003e\n\nTo run dockerized code (see parameters description in *Training* and *Inference* sections) for one table call:\n\n```bash\ndocker pull tdspora/syngen\ndocker run --rm \\\n  --user $(id -u):$(id -g) \\\n  -v PATH_TO_LOCAL_FOLDER:/src/model_artifacts tdspora/syngen \\\n  --task=train \\\n  --table_name=TABLE_NAME \\\n  --source=./model_artifacts/YOUR_CSV_FILE.csv\n\ndocker run --rm \\\n  --user $(id -u):$(id -g) \\\n  -v PATH_TO_LOCAL_FOLDER:/src/model_artifacts tdspora/syngen \\\n  --task=infer \\\n  --table_name=TABLE_NAME\n```\n\nPATH_TO_LOCAL_FOLDER is an absolute path to the folder where your original csv is stored.\n\nYou can add any arguments listed in the corresponding sections for infer and training processes in the CLI call.\n\nTo run dockerized code by providing the metadata file simply call:\n\n```bash\ndocker pull tdspora/syngen\ndocker run --rm \\\n  --user $(id -u):$(id -g) \\\n  -v PATH_TO_LOCAL_FOLDER:/src/model_artifacts tdspora/syngen \\\n  --task=train \\\n  --metadata_path=./model_artifacts/PATH_TO_METADATA_YAML\n\ndocker run --rm \\\n  --user $(id -u):$(id -g) \\\n  -v PATH_TO_LOCAL_FOLDER:/src/model_artifacts tdspora/syngen \\\n  --task=infer \\\n  --metadata_path=./model_artifacts/PATH_TO_METADATA_YAML\n```\n\nYou can add any arguments listed in the corresponding sections for infer and training processes in the CLI call, however, they will be\noverwritten by corresponding arguments in the metadata file.\n\n#### MLflow monitoring\n\nSet the `MLFLOW_TRACKING_URI` environment variable to the desired MLflow tracking server, for instance:\nhttp://localhost:5000/. You can also set the `MLFLOW_ARTIFACTS_DESTINATION` environment variable to your preferred path\n(including the cloud path), where the artifacts should be stored. Additionally, set the `MLFLOW_EXPERIMENT_NAME`\nenvironment variable to the name you prefer for the experiment.\nTo get the system metrics, please set the `MLFLOW_ENABLE_SYSTEM_METRICS_LOGGING` environment variable to `true`.\nBy default, the metrics are logged every 10 seconds, but the interval may be changed by setting the environment variable\n`MLFLOW_SYSTEM_METRICS_SAMPLING_INTERVAL` (for more detailed description look [here](https://mlflow.org/docs/latest/system-metrics/index.html))\n\nWhen using Docker, ensure the environmental variables are set before running the container.\n\nThe provided environmental variables allow to track the training process, and the inference process, and store\nthe artifacts in the desired location.\nYou can access the MLflow UI by navigating to the provided URL in your browser. If you store artifacts in remote storage,\nensure that all necessary credentials are provided before using Mlflow.\n\n```bash\ndocker pull tdspora/syngen:latest\ndocker run --rm -it \\\n  --user $(id -u):$(id -g) \\\n  -e MLFLOW_TRACKING_URI='http://localhost:5000' \\\n  -e MLFLOW_ARTIFACTS_DESTINATION=MLFLOW_ARTIFACTS_DESTINATION \\\n  -e MLFLOW_EXPERIMENT_NAME=test_name \\\n  -e MLFLOW_ENABLE_SYSTEM_METRICS_LOGGING=true \\\n  -e MLFLOW_SYSTEM_METRICS_SAMPLING_INTERVAL 10 \\\n  -v PATH_TO_LOCAL_FOLDER:/src/model_artifacts tdspora/syngen \\\n  --task=train \\\n  --metadata_path=./model_artifacts/PATH_TO_METADATA_YAML\n\ndocker run --rm -it \\\n  --user $(id -u):$(id -g) \\\n  -e MLFLOW_TRACKING_URI='http://localhost:5000' \\\n  -e MLFLOW_ARTIFACTS_DESTINATION=MLFLOW_ARTIFACTS_DESTINATION \\\n  -e MLFLOW_EXPERIMENT_NAME=test_name \\\n  -e MLFLOW_ENABLE_SYSTEM_METRICS_LOGGING=true \\\n  -e MLFLOW_SYSTEM_METRICS_SAMPLING_INTERVAL 10 \\\n  -v PATH_TO_LOCAL_FOLDER:/src/model_artifacts tdspora/syngen \\\n  --task=infer \\\n  --metadata_path=./model_artifacts/PATH_TO_METADATA_YAML\n```\n\n### How to keep the original data secure\n\nIn the current implementation, a sample of the original data is securely stored on disk.\nTo ensure data security, it is recommended to provide the name of the environment variable that kept the value of the fernet key value via the `fernet_key` parameter,\neither through the command-line interface (CLI) or a metadata file.\nThe Fernet key enables encryption of the stored data, ensuring its protection.\n\n*Fernet key usage during inference*:\nDuring inference, previously encrypted data may need to be decrypted to enable comparisons with synthetic data for report generation.\nIf the data was encrypted during the training process, the same Fernet key used for encryption must be provided during inference to successfully\ndecrypt the data and generate reports.\n\n*Please, pay attention:*\nPlease, store the Fernet key securely. If the key is lost, encrypted data cannot be recovered.\n\n*Note:* To generate a Fernet key, you can use the following code:\n\n```python\nfrom cryptography.fernet import Fernet\n\ncipher = Fernet.generate_key().decode(\"utf-8\")\n```\nThen you should set the generated key as an environment variable in your terminal:\n\n```bash\nexport YOUR_FERNET_KEY_NAME='YOUR_GENERATED_FERNET_KEY'\n```\n\n\n## Using SDK (Programmatic Interface)\n\nIn addition to the CLI, *Syngen* provides a Python SDK for programmatic access to the main functionality. The SDK is useful when you want to integrate synthetic data generation into your Python applications, notebooks, or data pipelines.\n\n### SDK Classes\n\nThe SDK provides two main classes:\n\n#### `Syngen` - Core functionality for a training, inference, and report generation\n\n```python\nfrom syngen.sdk import Syngen\n\n\n# Training\nSyngen(source=\"path/to/data.csv\", table_name=\"my_table\").train(\n    epochs=10,\n    row_limit=1000,\n    batch_size=32,\n    log_level=\"DEBUG\",\n    reports=\"all\"\n)\n\nSyngen(metadata_path=\"path/to/metadata.yaml\").train(log_level=\"DEBUG\")\n\n# Inference\nSyngen(source=\"path/to/data.csv\", table_name=\"my_table\").infer(\n  size=1000,\n  random_seed=42,\n  reports=\"accuracy\"\n)\n\nSyngen(metadata_path=\"path/to/metadata.yaml\").infer(log_level=\"DEBUG\")\n\n# Generate reports separately for a certain table\nSyngen(metadata_path=\"path/to/metadata.yaml\").generate_quality_reports(\n  table_name=\"my_table\",\n  reports=[\"accuracy\", \"sample\"]\n)\n```\n\n#### `DataIO` - Data loading and saving\n\n```python\nfrom syngen.sdk import DataIO\n\ndata_io = DataIO(\n  path=\"data.csv\",\n  sep=',',\n  encoding='utf-8',\n  header=0\n)\ndf = data_io.load_data()\n\ndata_io.save_data(df)\n```\n\n### Key SDK features\n\n- **Training and inference**: All CLI parameters are available as method arguments\n- **Report generation**: Generate quality reports separately for a certain table after a training/inference processes\n- **Data I/O**: Load and save data in multiple formats (*CSV*, *Avro*, *Excel*) with custom settings\n- **Encryption support**: Use a Fernet key for secure data handling\n- **Metadata support**: Use a metadata file for complex workflows with multiple tables\n- **Format configuration**: Customize delimiters, encodings, and other format-specific settings for loading data\n- **Loader function**: Provide a custom data loader function for advanced data loading scenarios with an opportunity to skip the process of saving the sample of the original data on the disk\n\n### Custom data loader function\n\nSDK allows you to provide a custom data loader function instead of `source` during the initialization of the `Syngen` class.\nThis is useful when you need to load the original data with specific parameters, or from formats that require\ncustom handling, and at the same time keep the original data secure by skipping the process\nof saving the sample of the original data on the disk.\n\n#### How it works\n\nThe `loader` attribute of the class `Syngen` accepts an object of the function.\n\n##### Requirements for a custom loader function\n\nYour custom loader function must:\n1. Accept a single parameter: the name of the table as a string\n2. Return a pandas DataFrame\n3. Be importable from Python's module system\n\n```python\nimport pandas as pd\n\ndef my_custom_loader(table_name: str) -\u003e pd.DataFrame:\n    # Your custom loading logic here\n    pass\n```\n\n#### The example: the complete workflow\n\n1. Create a custom loader function\n2. Use it for training and inference:\n\n```python\nfrom syngen.sdk import Syngen\nimport pandas as pd\n\n\ndef my_custom_loader(table_name: str) -\u003e pd.DataFrame:\n    # Custom logic to load data based on table_name\n    if table_name == \"my_table\":\n        return pd.read_csv(f\"path/to/{table_name}.csv\")\n    else:\n        raise ValueError(f\"Unknown table name: {table_name}\")\n\n\nlauncher = Syngen(loader=my_custom_loader, table_name=\"my_table\")\n\nlauncher.train(\n    epochs=10,\n    row_limit=1000,\n    batch_size=32,\n    log_level=\"DEBUG\",\n    reports=\"all\"\n)\n\nlauncher.infer(\n  size=1000,\n  random_seed=42,\n  reports=\"accuracy\"\n)\n```\n\n### SDK Examples\n\nFor detailed examples and usage patterns, please refer to the [SDK demonstration notebook](demo-notebooks/SDK_demonstration.ipynb)\n\n\n## Syngen Installation Guide for MacOS ARM (M1/M2) with Python 3.10 or 3.11\n\n### Prerequisites\n\nBefore you begin, make sure you have the following installed:\n\n- Python 3.10 or 3.11\n- Homebrew (optional but recommended for managing dependencies)\n\n### Installation Steps\n\n1. **Upgrade pip**: Ensure you have the latest version of `pip`.\n\n    ```sh\n    pip install --upgrade pip\n    ```\n\n2. **Install Setuptools, Wheel, and Cython**: These packages are necessary for building and installing other dependencies.\n\n    ```sh\n    pip install setuptools wheel 'Cython\u003c3'\n    ```\n\n3. **Install Fastavro**: Install a specific version of `fastavro` to avoid build issues.\n\n    ```sh\n    pip install --no-build-isolation fastavro==1.5.1\n    ```\n\n4. **Install Syngen**: Now, you can install the Syngen package.\n\n    ```sh\n    pip install syngen\n    ```\n\n5. **Install TensorFlow Metal**: This package leverages the GPU capabilities of M1/M2 chips for TensorFlow.\n\n    ```sh\n    pip install tensorflow-metal\n    ```\n\n#### From source (development)\n\nDownload repository from GitHub by cloning or zip file.\nThen install it in editable mode.\n\n```sh\n    pip install -e .\n```\n\nTo also install the test dependencies (`pytest` and friends), use the `test` extra:\n\n```sh\n    pip install -e \".[test]\"\n```\n\n### Additional Information\n\n- **Homebrew**: If you do not have Homebrew installed, you can install it by running:\n\n    ```sh\n    /bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"\n    ```\n\n- **Python 3.10**: Ensure you have Python 3.10 installed. You can use pyenv to manage different Python versions:\n\n    ```sh\n    brew install pyenv\n    pyenv install 3.10.0\n    pyenv global 3.10.0\n    ```\nOR\n- **Python 3.11**: Ensure you have Python 3.11 installed. You can use pyenv to manage different Python versions:\n\n    ```sh\n    brew install pyenv\n    pyenv install 3.11.0\n    pyenv global 3.11.0\n    ```\n\n### Verifying Installation\n\nTo verify the installation, run the following command to check if Syngen is installed correctly:\n\n```sh\npython -c \"import syngen; print(syngen.__version__)\"\n```\n\nIf the command prints the version of Syngen without errors, the installation was successful.\n\n### Troubleshooting\n\nIf you encounter any issues during installation, consider the following steps:\n\n- Ensure all dependencies are up-to-date.\n- Check for any compatibility issues with other installed packages.\n- Consult the Syngen [documentation](https://github.com/tdspora/syngen) or raise an issue on GitHub.\n\n## Contribution\n\nWe welcome contributions from the community to help us improve and maintain our public GitHub repository. We appreciate any feedback, bug reports, or feature requests, and we encourage developers to submit fixes or new features using issues.\n\nIf you have found a bug or have a feature request, please submit an issue to our GitHub repository. Please provide as much detail as possible, including steps to reproduce the issue or a clear description of the feature request. Our team will review the issue and work with you to address any problems or discuss any potential new features.\n\nIf you would like to contribute a fix or a new feature, please submit a pull request to our GitHub repository. Please make sure your code follows our coding standards and best practices. Our team will review your pull request and work with you to ensure that it meets our standards and is ready for inclusion in our codebase.\n\nWe appreciate your contributions, and thank you for your interest in helping us maintain and improve our public GitHub repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftdspora%2Fsyngen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftdspora%2Fsyngen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftdspora%2Fsyngen/lists"}