{"id":34109606,"url":"https://github.com/martinjurkovic/syntherela","last_synced_at":"2026-04-02T03:19:29.460Z","repository":{"id":263817706,"uuid":"753127673","full_name":"martinjurkovic/syntherela","owner":"martinjurkovic","description":"A package for benchmarking synthetic relational data generation methods","archived":false,"fork":false,"pushed_at":"2026-03-25T11:56:31.000Z","size":7221,"stargazers_count":61,"open_issues_count":8,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-03-26T14:22:51.617Z","etag":null,"topics":["benchmark","deep-learning","graph-neural-networks","machine-learning","pytorch","pytorch-geometric","relational-data","relational-deep-learning","synthetic-data","tabular-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/martinjurkovic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-02-05T14:23:07.000Z","updated_at":"2026-03-20T06:56:30.000Z","dependencies_parsed_at":"2025-01-20T17:40:58.263Z","dependency_job_id":"81e667d6-90b6-4876-a782-cda4626d04d2","html_url":"https://github.com/martinjurkovic/syntherela","commit_stats":null,"previous_names":["martinjurkovic/syntherela"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/martinjurkovic/syntherela","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinjurkovic%2Fsyntherela","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinjurkovic%2Fsyntherela/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinjurkovic%2Fsyntherela/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinjurkovic%2Fsyntherela/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/martinjurkovic","download_url":"https://codeload.github.com/martinjurkovic/syntherela/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinjurkovic%2Fsyntherela/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31295053,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T01:43:37.129Z","status":"online","status_checked_at":"2026-04-02T02:00:08.535Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","deep-learning","graph-neural-networks","machine-learning","pytorch","pytorch-geometric","relational-data","relational-deep-learning","synthetic-data","tabular-data"],"created_at":"2025-12-14T18:34:59.208Z","updated_at":"2026-04-02T03:19:29.448Z","avatar_url":"https://github.com/martinjurkovic.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SyntheRela - Synthetic Relational Data Generation Benchmark\n\n\u003ch2 align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/martinjurkovic/syntherela/refs/heads/main/docs/SyntheRela.png\" height=\"150px\"\u003e\n    \u003cdiv align=\"center\"\u003e\n      \u003ca href=\"https://pypi.org/project/syntherela/\"\u003e\n        \u003cimg src=\"https://img.shields.io/pypi/v/syntherela\" alt=\"PyPI\"\u003e\n      \u003c/a\u003e\n      \u003ca href=\"https://github.com/martinjurkovic/syntherela/blob/main/LICENSE\"\u003e\n        \u003cimg alt=\"MIT License\" src=\"https://img.shields.io/badge/License-MIT-yellow.svg\"\u003e\n      \u003c/a\u003e\n      \u003ca href=\"https://openreview.net/forum?id=ZfQofWYn6n\"\u003e\n        \u003cimg alt=\"Paper URL\" src=\"https://img.shields.io/badge/OpenReview-ZfQofWYn6n-B31B1B.svg\"\u003e\n      \u003c/a\u003e\n      \u003ca href=\"https://huggingface.co/spaces/SyntheRela/leaderboard\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/🤗-Leaderboard-yellow.svg\" alt=\"Hugging Face Leaderboard\"\u003e\n      \u003c/a\u003e\n  \u003c/div\u003e\n\u003c/h2\u003e\n\n## About SyntheRela\n\nSyntheRela is a comprehensive benchmark designed to evaluate and compare synthetic relational database generation methods. It provides a standardized framework for assessing both the fidelity and utility of synthetic data across multiple real-world databases. The benchmark includes novel evaluation metrics, particularly for relational data, and supports various open-source and commercial synthetic data generation methods.\n\nSyntheRela is highly extensible, allowing users to benchmark on their own custom datasets and implement new evaluation metrics to suit specific use cases.\n\nOur research on SyntheRela is presented in the paper **\"SyntheRela: A Benchmark For Synthetic Relational Database Generation\"** at the ICLR 2025 Workshop \"Will Synthetic Data Finally Solve the Data Access Problem?\", available on [OpenReview](https://openreview.net/forum?id=ZfQofWYn6n).\n\nWe maintain a [public leaderboard on Hugging Face](https://huggingface.co/spaces/SyntheRela/leaderboard) where you can compare the performance of different synthetic data generation methods.\n\n## Installation\nTo install only the benchmark package, run the following command:\n\n```bash\npip install syntherela\n```\n## Using SyntheRela\n\nTo evaluate your synthetic relational data, configure the `Benchmark` class with your desired metrics and run the evaluation pipeline:\n\n```python\nfrom syntherela.benchmark import Benchmark\nfrom syntherela.metrics.single_column.statistical import ChiSquareTest\nfrom syntherela.metrics.single_table.distance import MaximumMeanDiscrepancy\nfrom syntherela.metrics.multi_table.statistical import CardinalityShapeSimilarity\nfrom syntherela.metrics.multi_table.detection import AggregationDetection\nfrom xgboost import XGBClassifier\n\n# Initialize the benchmark with specific metrics\nbenchmark = Benchmark(\n    real_data_dir=\"path/to/real_data\",\n    synthetic_data_dir=\"path/to/synthetic_data\",\n    results_dir=\"results\",\n    single_column_metrics=[ChiSquareTest()],\n    single_table_metrics=[MaximumMeanDiscrepancy()],\n    multi_table_metrics=[\n        CardinalityShapeSimilarity(),\n        AggregationDetection(classifier_cls=XGBClassifier, random_state=42)\n    ],\n    datasets=[\"your_dataset_name\"],\n    methods=[\"your_method_name\"]\n)\n\n# Execute evaluation\nbenchmark.run()\n```\n\n## Examples\n\nWe provide example notebooks to help you get started with `syntherela` in the [examples/](examples/) directory.\n\n- [Evaluating Rossmann Subsampled Dataset](examples/evaluate_rossmann_subsampled.ipynb): A step-by-step guide to evaluating a subsampled version of the Rossmann dataset using various metrics.\n\n## Replicating the paper's results\n\nFor detailed instructions on how to replicate the paper's results, please refer to [docs/REPLICATING_RESULTS.md](/docs/REPLICATING_RESULTS.md).\n\n## Adding a new metric\nThe documentation for adding a new metric can be found in [docs/ADDING_A_METRIC.md](/docs/ADDING_A_METRIC.md).\n\n\n\n\\* Denotes the method does not have a public implementation available.\n\n## 🏆 Leaderboard Submission\n\nWe maintain an official leaderboard to benchmark synthetic relational data generation methods. To ensure fairness and reproducibility, **all evaluations are performed by the SyntheRela maintainers** on standardized hardware.\n\n### Evaluation Overview\n| Feature | Specification |\n| :--- | :--- |\n| **Compute** | Single NVIDIA H100 (80GB) |\n| **Time Limit** | 48 hours execution time **per dataset** |\n| **Submission Frequency** | 1 submission per 30-day period |\n| **Capacity** | Up to 2 model variants/checkpoints per submission |\n\n### How to Submit\n1. **Prepare your code:** Ensure your method is reproducible and includes a clear `README` and `requirements.txt`.\n2. **Open an Issue:** Create a new [GitHub Issue](https://github.com/martinjurkovic/syntherela/issues) using the title prefix `[Model Submission]`.\n\nFor the complete requirements regarding environment setup, logging, and our privacy/confidentiality policy, please refer to our **[Full Submission Guidelines](https://docs.google.com/document/d/1ae16L_vvT5PFt2OeN7FJauA_ayd_A6xCkhVJFoYcx04)**.\n\n## Conflicts of Interest\nThe authors declare no conflict of interest and are not associated with any of the evaluated commercial synthetic data providers.\n\n## Citation\nIf you use SyntheRela in your work, please cite our paper:\n```\n@inproceedings{\n    iclrsyntheticdata2025syntherela,\n    title={SyntheRela: A Benchmark For Synthetic Relational Database Generation},\n    author={Martin Jurkovic and Valter Hudovernik and Erik {\\v{S}}trumbelj},\n    booktitle={Will Synthetic Data Finally Solve the Data Access Problem?},\n    year={2025},\n    url={https://openreview.net/forum?id=ZfQofWYn6n}\n}\n```\n\n## License\nThis project is licensed under the [MIT License](/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartinjurkovic%2Fsyntherela","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmartinjurkovic%2Fsyntherela","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartinjurkovic%2Fsyntherela/lists"}