{"id":19399868,"url":"https://github.com/worldbank/realtabformer","last_synced_at":"2026-01-04T10:14:18.456Z","repository":{"id":65922181,"uuid":"563089528","full_name":"worldbank/REaLTabFormer","owner":"worldbank","description":"A suite of auto-regressive and Seq2Seq (sequence-to-sequence) transformer models for tabular and relational synthetic data generation.","archived":false,"fork":false,"pushed_at":"2025-03-11T18:33:51.000Z","size":12776,"stargazers_count":225,"open_issues_count":30,"forks_count":26,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-13T00:48:54.934Z","etag":null,"topics":["data-generation","deep-learning","gpt","gpt-2","seq2seq-model","synthetic-data","synthetic-dataset-generation","tabular-data","transformers"],"latest_commit_sha":null,"homepage":"https://worldbank.github.io/REaLTabFormer/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/worldbank.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-07T22:06:58.000Z","updated_at":"2025-04-11T23:31:36.000Z","dependencies_parsed_at":"2023-12-18T15:13:21.741Z","dependency_job_id":"cf6c50ff-b307-4755-ab4e-b74ebdc73dc5","html_url":"https://github.com/worldbank/REaLTabFormer","commit_stats":{"total_commits":29,"total_committers":3,"mean_commits":9.666666666666666,"dds":0.06896551724137934,"last_synced_commit":"e14fab3d217a6e4f2527d217fc3f8ec918bbc392"},"previous_names":["worldbank/realtabformer","avsolatorio/realtabformer"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2FREaLTabFormer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2FREaLTabFormer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2FREaLTabFormer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2FREaLTabFormer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/worldbank","download_url":"https://codeload.github.com/worldbank/REaLTabFormer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650436,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-generation","deep-learning","gpt","gpt-2","seq2seq-model","synthetic-data","synthetic-dataset-generation","tabular-data","transformers"],"created_at":"2024-11-10T11:11:59.525Z","updated_at":"2026-01-04T10:14:18.420Z","avatar_url":"https://github.com/worldbank.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://colab.research.google.com/github/avsolatorio/RealTabFormer/blob/main/colab/REaLTabFormer_GeoValidator_Example.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e [![Downloads](https://static.pepy.tech/badge/realtabformer)](https://pepy.tech/project/realtabformer) [![Downloads](https://static.pepy.tech/badge/realtabformer/month)](https://pepy.tech/project/realtabformer) [![Downloads](https://static.pepy.tech/badge/realtabformer/week)](https://pepy.tech/project/realtabformer)\n\n# REaLTabFormer\n\nThe REaLTabFormer (Realistic Relational and Tabular Data using Transformers) offers a unified framework for synthesizing tabular data of different types. A sequence-to-sequence (Seq2Seq) model is used for generating synthetic relational datasets. The REaLTabFormer model for a non-relational tabular data uses GPT-2, and can be used out-of-the-box to model any tabular data with independent observations.\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/avsolatorio/RealTabFormer/raw/main/img/REalTabFormer_Final_EQ.png\" style=\"width:75%\"/\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003cstrong\u003eREaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers\u003c/strong\u003e\n\u003cbr\u003e\n\u003ca href=\"https://arxiv.org/abs/2302.02041\"\u003ePaper on ArXiv\u003c/a\u003e\n\u003c/p\u003e\n\u003cbr\u003e\n\u003c!---\nTabular data is a common form of organizing data. Multiple models are available to generate synthetic tabular datasets where observations are independent, but few have the ability to produce relational datasets. Modeling relational data is challenging as it requires modeling both a \"parent\" table and its relationships across tables. We introduce REaLTabFormer (Realistic Relational and Tabular Transformer), a tabular and relational synthetic data generation model. It first creates a parent table using an autoregressive GPT-2 model, then generates the relational dataset conditioned on the parent table using a sequence-to-sequence (Seq2Seq) model. We implement target masking to prevent data copying and propose the $Q_\\delta$ statistic and statistical bootstrapping to detect overfitting. Experiments using real-world datasets show that REaLTabFormer captures the relational structure better than a baseline model. REaLTabFormer also achieves state-of-the-art results on prediction tasks, \"out-of-the-box\", for large non-relational datasets without needing fine-tuning. ---\u003e\n\n\n## Installation\n\nREaLTabFormer is available on PyPi and can be easily installed with [pip](https://pypi.org/project/pip/) (Python version \u003e= 3.7):\n\n```bash\npip install realtabformer\n```\n\n## Usage\n\nWe show examples of using the REaLTabFormer for modeling and generating synthetic data from a trained model.\n\n\u003e [!NOTE]\n\u003e The model implements an optimal stopping criterion based on the synthetic data distribution when training a non-relational tabular model.\n\u003e The model will stop training when the synthetic data distribution is close to the real data distribution.\n\u003e\n\u003e **Make sure to set the `epochs` parameter to a large number to allow the model to fit the data better.**\n\u003e The model will stop training when the optimal stopping criterion is met.\n\n### REaLTabFormer for regular tabular data\n\n\n```Python\n# pip install realtabformer\nimport pandas as pd\nfrom realtabformer import REaLTabFormer\n\ndf = pd.read_csv(\"foo.csv\")\n\n# NOTE: Remove any unique identifiers in the\n# data that you don't want to be modeled.\n\n# Non-relational or parent table.\nrtf_model = REaLTabFormer(\n    model_type=\"tabular\",\n    gradient_accumulation_steps=4,\n    logging_steps=100)\n\n# Fit the model on the dataset.\n# Additional parameters can be\n# passed to the `.fit` method.\nrtf_model.fit(df)\n\n# Save the model to the current directory.\n# A new directory `rtf_model/` will be created.\n# In it, a directory with the model's\n# experiment id `idXXXX` will also be created\n# where the artefacts of the model will be stored.\nrtf_model.save(\"rtf_model/\")\n\n# Generate synthetic data with the same\n# number of observations as the real dataset.\nsamples = rtf_model.sample(n_samples=len(df))\n\n# Load the saved model. The directory to the\n# experiment must be provided.\nrtf_model2 = REaLTabFormer.load_from_dir(\n    path=\"rtf_model/idXXXX\")\n```\n\n\n### REaLTabFormer for relational data\n\n```Python\n# pip install realtabformer\nimport os\nimport pandas as pd\nfrom pathlib import Path\nfrom realtabformer import REaLTabFormer\n\nparent_df = pd.read_csv(\"foo.csv\")\nchild_df = pd.read_csv(\"bar.csv\")\njoin_on = \"unique_id\"\n\n# Make sure that the key columns in both the\n# parent and the child table have the same name.\nassert ((join_on in parent_df.columns) and\n        (join_on in child_df.columns))\n\n# Non-relational or parent table. Don't include the\n# unique_id field.\nparent_model = REaLTabFormer(model_type=\"tabular\")\nparent_model.fit(parent_df.drop(join_on, axis=1))\n\npdir = Path(\"rtf_parent/\")\nparent_model.save(pdir)\n\n# # Get the most recently saved parent model,\n# # or a specify some other saved model.\n# parent_model_path = pdir / \"idXXX\"\nparent_model_path = sorted([\n    p for p in pdir.glob(\"id*\") if p.is_dir()],\n    key=os.path.getmtime)[-1]\n\nchild_model = REaLTabFormer(\n    model_type=\"relational\",\n    parent_realtabformer_path=parent_model_path,\n    output_max_length=None,\n    train_size=0.8)\n\nchild_model.fit(\n    df=child_df,\n    in_df=parent_df,\n    join_on=join_on)\n\n# Generate parent samples.\nparent_samples = parent_model.sample(len(parend_df))\n\n# Create the unique ids based on the index.\nparent_samples.index.name = join_on\nparent_samples = parent_samples.reset_index()\n\n# Generate the relational observations.\nchild_samples = child_model.sample(\n    input_unique_ids=parent_samples[join_on],\n    input_df=parent_samples.drop(join_on, axis=1),\n    gen_batch=64)\n\n```\n\n## Validators for synthetic samples\n\nThe REaLTabFormer framework provides an interface to easily build observation validators for filtering invalid synthetic samples. We show an example of using the `GeoValidator` below. The chart on the left shows the distribution of the generated latitude and longitude without validation. The chart on the right shows the synthetic samples with observations that have been validated using the `GeoValidator` with the California boundary. Still, even when we did not optimally train the model for generating this, the invalid samples (falling outside of the boundary) are scarce from the generated data with no validator.\n\n\u003cdiv align=center\u003e\n    \u003cimg src=\"https://github.com/avsolatorio/RealTabFormer/raw/main/img/CA-housing-raw-samples.png\" style=\"width:35%; padding-right:5%;\"/\u003e\n    \u003cimg src=\"https://github.com/avsolatorio/RealTabFormer/raw/main/img/CA-housing-validated-samples.png\" style=\"width:35%; padding-left:5%;\"/\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n```Python\n# !pip install geopandas \u0026\u003e /dev/null\n# !pip install realtabformer \u0026\u003e /dev/null\n# !git clone https://github.com/joncutrer/geopandas-tutorial.git \u0026\u003e /dev/null\nimport geopandas\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom realtabformer import REaLTabFormer\nfrom realtabformer import rtf_validators as rtf_val\nfrom shapely.geometry import Polygon, LineString, Point, MultiPolygon\nfrom sklearn.datasets import fetch_california_housing\n\n\ndef plot_sf(data, samples, title=None):\n    xlims = (-126, -113.5)\n    ylims = (31, 43)\n    bins = (50, 50)\n\n    dd = samples.copy()\n    pp = dd.loc[\n        dd[\"Longitude\"].between(data[\"Longitude\"].min(), data[\"Longitude\"].max()) \u0026\n        dd[\"Latitude\"].between(data[\"Latitude\"].min(), data[\"Latitude\"].max())\n    ]\n\n    g = sns.JointGrid(data=pp, x=\"Longitude\", y=\"Latitude\", marginal_ticks=True)\n    g.plot_joint(\n        sns.histplot,\n        bins=bins,\n    )\n\n    states[states['NAME'] == 'California'].boundary.plot(ax=g.ax_joint)\n    g.ax_joint.set_xlim(*xlims)\n    g.ax_joint.set_ylim(*ylims)\n\n    g.plot_marginals(sns.histplot, element=\"step\", color=\"#03012d\")\n\n    if title:\n        g.ax_joint.set_title(title)\n\n    plt.tight_layout()\n\n# Get geographic files\nstates = geopandas.read_file('geopandas-tutorial/data/usa-states-census-2014.shp')\nstates = states.to_crs(\"EPSG:4326\")  # GPS Projection\n\n# Get the California housing dataset\ndata = fetch_california_housing(as_frame=True).frame\n\n# We create a model with small epochs for the demo, default is 200.\nrtf_model = REaLTabFormer(\n    model_type=\"tabular\",\n    batch_size=64,\n    epochs=10,\n    gradient_accumulation_steps=4,\n    logging_steps=100)\n\n# Fit the specified model. We also reduce the num_bootstrap, default is 500.\nrtf_model.fit(data, num_bootstrap=10)\n\n# Save the trained model\nrtf_model.save(\"rtf_model/\")\n\n# Sample raw data without validator\nsamples_raw = rtf_model.sample(n_samples=10240, gen_batch=512)\n\n# Sample data with the geographic validator\nobs_validator = rtf_val.ObservationValidator()\nobs_validator.add_validator(\n    \"geo_validator\",\n    rtf_val.GeoValidator(\n        MultiPolygon(states[states['NAME'] == 'California'].geometry[0])),\n    (\"Longitude\", \"Latitude\")\n)\n\nsamples_validated = rtf_model.sample(\n    n_samples=10240, gen_batch=512,\n    validator=obs_validator,\n)\n\n# Visualize the samples\nplot_sf(data, samples_raw, title=\"Raw samples\")\nplot_sf(data, samples_validated, title=\"Validated samples\")\n```\n\n## Citation\n\nPlease cite our work if you use the REaLTabFormer in your projects or research.\n\n``` bibtex\n@article{solatorio2023realtabformer,\n  title={REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers},\n  author={Solatorio, Aivin V. and Dupriez, Olivier},\n  journal={arXiv preprint arXiv:2302.02041},\n  year={2023}\n}\n```\n\n## Acknowledgments\n\nWe thank the [World Bank-UNHCR Joint Data Center on Forced Displacement (JDC)](https://www.jointdatacenter.org/) for funding the project \"Enhancing Responsible Microdata Access to Improve\nPolicy and Response in Forced Displacement Situations\" (KP-P174174-GINP-TF0B5124). A part of the fund went into supporting the development of the REaLTabFormer framework which was used to generate the synthetic population for research on disclosure risk and the mosaic effect.\n\nWe also send :hugs: to the [HuggingFace](https://huggingface.co/) :hugs: for all the open-sourced software they release. And to all open-sourced projects, thank you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworldbank%2Frealtabformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fworldbank%2Frealtabformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworldbank%2Frealtabformer/lists"}