{"id":42651697,"url":"https://github.com/monch1962/test-data-mgmt-spike","last_synced_at":"2026-01-29T07:34:26.273Z","repository":{"id":64473294,"uuid":"302802163","full_name":"monch1962/test-data-mgmt-spike","owner":"monch1962","description":"Spike to try out some ideas with Test Data Management","archived":false,"fork":false,"pushed_at":"2026-01-13T05:14:48.000Z","size":67,"stargazers_count":0,"open_issues_count":60,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-13T08:36:09.388Z","etag":null,"topics":["docker","python-3","testdata"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monch1962.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-10-10T02:59:33.000Z","updated_at":"2023-06-10T01:14:16.000Z","dependencies_parsed_at":"2023-10-05T03:45:24.974Z","dependency_job_id":"e2474e44-4863-4eed-93b7-590daa3bd68d","html_url":"https://github.com/monch1962/test-data-mgmt-spike","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/monch1962/test-data-mgmt-spike","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monch1962%2Ftest-data-mgmt-spike","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monch1962%2Ftest-data-mgmt-spike/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monch1962%2Ftest-data-mgmt-spike/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monch1962%2Ftest-data-mgmt-spike/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monch1962","download_url":"https://codeload.github.com/monch1962/test-data-mgmt-spike/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monch1962%2Ftest-data-mgmt-spike/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28870469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T06:56:44.678Z","status":"ssl_error","status_checked_at":"2026-01-29T06:56:35.794Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","python-3","testdata"],"created_at":"2026-01-29T07:34:25.733Z","updated_at":"2026-01-29T07:34:26.262Z","avatar_url":"https://github.com/monch1962.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/monch1962/test-data-mgmt-spike)\n[![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://lbesson.mit-license.org/)\n[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)\n[![pylint](https://raw.githubusercontent.com/monch1962/test-data-mgmt-spike/main/pylint-badge.svg)](https://github.com/monch1962/test-data-mgmt-spike)\n[![coverage](https://raw.githubusercontent.com/monch1962/test-data-mgmt-spike/main/coverage.svg)](https://github.com/monch1962/test-data-mgmt-spike)\n[![Build Status](https://dev.azure.com/monch1962/monch1962/_apis/build/status/monch1962.test-data-mgmt-spike?branchName=main)](https://dev.azure.com/monch1962/monch1962/_build/latest?definitionId=14\u0026branchName=main)\n\n# test-data-mgmt-spike\nSpike to try out some ideas with Test Data Management.\n\nThis tool is based around Python's SQLAlchemy library, in an attempt to work with just about any database product (e.g. SQLite, MySQL, Postgres, MS SQL, AWS DynamoDB) that I'm likely to encounter. If different databases appear, there's a very good chance they will be supported by SQLAlchemy fairly quickly, so supporting them should simply be a matter of `pip install`ing the appropriate database driver.\n\n...Or so I hope.\n\n### High level approach\n- generate a set of test data, using the `http://generatedata.com` toolset. The generated data should be saved as JSON format, with one JSON file for each database table to be populated\n- create a YAML file to map the JSON files to the database/tables/fields you wish to populate\n- create a small script around the `TestDataManagement` class in the `data_loader.py` file to populate your test data into the database. An example of how to use the `TestDataManagement` class is included at the bottom of `data_loader.py`, and should be adapted to suit your needs\n\n## Data generation\n\nData is generated using the http://generatedata.com tool. This is an open-source tool that supports a range of different data types, can be extended, and has a Dockerfile to make it easily hosted inside test infrastructure.\n\nOne possible use case would be to include a generatedata.com Docker container *inside* your test environment, then create \u0026 store common test data definitions inside the generatedata instance so you can quickly retrieve them \u0026 generate new data on demand.\n\nUse this tool to generate data for the table/s you wish to populate, then save these files as JSON. A sample file is under `./generated-data` in this repository\n\n## Data Mapping\n\nThe `datamap.yaml` file in this repository contains a sample for mapping generated data files to the database, tables \u0026 fields you wish to update\n\n### Sample database URIs\n\nDatabase URIs in the `datamap.yaml` file follow this format:\n- dialect+driver://username:password@host:port/database\n\nSome examples:\n| Database type | Sample URL |\n| --- | --- |\n| MySQL | `mysql://scott:tiger@localhost/foo` |\n| Postgres | `postgresql://scott:tiger@localhost:5432/mydatabase`\n| AWS Aurora (Postgres) | `postgresql+auroradataapi://:@/my_db_name`\n| SQLite | `sqlite:///directory/file.db`\n| Oracle | `oracle://scott:tiger@127.0.0.1:1521/sidname`\n| Microsoft SQL Server | `mssql+pymssql://scott:tiger@hostname:port/dbname`\n| AWS Redshift | `redshift+psycopg2://username@host.amazonaws.com:5439/database`\n| Google BigQuery | `bigquery://project`\n| Google Sheets | `gsheets://`\n| IBM DB2 | `db2+ibm_db://user:pass@host[:port]/database`\n| SAP HANA | `hana://username:password@example.de:30015`\n| Snowflake | `snowflake://{user}:{password}@{account}/`\n| Teradata | `teradatasql://guest:please@whomooz`\n| Firebird | `firebird://username:password@localhost///home/paulgd/projects/databases/my_project.fdb`\n\n\n\n## To Use\n\n`$ DATAMAP=datamap.yaml ./data_loader.py` should run as a sample to update the `generated-data/database.db` SQLite database.\n\n## To update PyLint \u0026 coverage badges\n\nAfter making code changes, but before committing and pushing to github\n\n`$ pylint-badge data_loader.py pylint-badge.svg`\n\n`$ coverage-badge -o coverage.svg`\n\n## TODO\n- implement upsert functionality\n- add some useful test cases\n- confirm things work for various database types\n- create some more example use cases for the `TestDataManagement` class\n- add environment variable to point to the datamap YAML file\n- get Dockerfile working to read datamap YAML from an environment variable and test data to populate from a volume\n- add some documentation to the `datamap.yaml` file, describing what the various YAML elements are doing","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonch1962%2Ftest-data-mgmt-spike","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonch1962%2Ftest-data-mgmt-spike","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonch1962%2Ftest-data-mgmt-spike/lists"}