{"id":21689103,"url":"https://github.com/tinram/mysql-filler","last_synced_at":"2026-05-15T08:32:37.021Z","repository":{"id":83640080,"uuid":"423909456","full_name":"Tinram/MySQL-Filler","owner":"Tinram","description":"Populate all MySQL database tables in a single step.","archived":false,"fork":false,"pushed_at":"2023-06-26T17:38:10.000Z","size":94,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-20T12:48:30.974Z","etag":null,"topics":["database-filler","database-testing","fake-data","joins","junk","mysql","mysql-database"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tinram.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-02T16:00:21.000Z","updated_at":"2022-06-14T09:49:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"838744bc-b9aa-438d-8427-8949fb51ed29","html_url":"https://github.com/Tinram/MySQL-Filler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Tinram/MySQL-Filler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FMySQL-Filler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FMySQL-Filler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FMySQL-Filler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FMySQL-Filler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tinram","download_url":"https://codeload.github.com/Tinram/MySQL-Filler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FMySQL-Filler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264502387,"owners_count":23618587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-filler","database-testing","fake-data","joins","junk","mysql","mysql-database"],"created_at":"2024-11-25T17:21:15.057Z","updated_at":"2025-10-19T06:53:19.926Z","avatar_url":"https://github.com/Tinram.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# MySQL Filler\n\n#### Fill all MySQL database tables.\n\n\n## Purpose\n\nPopulate all the tables of a MySQL database with a specified number of rows of junk data, and optionally jumble the foreign keys to create sufficient fake key relationships between tables for SQL joins.\n\n\n## Background\n\nSome tasks require populating databases for testing. Database dumps are huge and the data usually *verboten* without anonymisation (i.e. contains Personally Identifiable Information). My [old script](https://github.com/Tinram/Database-Filler) fills database tables \u0026ndash; useful for schema design but not queries.\n\n\n## Example Databases\n\ndatabase | support | notes |\n:- | :-: | :- |\n[*basketball*](https://github.com/Tinram/Database-Filler/blob/master/basketball.sql) | :heavy_check_mark: | substantial intersection table |\n[*classicmodels*](https://www.mysqltutorial.org/wp-content/uploads/2018/03/mysqlsampledatabase.zip) | :heavy_check_mark: | |\n[employees](https://github.com/ronaldbradford/schema/blob/master/employees.sql) \u003csmall\u003e(old)\u003c/small\u003e | :heavy_check_mark: | |\n[Joomla](https://github.com/ronaldbradford/schema/blob/master/joomla.sql) \u003csmall\u003e(old)\u003c/small\u003e | :heavy_check_mark: | 60 tables |\n[Northwind](https://docs.yugabyte.com/latest/sample-data/northwind/) | :heavy_check_mark: | |\n[*Sakila*](https://dev.mysql.com/doc/index-other.html) | :heavy_multiplication_x: | |\ntpcc | :heavy_multiplication_x: | partial with option fudges |\n[*world*](https://dev.mysql.com/doc/index-other.html) | :heavy_check_mark: | |\n[WordPress](https://github.com/ronaldbradford/schema/blob/master/wordpress.sql) \u003csmall\u003e(old)\u003c/small\u003e | :heavy_check_mark: | |\n\n*Sakila* uses sophisticated spatial data types.\n\n[Old schemas](https://github.com/ronaldbradford/schema) reference.\n\n\n## Requirements\n\n+ An *empty* or *truncated* or *throwaway* database schema already present in the MySQL server.\n+ Sufficient privileges granted for the connecting user.\n\n\n## Limitations\n\n+ No support of MySQL spatial data types.\n+ Capped length of longer data types.\n+ Composite primary keys can be troublesome.\n\n\n## Usage\n\nImport a database schema-only file into MySQL or use an existing 'throwaway' database already active within MySQL.\n\n**Do not use this package on a database that you care about: *MySQL Filler* will so surely trash it.**\n\nEnsure `SELECT, INSERT, UPDATE, DROP` privileges for the connecting user (`DROP` is required for table wiping.)\n\nIn *config.py*, edit the database credentials, and the *MySQL Filler* options required, then:\n\n```bash\npython3 main.py\n```\n\nFor multiprocessing support (`PROCS = \u003cnum_cpu_cores\u003e`) and a significant speed increase, copy the *config.py* file imports and global variables into *src/mysql_filler.py* and run as a standalone script.  \nIt's un-pythonic and ugly, but executes multiprocessing more reliably.\n\n\n## Options\n\n```python\n\nNUM_ROWS = 10                     # number of rows to add to all database tables\nPROCS = 1                         # number of processes to spawn\n\nJUMBLE_FKS = True                 # toggle random jumbling of foreign keys for joins\nFK_PCT_REPLACE = 25               # percentage of NUM_ROWS of foreign keys to jumble\n\nSTRICT_INSERT = False             # toggle INSERT IGNOREs for duplicate hits / bypass strict SQL mode (warnings versus errors)\n\nPROCESS_INT_FKS = True            # process (True) or skip (False) integer foreign keys (TPCC schema with tinyint PKs)\nCOMPOSITE_PK_INCREMENT = False    # skip (False) or increment (True) composite primary keys (TPCC schema)\n\nCOMPLEX_JSON = False              # False for simple fixed JSON data; True to generate variable JSON data\n\nBYTES_DECODE = 'utf-8'            # character set used for byte data type conversion\nMAX_PACKET = False                # True maximises the packet size (root user only)\n\nDEBUG = False                     # debug output toggle\nEXTENDED_DEBUG = False            # verbose debug output toggle\n\nTRUNCATE_TABLES = False           # toggle truncation of all database tables (instead of populating)\n\n```\n\n\n## Example Run\n\nUsing the simple MySQL [*world*](https://dev.mysql.com/doc/index-other.html) database,  \nimport the database from the compressed file:\n\n```bash\n$ tar -xzOf world-db.tar.gz | mysql -h localhost -u root -p\n```\n\nAllocate a user with the required privileges:\n\n```sql\nmysql\u003e GRANT SELECT, INSERT, UPDATE, DROP ON world.* TO 'general'@'localhost' IDENTIFIED BY 'P@55w0rd';\nmysql\u003e FLUSH PRIVILEGES;\n```\n\nEdit the *config.py* file and wipe all data that the *world* database ships with by setting `TRUNCATE_TABLES` to `True`:\n\n\n```python\nTRUNCATE_TABLES = True\n```\n\n```bash\npython3 main.py\n```\n\n    Truncating all tables of `world` database ...\n\nChange `TRUNCATE_TABLES = False` and execute *main.py* again:\n\n```bash\npython3 main.py\n```\n\n    localhost\n    world\n    +10 rows ...\n\n```sql\nmysql\u003e USE world;\nmysql\u003e SELECT ID, CountryCode, country.Code, Code2 FROM city INNER JOIN country ON country.Code = city.CountryCode;\n```\n\nUnlikely to be any results returned by adding just 10 rows; perhaps one 'lucky' row returned for this database in 100 rows added.\n\n```python\nNUM_ROWS = 200\n...\nJUMBLE_FKS = True\n```\n\nExecute *main.py*\n\n    localhost\n    world\n    +200 rows\n\n    `city`\n    `country`\n    `countrylanguage`\n\n    foreign keys jumbled\n\nA few results now returned from the previous query, and a CountryCode can be selected:\n\n```sql\nSELECT ID, CountryCode, country.Code FROM city INNER JOIN country ON country.Code = city.CountryCode WHERE CountryCode = 'ewu';\n+----+-------------+------+\n| ID | CountryCode | Code |\n+----+-------------+------+\n| 74 | ewu         | EWU  |\n+----+-------------+------+\n1 row in set (0.00 sec)\n```\n\nMeaningless data, but the foreign keys are starting to gain relationships, and so SQL joins between tables are now realised. More rows can be added to increase the number of results.\n\n\n## Speed\n\nSpeed with multiprocessing (combining *config.py* into one script) is okay. Speed never was on the agenda.\n\nFor serious speed, there's Percona's Go-based [mysql_random_data_load](https://github.com/Percona-Lab/mysql_random_data_load). Currently, this tool fills one table at a time \u0026ndash; fast \u0026ndash; yet somewhat laborious for databases with lots of tables, whereas I wanted all database tables populated with one command.\n\n\n## MariaDB\n\nMariaDB has limited support. It has more restrictions on key constraints and is less forgiving than MySQL 5.7 or 8.0\n\n\n## Other\n\nTested using MySQL 5.7 and 8.0, and MariaDB 10.4\n\nThis package cannot hope to support all variations (good and bad) of MySQL schemas.\n\nFor example, adding 1,000 rows to the following real-world table is not going to run smoothly:\n\n```sql\n+----------------+------------+------+-----+---------+\n| Field          | Type       | Null | Key | Default |\n+----------------+------------+------+-----+---------+\n| google_channel | char(3)    | YES  |     | NULL    |\n| locale_id      | tinyint(3) | YES  | UNI | NULL    |\n+----------------+------------+------+-----+---------+\n```\n\n \u0026ndash; because of the restricted range of TINYINT values, the number of rows, and the unique key.\n\nComposite primary keys can cause trouble. However, the *config.py* options allow overrides to enable at least basic table population.\n\n\n## License\n\n*MySQL Filler* is released under the [GPL v.3](https://www.gnu.org/licenses/gpl-3.0.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinram%2Fmysql-filler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinram%2Fmysql-filler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinram%2Fmysql-filler/lists"}