{"id":16111760,"url":"https://github.com/cliffano/datagen","last_synced_at":"2025-09-22T06:31:53.574Z","repository":{"id":3128024,"uuid":"4155886","full_name":"cliffano/datagen","owner":"cliffano","description":"Multi-process test data files generator","archived":false,"fork":false,"pushed_at":"2022-10-11T09:31:18.000Z","size":745,"stargazers_count":40,"open_issues_count":0,"forks_count":4,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-01-12T01:35:33.947Z","etag":null,"topics":["cli","data-generator","nodejs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cliffano.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-04-27T06:58:20.000Z","updated_at":"2024-08-25T00:21:51.000Z","dependencies_parsed_at":"2022-09-10T20:42:41.151Z","dependency_job_id":null,"html_url":"https://github.com/cliffano/datagen","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliffano%2Fdatagen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliffano%2Fdatagen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliffano%2Fdatagen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliffano%2Fdatagen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cliffano","download_url":"https://codeload.github.com/cliffano/datagen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233828168,"owners_count":18736581,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","data-generator","nodejs"],"created_at":"2024-10-09T19:45:25.698Z","updated_at":"2025-09-22T06:31:48.273Z","avatar_url":"https://github.com/cliffano.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg align=\"right\" src=\"https://raw.github.com/cliffano/datagen/master/avatar.jpg\" alt=\"Avatar\"/\u003e\n\n[![Build Status](https://img.shields.io/travis/cliffano/datagen.svg)](http://travis-ci.org/cliffano/datagen)\n[![Dependencies Status](https://img.shields.io/david/cliffano/datagen.svg)](http://david-dm.org/cliffano/datagen)\n[![Coverage Status](https://img.shields.io/coveralls/cliffano/datagen.svg)](https://coveralls.io/r/cliffano/datagen?branch=master)\n[![Published Version](https://img.shields.io/npm/v/datagen.svg)](http://www.npmjs.com/package/datagen)\n\u003cbr/\u003e\n[![npm Badge](https://nodei.co/npm/datagen.png)](http://npmjs.org/package/datagen)\n\nDataGen\n-------\n\nDataGen is a multi-process test data files generator.\n\nThis is handy when you want to generate large test data files (e.g. XMLs, JSONs, CSVs, etc), over multiple processes, utilising available CPU cores on your machine. It's also very easy to generate random numbers, dates, and strings as test data. You only need to create template files, no scripting involved.\n\nInstallation\n------------\n\n    npm install -g datagen \n\nUsage\n-----\n\nCreate example header, segment, and footer template files:\n\n    datagen init\n\nGenerate 8 data files containing 1 million segments in each file, written to data1 ... data8 output files, running over 8 processes, each process is used to generate 1 file:\n\n    datagen gen -s 1000000 -w 8 -o data\n\nLimit the number of worker processes that can run concurrently to just 3:\n\n    datagen gen -s 1000000 -w 8 -m 3\n\nTemplates\n---------\n\nDataGen uses three template files: header, segment, and footer. These templates are simple text files which are used to construct a data file in this format:\n\n    header\n    segment 1\n    segment 2\n    ...\n    segment N (number of segments)\n    footer\n\nExample header:\n    \n    \u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n    \u003cdata\u003e\n\nExample segment:\n\n    \u003csegment\u003e\n      \u003cid\u003e{gen_id}-{worker_id}-{segment_id}\u003c/id\u003e\n      \u003cname\u003e{first_name()} {last_name()}\u003c/name\u003e\n      \u003cdob\u003e{date('dd-mm-yyyy')}\u003c/dob\u003e\n    \u003c/segment\u003e\n\nExample footer:\n\t\n    \u003c/data\u003e\n\nThe above templates will generate an XML like this:\n\n    \u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n    \u003cdata\u003e\n    \u003csegment\u003e\n      \u003cid\u003e1-1-1\u003c/id\u003e\n      \u003cname\u003eNiels Bryant\u003c/name\u003e\n      \u003cdob\u003e12-08-1992\u003c/dob\u003e\n    \u003c/segment\u003e\n    \u003csegment\u003e\n      \u003cid\u003e1-1-2\u003c/id\u003e\n      \u003cname\u003eJohn Bohr\u003c/name\u003e\n      \u003cdob\u003e01-11-1970\u003c/dob\u003e\n    \u003c/segment\u003e\n    ...\n    \u003c/data\u003e\n\nCurly brackets in templates (e.g. JSON format) need to be escaped with double brackets:\n\n    {{ \"id\": \"{gen_id}-{worker_id}-{segment_id}\", \"name\": \"{first_name()} {last_name()}\", \"dob\": \"{date('dd-mm-yyyy')}\" }}\n\nAs for CSV, check out the example below.\n\nExample header:\n\n    (Column1,Column2,Column3,Column4,Column5)\n    FirstName,LastName,Age,EmployeeNumber,DateJoint\n\nExample segment:\n\n    john,Smith,{integer(18, 65)},{integer()},{date('yyyy-mm-dd', 2000, 2018)}\n\nFooter is not required for CSV unless you ar using an ETL which my require a data record count.\n\nTemplates can contain the following parameters:\n\n\u003ctable\u003e\n\u003ctr\u003e\u003ctd\u003e{gen_id}\u003c/td\u003e\u003ctd\u003eUnique to each datagen execution. Default value is master process PID, can be overridden via -i flag.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{worker_id}\u003c/td\u003e\u003ctd\u003eUnique to each worker. Value from 1 to number of workers.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{segment_id}\u003c/td\u003e\u003ctd\u003eUnique to each segment within the generated data file, repeated in each file. Value from 1 to number of segments. Not available in header and footer templates.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{integer()}\u003c/td\u003e\u003ctd\u003eRandom integer.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{integer(100, 200)}\u003c/td\u003e\u003ctd\u003eRandom integer between 100 and 200.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{float()}\u003c/td\u003e\u003ctd\u003erandom float.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{float(1.23, 5.67)}\u003c/td\u003e\u003ctd\u003eRandom float between 1.23 and 5.67 .\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{date()}\u003c/td\u003e\u003ctd\u003erandom date between 1970 and 2020 in ISO format.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{date('yyyy-mm-dd')}\u003c/td\u003e\u003ctd\u003eRandom date between 1970 and 2020 in yyyy-mm-dd format.\u003cbr/\u003eCheck out \u003ca href=\"http://github.com/felixge/node-dateformat\"\u003efelixge/node-dateformat\u003c/a\u003e for more date formats.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{date(2000, 2010)}\u003c/td\u003e\u003ctd\u003eRandom date between 2000 and 2010 in ISO format.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{date('yyyy-mm-dd', 2000, 2010)}\u003c/td\u003e\u003ctd\u003eRandom date between 2000 and 2010 in yyyy-mm-dd format.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{select('apple', 'orange', ..., 'kiwi')}\u003c/td\u003e\u003ctd\u003eSelect one item from the arguments. You can have as many arguments as you want.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{word()}\u003c/td\u003e\u003ctd\u003eRandom word from Lorem Ipsum.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{word(3)}\u003c/td\u003e\u003ctd\u003eRandom 3 words from Lorem Ipsum.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{first_name()}\u003c/td\u003e\u003ctd\u003eRandom first name.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{last_name()}\u003c/td\u003e\u003ctd\u003eRandom last name.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{email()}\u003c/td\u003e\u003ctd\u003eRandom email address.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{phone()}\u003c/td\u003e\u003ctd\u003eRandom phone number. Default format #### ####.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e{phone('(###) ####-####')}\u003c/td\u003e\u003ctd\u003eRandom phone number with a custom format. Each '#' will be replaced by a random number.\u003c/td\u003e\u003c/tr\u003e\n\u003c/table\u003e\n\nScreenshots\n-----------\n\n![Command init](https://raw.github.com/cliffano/datagen/master/screenshots/init.jpg)\n![Command gen with flags](https://raw.github.com/cliffano/datagen/master/screenshots/gen-flags.jpg)\n\nColophon\n--------\n\n[Developer's Guide](https://cliffano.github.io/developers_guide.html#nodejs)\n\nBuild reports:\n\n* [Code complexity report](https://cliffano.github.io/datagen/complexity/plato/index.html)\n* [Unit tests report](https://cliffano.github.io/datagen/test/buster.out)\n* [Test coverage report](https://cliffano.github.io/datagen/coverage/buster-istanbul/lcov-report/lib/index.html)\n* [Integration tests report](https://cliffano.github.io/datagen/test-integration/cmdt.out)\n* [API Documentation](https://cliffano.github.io/datagen/doc/dox-foundation/index.html)\n\nArticles:\n\n* [DataGen Workers Optimisation](http://blog.cliffano.com/2013/08/03/datagen-workers-optimisation/)\n* [DataGen: Generate Large Test Data Files – Like A Boss](http://blog.cliffano.com/2012/07/08/datagen-generate-large-test-data-files-like-a-boss/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcliffano%2Fdatagen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcliffano%2Fdatagen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcliffano%2Fdatagen/lists"}