{"id":20675297,"url":"https://github.com/sobakavosne/generator","last_synced_at":"2025-05-10T15:30:29.130Z","repository":{"id":155596890,"uuid":"341459916","full_name":"sobakavosne/generator","owner":"sobakavosne","description":"An application for generating 1 billion records (MySQL). Research \u0026 development project","archived":false,"fork":false,"pushed_at":"2021-05-18T15:04:40.000Z","size":6453,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"clean-and-lazy","last_synced_at":"2023-11-25T15:35:23.435Z","etag":null,"topics":["functional-programming","lazy","mysql","node"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sobakavosne.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-02-23T07:02:19.000Z","updated_at":"2023-11-25T15:35:24.459Z","dependencies_parsed_at":"2023-11-25T15:45:27.782Z","dependency_job_id":null,"html_url":"https://github.com/sobakavosne/generator","commit_stats":null,"previous_names":["sobakavosne/generator"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sobakavosne%2Fgenerator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sobakavosne%2Fgenerator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sobakavosne%2Fgenerator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sobakavosne%2Fgenerator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sobakavosne","download_url":"https://codeload.github.com/sobakavosne/generator/tar.gz/refs/heads/clean-and-lazy","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224970079,"owners_count":17400294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functional-programming","lazy","mysql","node"],"created_at":"2024-11-16T21:09:07.241Z","updated_at":"2024-11-16T21:09:07.839Z","avatar_url":"https://github.com/sobakavosne.png","language":"JavaScript","readme":"**Generator**\n\nApplication aimed to generate a large number of records (MySQL).\n\n**To-do list:**\n\n- [x] Create Generator\n- [x] Create Buffer\n- [x] Create Scheduler\n- [ ] Create Client (full run usage)\n- [ ] Mark the phonesIO object on the heap to use it in the contact generation to avoid buffering\n\n**MySQL-generator usage**\n\nInstall npm packages:\n```bash\ncd mysql-generator\nnpm install\n```\n\nCreate `.env` file in `/mysql-generator` folder with the following content:\n\n```bash\ncat \u003e .env\n\nPORT=...                  # Server port (if needed)\n\nHOST=...                  # DB host\nUSER=...                  # DB username\nPSSWD=...                 # DB password\nDATABASE=...              # DB name\n\nN=...                     # Desired number of phones\nGENNUMBER=...             # Number of generations to send\nPHONELENGTH=...           # Phone length\nPSSWDLENGTH=...           # Password length\nHASHTAGLENGTH=...         # Hashtag length\nMINCONTACTSNUMBER=...     # Minimum of contacts per phone\nMAXCONTACTSNUMBER=...     # Maximum of contacts per phone\n\nTMPPATH=tmp\nTMPFILE=buffer.txt\n\nMARK=...                  # Mark of generated contacts\n                          # Mark increases on GENERATIONAMOUNT value\n                          # Mark indicates the scheduler entry point\n                          # Mark's default value equals 0\n                          # GENERATIONAMOUNT is a value that is calculated\n```\n\n**Phones** are generated in the following order:\n\n`f(`identity`) -\u003e Generation number (`GENERATION`) + Filling with zeros + Phone increment (`*F(identity)* ∌ GENERATION .. N`)`\n\n yumi_user_id | hashtag | password \n --- | --- | ---  \n 1000000000002 | 'fhdsuai' | 'fjsdig'\n ... | ... | ...\n\n**Contacts** are generated in the following order:\n\n`f(`phone`) -\u003e Last meaningful phone value (`*F(identity)* ∌ GENERATION .. N`) + Filling with zeros + Contact increment (`*F(identity)*`)`\n\n id | yumi_user_id | contact_user_id \n --- | --- | --- \n 1 | 1000000000002 | 2000000000001\n... | ... | ...\n\n**Minimum requirements** for generation process:\n - CPU - 2 physical cores\n - RAM - 12Gb\n\n**Restrictions**: generation number (`GENNUMBER`) concatenated with maximum phone increment (`~N`) cannot be more than phone length (`PHONELENGTH`).\n\nFirst monad (`phonesIO`): demonstration.\n\n![](static/demonstration/node-10M-generator.gif)\n\nSecond monad (`contactsIO`): The generation of a large number of records occurs through the sequential start of the child processes of `generateWithChildIOR` scheduler monad (`R` means ***recursively***).\n\n**Machine**:\n\n```bash\nOS: Ubuntu 16.04.6 LTS x86_64\nKernel: 4.4.0-1106-aws\nCPU: AMD EPYC 7571 (4) @ 2.199GHz \nRAM: 31797MiB\n```\n\n**Users** benchmarks:\n\n```bash\n1M records are generated for 1.9s.\n1M records are sent to MySQL for 1.5.\n1M records are inserted into DB for 8s.\n\n10M records are generated and sent by Node.js in: 35649.538ms.\n10M generation, sending and insertion occurs for 99906.295ms.\n```\n\n**Contacts** benchmarks (slow buffering):\n\n```bash\n1M contacts are generated and sent for 47s.\n1M records are inserted into DB for 10s.\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsobakavosne%2Fgenerator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsobakavosne%2Fgenerator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsobakavosne%2Fgenerator/lists"}