{"id":16174428,"url":"https://github.com/hellothisisflo/simple-mapreduce","last_synced_at":"2025-07-23T15:07:21.302Z","repository":{"id":93127271,"uuid":"82329384","full_name":"HelloThisIsFlo/Simple-MapReduce","owner":"HelloThisIsFlo","description":"A Simple MapReduce using Elixir and the power of OTP","archived":false,"fork":false,"pushed_at":"2017-07-11T10:30:54.000Z","size":23,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-05T23:08:57.738Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HelloThisIsFlo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-02-17T19:10:15.000Z","updated_at":"2017-02-17T19:10:31.000Z","dependencies_parsed_at":"2023-06-05T02:00:18.351Z","dependency_job_id":null,"html_url":"https://github.com/HelloThisIsFlo/Simple-MapReduce","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HelloThisIsFlo/Simple-MapReduce","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelloThisIsFlo%2FSimple-MapReduce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelloThisIsFlo%2FSimple-MapReduce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelloThisIsFlo%2FSimple-MapReduce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelloThisIsFlo%2FSimple-MapReduce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HelloThisIsFlo","download_url":"https://codeload.github.com/HelloThisIsFlo/Simple-MapReduce/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelloThisIsFlo%2FSimple-MapReduce/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266699694,"owners_count":23970542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T04:24:52.993Z","updated_at":"2025-07-23T15:07:21.295Z","avatar_url":"https://github.com/HelloThisIsFlo.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SimpleMapreduce\n\nA simple distributed and load-balanced MapReduce.\n\nIt is fault tolerent, but does not guarantee that all events are parsed.\n\nWhen a batch fail, the worker is immediately restarted and continue to accept new batches, but the failed batch is lost.\nThe same happen when a machine fail, the batch that was being parsed is lost.\n\n## Initial setup\nImplement the `SimpleMapreduce.HeavyWork` behavior, and provide it to the configuration file.\nAlso provide the names of the nodes: \n\n* `main_node` is where the pipeline is located, it should be the same on each machine. \n              It is used by the worker to know where is located the pipeline when connecting to it.\n\n* `workers_node` is the default node where the workers are started.\n                 Usually it is different on each node, and it is the name of the current node (`node()`).\n\nA good practice is to put the node configuration in an external file `nodes.exs` and import it in the main config.\nThat allows to remove the `node.exs` from version control.\nWhich is useful since it is different on each machine\n\nFinally set the max demand for each worker. (Check `GenStage.sync_subscribe` documentation for more details)\n\n```\nconfig :simple_mapreduce,\n  heavy_work_module: MyApplication.MyHeavyWorkModule,\n  max_demand_for_worker: 20\n\nconfig :simple_mapreduce,\n  main_node: :\"main@MACHINE-NAME\",\n  workers_node: :\"worker@MACHINE-NAME\"\n\n```\n\n### Important: After changing the configuration. Refetch the dependency\n\n```\nmix deps.clean simple_mapreduce\nmix deps.get\n```\nThis is a known bug, and will be addressed: [#1](https://github.com/ShockN745/Simple-MapReduce/issues/1)\n\n## After initial setup\n\n1. Start a pipeline by name:\n  *  `SimpleMapreduce.start_new_pipeline(PIPELINE_NAME)`\n\n2. Add workers on the main machine: \n  *  `SimpleMapreduce.add_worker(PIPELINE_NAME, NODE_WHERE_TO_SPAWN_WORKER)`\n  * If NODE_WHERE_TO_SPAWN_WORKER is not provided, will use `:workers_node` from config\n  * NODE_WHERE_TO_SPAWN_WORKER can be `node()`\n\n3. Add **Extra** workers on all extra machines.\n  *  `SimpleMapreduce.add_extra_worker(PIPELINE_NAME, NODE_WHERE_TO_SPAWN_WORKER)`\n  * If NODE_WHERE_TO_SPAWN_WORKER is not provided, will use `:workers_node` from config\n  * NODE_WHERE_TO_SPAWN_WORKER can be `node()`\n  \n4. Perform work super fast by passing a list to the pipeline.\n  * `SimpleMapreduce.do_heavy_work_on_all_elements(ELEMENTS, PIPELINE_NAME, TIMEOUT)`\n  * TIMEOUT\n    * Can be :infinity\n    * Default is 3000ms\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhellothisisflo%2Fsimple-mapreduce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhellothisisflo%2Fsimple-mapreduce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhellothisisflo%2Fsimple-mapreduce/lists"}