{"id":23979396,"url":"https://github.com/mrinalxdev/map-reduce-paper","last_synced_at":"2025-09-01T20:40:23.464Z","repository":{"id":270436734,"uuid":"909947839","full_name":"mrinalxdev/map-reduce-paper","owner":"mrinalxdev","description":"Implementation of the paper, MapReduce: Simplified Data Processing on Large Clusters","archived":false,"fork":false,"pushed_at":"2024-12-31T06:08:31.000Z","size":11,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-07T09:48:44.318Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrinalxdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-30T06:04:30.000Z","updated_at":"2024-12-31T06:08:34.000Z","dependencies_parsed_at":"2024-12-31T06:30:27.537Z","dependency_job_id":"c148c00b-4875-402d-abe3-788afc420066","html_url":"https://github.com/mrinalxdev/map-reduce-paper","commit_stats":null,"previous_names":["mrinalxdev/map-reduce-paper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrinalxdev%2Fmap-reduce-paper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrinalxdev%2Fmap-reduce-paper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrinalxdev%2Fmap-reduce-paper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrinalxdev%2Fmap-reduce-paper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrinalxdev","download_url":"https://codeload.github.com/mrinalxdev/map-reduce-paper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240557489,"owners_count":19820360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-07T09:48:51.149Z","updated_at":"2025-02-24T21:09:53.015Z","avatar_url":"https://github.com/mrinalxdev.png","language":"Go","readme":"# MapReduce Implementation in Go\n\nA distributed MapReduce implementation using Go, Redis, and RabbitMQ.\n\n## Prerequisites\n\n- Go 1.21 or higher\n- Docker and Docker Compose\n- Make (optional)\n\n## Setup and Running\n\n1. Clone the repository:\n```bash\ngit clone \u003cyour-repo\u003e\ncd mapreduce\n```\n\n2. Initialize Go module:\n```bash\ngo mod init mapreduce\ngo mod tidy\n```\n\n3. Build and run with Docker Compose:\n```bash\ndocker-compose up --build\n```\n\n## Testing the System\n\n1. Create an input file:\n```bash\necho \"example input data for testing word count\" \u003e input.txt\n```\n\n2. Process the file:\n```bash\ncat input.txt | docker-compose exec master ./master\n```\n\n3. View results:\n```bash\ndocker-compose exec redis redis-cli HGETALL map_results\n```\n\n## Architecture\n\n- Master Node: Coordinates the MapReduce workflow\n- Worker Nodes: Process map and reduce tasks\n- Redis: Stores intermediate results\n- RabbitMQ: Handles task distribution\n\n## Monitoring\n\n- RabbitMQ Management UI: http://localhost:15672 (guest/guest)\n- Redis Commander: http://localhost:8081\n\n## Scaling Workers\n\nTo scale the number of workers:\n```bash\ndocker-compose up --scale worker=5\n```\n\n# example_input.txt\nLorem ipsum dolor sit amet, consectetur adipiscing elit. \nSed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\nUt enim ad minim veniam, quis nostrud exercitation ullamco laboris \nnisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in \nreprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrinalxdev%2Fmap-reduce-paper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrinalxdev%2Fmap-reduce-paper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrinalxdev%2Fmap-reduce-paper/lists"}