{"id":38040428,"url":"https://github.com/jsteinberg4/icarus","last_synced_at":"2026-01-16T19:53:54.165Z","repository":{"id":229503721,"uuid":"776904716","full_name":"jsteinberg4/icarus","owner":"jsteinberg4","description":"I Could Actually Really Use Support (ICARUS): A custom implementation of MapReduce","archived":false,"fork":false,"pushed_at":"2024-04-14T00:19:34.000Z","size":100,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-14T00:23:30.317Z","etag":null,"topics":["cpp","distributed-systems","mapreduce"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jsteinberg4.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-24T19:05:32.000Z","updated_at":"2024-04-15T18:57:06.454Z","dependencies_parsed_at":"2024-04-15T19:07:10.255Z","dependency_job_id":null,"html_url":"https://github.com/jsteinberg4/icarus","commit_stats":null,"previous_names":["jsteinberg4/mapreduce","jsteinberg4/icarus"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jsteinberg4/icarus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsteinberg4%2Ficarus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsteinberg4%2Ficarus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsteinberg4%2Ficarus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsteinberg4%2Ficarus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jsteinberg4","download_url":"https://codeload.github.com/jsteinberg4/icarus/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsteinberg4%2Ficarus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28482107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","distributed-systems","mapreduce"],"created_at":"2026-01-16T19:53:54.101Z","updated_at":"2026-01-16T19:53:54.160Z","avatar_url":"https://github.com/jsteinberg4.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ICARUS ![build](https://github.com/jsteinberg4/icarus/actions/workflows/build.yml/badge.svg)\n\n_I Could Actually Really Use Support (ICARUS)_: A MapReduce Implementation\n\n\u003c!--toc:start--\u003e\n\n- [Getting Started](#getting-started)\n  - [Setup](#setup)\n  - [Usage](#usage)\n  - [Examples](#examples)\n- [Citations](#citations)\n  \u003c!--toc:end--\u003e\n\n## Getting Started\n\n### Setup\n\nDependencies: `make`, a Unix platform which supports C++14\n\nFrom the repository root, run the following:\n\n```bash\n# To compile the binaries and setup intermediate folders\n$ make all\n\n# To discard previous runs output \u0026 old binaries\n$ make clean all\n```\n\nThe following are some additional useful make targets:\n\n```bash\n# Delete the binaries in bin/*. Careful, this also deletes the intermediate files generated by mapper and reducer. Final map reduce outputs are left alone.\nmake clean\n\n# Only delete the intermediate files generated for map and reduce. Don't delete any binaries. Use if your disk is getting cluttered.\nmake reset-inputs\n```\n\nThis will generate executables to `bin/`.\n\n```sh\nbin/\n  master   # A MapReduce master node. Run on an isolated VM.\n  worker   # MapReduce workers. Run one per virtual machine.\n  mapper   # The Map binary. A word counting algorithm is implemented by default. Do not run directly.\n  reducer  # The Reduce binary. A summation for the word counting algorithm is implemented by default. Do not run directly.\n```\n\n**Optional Development Dependencies**: python3, [virtualenv](https://virtualenv.pypa.io/en/latest/), [compiledb](https://github.com/nickdiego/compiledb) \u003cbr\u003e\n`make` does not generate a `compile_commands.json` to help the clang LSP interpret code files. I use `compiledb` to generate these.\n\n```bash\n# 1) Make a virtual environment\npython3 -m pip install virtualenv # (you may already have this installed)\npython3 -m virtualenv venv\n\n# 2) Install dev dependencies\nsource venv/bin/activate\npip install -r requirements-compile.txt\n\n# 3) Generate compile_commands.json\nmake clean # Force a complete rebuild\ncompiledb make all\n```\n\n### Usage\n\nAll run instructions assume the repository root to be your working directory. Each of the usage messages will be printed by running `bin/master` or `bin/worker` with no arguments.\n\n**Running the Master**:\u003cbr /\u003e\n\nNote, the final result will be written to the folder `mapReduceOutputs/`. The master prints the absolute file path to stdout upon completion.\n\n```\n$ bin/master\nUsage:\n    bin/master [port] [root directory] [input path] [# mappers]\n\n    port: Specify which TCP port to listen at\n    root directory: Specify an absolute path as the working directory. All other filepaths internally will use this as a base. It will almost always be the repository root.\n    input path: Specify the task's input file. Assumed relative to the root.\n    mappers: Specify the number of map tasks to create from the input file\n```\n\n**Running the Workers**:\u003cbr /\u003e\n\n```\n$ bin/worker\nUsage:\n  bin/worker \u003cmaster IP\u003e \u003cmaster port\u003e \u003cnum workers\u003e [(optional) failure chance]\n\n  master ip: the IP address used by bin/master\n\n  master port: the open port specified as \u003cport\u003e when running bin/master\n\n  num workers: Specify the size of the worker pool. A value of 0 will run a single worker instance which exits the whole program on errors. Any value 1...N will maintain a pool of N child processes, each of which independently connects to the master.\n\n  failure chance: If provided, enables simulated worker failures by killing child processes with probability 1 in \u003cfailure chance\u003e. For example, a value of 5 means workers will be killed with probability 1 in 5 (20%). Num workers must be at least 1. If not provided, failure simulation is skipped.\n```\n\n### Examples:\n\nTo run the Word Counter benchmark with the master node listening on port 80080 and 100 map partitions. Run 4 virtual nodes per bin/worker execution with a 20% chance of simulated failures.\n\n```bash\n# On one virtual machine (or terminal):\n$ bin/master 80080 $(pwd) inputs/triple_large.txt 100\n\n# On different virtual machines/terminals\nbin/worker \u003cmaster_ip\u003e 80080 4 5\n```\n\nTo run the word counter algorithm, using the complete works of Shakespeare partitioned into 100 map tasks, using 4 workers and no simulated failures.\n\n```bash\n# Run the master at host 12.34.56.78:54321\n$ bin/master 54321  $(pwd) inputs/complete_shakespeare.txt 100\n\n# Run the worker pool\n$ bin/worker 12.34.56.78 54321 4\n# Equivalent: bin/worker 12.34.56.78 54321 4 0\n```\n\n## Citations\n\n[Data sources](/inputs/CITATIONS.md)\n\n### Original Authors\n\n\u003e Jeffrey Dean and Sanjay Ghemawat. OSDI'04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA (2004), pp. 137-150. https://research.google/pubs/mapreduce-simplified-data-processing-on-large-clusters/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsteinberg4%2Ficarus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjsteinberg4%2Ficarus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsteinberg4%2Ficarus/lists"}