{"id":26824124,"url":"https://github.com/serihiro/simple_map_reduce","last_synced_at":"2025-04-28T11:45:52.386Z","repository":{"id":28192759,"uuid":"116650595","full_name":"serihiro/simple_map_reduce","owner":"serihiro","description":"Distributed MapReduce implementation written in ruby.","archived":false,"fork":false,"pushed_at":"2025-03-01T02:58:37.000Z","size":245,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-01T03:19:38.437Z","etag":null,"topics":["distributed-systems","mapreduce","ruby","sinatra"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serihiro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-08T08:33:36.000Z","updated_at":"2025-03-01T02:58:41.000Z","dependencies_parsed_at":"2025-03-01T03:29:29.016Z","dependency_job_id":null,"html_url":"https://github.com/serihiro/simple_map_reduce","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serihiro%2Fsimple_map_reduce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serihiro%2Fsimple_map_reduce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serihiro%2Fsimple_map_reduce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serihiro%2Fsimple_map_reduce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serihiro","download_url":"https://codeload.github.com/serihiro/simple_map_reduce/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246296569,"owners_count":20754635,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","mapreduce","ruby","sinatra"],"created_at":"2025-03-30T09:18:06.588Z","updated_at":"2025-03-30T09:18:07.195Z","avatar_url":"https://github.com/serihiro.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Gem Version](https://badge.fury.io/rb/simple_map_reduce.svg)](https://badge.fury.io/rb/simple_map_reduce)\n\n# SimpleMapReduce\nThis is a [MapReduce](https://research.google.com/archive/mapreduce.html) distributed framework written in Ruby.\nThis project is an experimental project. So all the specifications will be changed suddenly.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'simple_map_reduce'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install simple_map_reduce\n\n## Quick start in local environment with minio\n\n### 1. Start minio server\n\n```sh\n$ docker run -p 9000:9000 -p 9001:9001  \\\n-e \"MINIO_ROOT_USER=MINIO_ACCESS_KEY\" -e \"MINIO_ROOT_PASSWORD=MINIO_SECRET_KEY\" -e \"MINIO_REGION=us-east-1\" \\\nminio/minio server /data --console-address :9001\n```\n\n### 2. Start job tracker\n\n```sh\n$ bundle exec simple_map_reduce run_job_tracker! \\\n  --job-tracker-url=http://job_tracker:4567 \\\n  --server-port=4567 \\\n  --s3_config=access_key_id:'MINIO_ACCESS_KEY' \\\n              secret_access_key:'MINIO_SECRET_KEY' \\\n              endpoint:'http://127.0.0.1:9000' \\\n              region:'us-east-1' \\\n              force_path_style:true\n```\n\n### 3. Start job worker(s)\n\n```sh\n$ bundle exec simple_map_reduce run_job_worker! \\\n  --job-tracker-url=http://localhost:4567 \\\n  --job-worker-url=http://localhost:4568 \\\n  --server-port=4568 \\\n  --s3_config=access_key_id:'MINIO_ACCESS_KEY' \\\n              secret_access_key:'MINIO_SECRET_KEY' \\\n              endpoint:'http://127.0.0.1:9000' \\\n              region:'us-east-1' \\\n              force_path_style:true\n```\n\n### 4. Generate and upload test data\n\n```sh\n$ bundle exec simple_map_reduce generate_lorem_text_data --upload=true\n```\n\n### 5. Execute word count job\n\n```sh\n$ bundle exec simple_map_reduce execute_word_count\n```\n\n## Quick start in Docker Compose\n\n- You can setup a simple_map_reduce cluster by docker compose.\n\n```sh\n$ clone git@github.com:serihiro/simple_map_reduce.git\n$ cd simple_map_reduce\n$ docker compose up\n```\n\n- You can execute word count sample by executing following commands\n\n```sh\n$ docker compose exec job_tracker bundle exec simple_map_reduce generate_lorem_text_data --upload=true\n$ docker compose exec job_tracker bundle exec simple_map_reduce execute_word_count\n```\n\n## Motivation of this project\nI would have liked to lean the theory of distributed systems, big data processing, and MapReduce algorhythm.\nIn my experiences, I believed that an implementation of them is the best way to learn them.\nSo I decided to create an experimental implementation, and keep adding new features in order to get an practical experiences of the theories.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/simple_map_reduce. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the SimpleMapReduce project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/simple_map_reduce/blob/master/CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserihiro%2Fsimple_map_reduce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserihiro%2Fsimple_map_reduce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserihiro%2Fsimple_map_reduce/lists"}