{"id":41951033,"url":"https://github.com/adamretter/shadoop","last_synced_at":"2026-02-04T19:00:28.451Z","repository":{"id":9827093,"uuid":"11813710","full_name":"adamretter/Shadoop","owner":"adamretter","description":"A wrapper for Hadoop in Scala","archived":false,"fork":true,"pushed_at":"2014-09-16T17:10:27.000Z","size":445,"stargazers_count":10,"open_issues_count":5,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-06T00:42:34.501Z","etag":null,"topics":["hadoop","mapreduce","scala"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"bsdfish/ScalaHadoop","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adamretter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-08-01T10:04:40.000Z","updated_at":"2018-09-04T00:30:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/adamretter/Shadoop","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/adamretter/Shadoop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamretter%2FShadoop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamretter%2FShadoop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamretter%2FShadoop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamretter%2FShadoop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adamretter","download_url":"https://codeload.github.com/adamretter/Shadoop/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamretter%2FShadoop/sbom","scorecard":{"id":165492,"data":{"date":"2025-08-11","repo":{"name":"github.com/adamretter/Shadoop","commit":"e23471e1d3346ca826a033145af4cd30f18b0006"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.9,"checks":[{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":1,"reason":"Found 4/29 approved changesets -- score normalized to 1","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 5 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"18 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-5m48-vr54-vmh3","Warn: Project is vulnerable to: GHSA-6phf-73q6-gh87","Warn: Project is vulnerable to: GHSA-wxr5-93ph-8wr9","Warn: Project is vulnerable to: GHSA-6hgm-866r-3cjv","Warn: Project is vulnerable to: GHSA-fjq5-5j5f-mvxh","Warn: Project is vulnerable to: GHSA-pvp8-3xj6-8c6x","Warn: Project is vulnerable to: GHSA-3832-9276-x7gf","Warn: Project is vulnerable to: GHSA-78wr-2p64-hpwj","Warn: Project is vulnerable to: GHSA-gwrp-pvrq-jmwv","Warn: Project is vulnerable to: GHSA-j288-q9x7-2f5v","Warn: Project is vulnerable to: GHSA-cgp8-4m63-fhh5","Warn: Project is vulnerable to: GHSA-c27h-mcmw-48hv","Warn: Project is vulnerable to: GHSA-r6j9-8759-g62w","Warn: Project is vulnerable to: GHSA-56h3-78gp-v83r","Warn: Project is vulnerable to: GHSA-7rf3-mqpx-h7xg","Warn: Project is vulnerable to: GHSA-grr4-wv38-f68w","Warn: Project is vulnerable to: GHSA-q6g2-g7f3-rr83","Warn: Project is vulnerable to: GHSA-x27m-9w8j-5vcw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-16T14:42:42.079Z","repository_id":9827093,"created_at":"2025-08-16T14:42:42.079Z","updated_at":"2025-08-16T14:42:42.079Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29092981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-04T03:31:03.593Z","status":"ssl_error","status_checked_at":"2026-02-04T03:29:50.742Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","mapreduce","scala"],"created_at":"2026-01-25T21:00:21.273Z","updated_at":"2026-02-04T19:00:28.441Z","avatar_url":"https://github.com/adamretter.png","language":"Scala","funding_links":[],"categories":["Big Data"],"sub_categories":[],"readme":"# Shadoop\n\nA Hadoop DSL and lightweight wrapper for Scala\n\n\n[![Build Status](https://travis-ci.org/adamretter/Shadoop.png?branch=master)](https://travis-ci.org/adamretter/Shadoop)\n\nThis fork of ScalaHadop is mostly just cherry-picked commits from the forks by [@hito-asa](https://github.com/hiti-asa/ScalaHadoop), [@ivmaykov](https://github.com/ivmaykov/ScalaHadoop) and [@oscarrenalis](https://github.com/oscarrenalias/ScalaHadoop), of the original work by [@bsdfish](https://github.com/bsdfish/ScalaHadoop). In addition there are a few extra features and a cleaned up Maven build.\n\nThis code provides some syntactic sugar on top of Hadoop in order to make\nit more usable from Scala.  Take a look at src/main/scala/net/renalias/scoop/examples/WordCount.scala for more\ndetails.\n\n## License\n[Apache License, Version 2.0](http://opensource.org/licenses/Apache-2.0)\n\n## Usage\n### Basic Usage\nA basic mapper looks like:\n\n```scala\nval mapper = new Mapper[LongWritable, Text, Text, LongWritable] {\n    mapWith {\n        (k, v) =\u003e\n            (v split \" |\\t\").map(x =\u003e (new Text(x), new LongWritable(1L))).toList\n    }\n}\n```\n\na reducer looks like this:\n\n```scala\nval reducer = new Reducer[Text, LongWritable, Text, LongWritable] {\n    reduceWith {\n        (k, v) =\u003e\n            List((k, (0L /: v)((total, next) =\u003e total + next)))\n    }\n}\n```\n\nand, the pipeline to bind them together may look like this:\n\n```scala\nTextInput[LongWritable, Text](\"/tmp/input.txt\") --\u003e\nMapReduceTask(mapper, reducer, \"Word Count\")    --\u003e\nTextOutput[Text, LongWritable](\"/tmp/output\")   execute\n```\n\n\nThe key difference here between standard mappers and reducers is that the map and reduce parts are written as side-effect\nfree functions that accept a key and a value, and return an iterable; code behind the scenes will take care of\nupdating Hadoop's Context object.\n\nSome note still remains to be done to polish the current interface, to remove things like .toList from the mapper and\nthe creation of Hadoop's specific Text and LongWritable objects.\n\nNote that implicit conversion is used to convert between LongWritable and longs, as well as Text\nand Strings.  The types of the input and output parameters only need to be stated as the\ngeneric specializers of the class it extends.\n\nThese mappers and reducers can be chained together with the --\u003e operator:\n\n```scala\nobject WordCount extends ScalaHadoop {\n  def run(args: Array[String]) : Int = {\n    TextInput[LongWritable, Text](args(0)) --\u003e\n    MapReduceTask(mapper, reducer, \"Main task\") --\u003e\n    TextOutput[Text, LongWritable](args(1)) execute\n\n    0 //result code\n  }\n}\n```\n\n### Multiple map/reduce\nMultiple map/reduce runs may be chained together:\n\n```scala\nobject WordsWithSameCount extends ScalaHadoop {\n  def run(args: Array[String]) : Int = {\n    TextInput[LongWritable, Text](args(0)) --\u003e\n    MapReduceTask(tokenizerMap1, sumReducer, \"Sum\") --\u003e\n    MapReduceTask(flipKeyValueMap, wordListReducer, \"Reduce\") --\u003e\n    TextOutput[LongWritable, Text](args(1)) execute\n\n    0 //result code\n  }\n}\n```\n\n## Contributors\n- **Alex Simma**: Developer of original version of ScalaHadoop. https://github.com/bsdfish/ScalaHadoop\n- **ASAI Hitoshi**: Cherry-picked - Code re-organisation and initial Maven build. https://github.com/hiti-asa/ScalaHadoop\n- **Ilya Maykov**: Cherry-picked - Various fixes, and support for Multiple Input Paths. https://github.com/ivmaykov/ScalaHadoop\n- **Oscar Renalias**: Cherry-picked - Scala Syntax improvements. https://github.com/oscarrenalias/ScalaHadoop\n- **Rob Walpole**: Various bug fixes: https://github.com/rwalpole/ScalaHadoop\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamretter%2Fshadoop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadamretter%2Fshadoop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamretter%2Fshadoop/lists"}