{"id":15060689,"url":"https://github.com/googleclouddataproc/hadoop-connectors","last_synced_at":"2026-03-09T10:04:29.387Z","repository":{"id":16922943,"uuid":"19684359","full_name":"GoogleCloudDataproc/hadoop-connectors","owner":"GoogleCloudDataproc","description":"Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.","archived":false,"fork":false,"pushed_at":"2026-02-27T13:15:38.000Z","size":69052,"stargazers_count":289,"open_issues_count":101,"forks_count":260,"subscribers_count":93,"default_branch":"master","last_synced_at":"2026-02-27T18:33:05.009Z","etag":null,"topics":["bigquery","google-cloud-dataproc","hadoop","hadoop-filesystem","hadoop-hcfs"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudDataproc.png","metadata":{"files":{"readme":"README-template.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2014-05-12T03:11:55.000Z","updated_at":"2026-02-16T04:21:40.000Z","dependencies_parsed_at":"2023-10-20T11:43:35.908Z","dependency_job_id":"71fd0bed-3b97-4652-8d00-b231f8196828","html_url":"https://github.com/GoogleCloudDataproc/hadoop-connectors","commit_stats":{"total_commits":1459,"total_committers":132,"mean_commits":"11.053030303030303","dds":0.5243317340644277,"last_synced_commit":"22e60aa756c83b87aa35fe4a088d4336c98d1d43"},"previous_names":["googlecloudplatform/bigdata-interop"],"tags_count":127,"template":false,"template_full_name":null,"purl":"pkg:github/GoogleCloudDataproc/hadoop-connectors","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudDataproc","download_url":"https://codeload.github.com/GoogleCloudDataproc/hadoop-connectors/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors/sbom","scorecard":{"id":57642,"data":{"date":"2025-08-11","repo":{"name":"github.com/GoogleCloudDataproc/hadoop-connectors","commit":"7ffa95db5a5f1bdd6416667aeceb5af60e64272c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":5.2,"checks":[{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":10,"reason":"14 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: cloudbuild/Dockerfile:16: pin your Docker image by updating openjdk:11-jdk to openjdk:11-jdk@sha256:99bac5bf83633e3c7399aed725c8415e7b569b54e03e4599e580fc9cdb7c21ab","Warn: downloadThenRun not pinned by hash: cloudbuild/presubmit.sh:31","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   1 downloadThenRun dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v3.0.9 not signed: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/225487494","Warn: release artifact v3.1.3 not signed: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/221841517","Warn: release artifact v3.0.8 not signed: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/221781798","Warn: release artifact v2.2.28 not signed: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/221781737","Warn: release artifact v2.2.27 not signed: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/217260445","Warn: release artifact v3.0.9 does not have provenance: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/225487494","Warn: release artifact v3.1.3 does not have provenance: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/221841517","Warn: release artifact v3.0.8 does not have provenance: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/221781798","Warn: release artifact v2.2.28 does not have provenance: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/221781737","Warn: release artifact v2.2.27 does not have provenance: https://api.github.com/repos/GoogleCloudDataproc/hadoop-connectors/releases/217260445"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-15T01:05:52.023Z","repository_id":16922943,"created_at":"2025-08-15T01:05:52.024Z","updated_at":"2025-08-15T01:05:52.024Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30291002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T02:57:19.223Z","status":"ssl_error","status_checked_at":"2026-03-09T02:56:26.373Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","google-cloud-dataproc","hadoop","hadoop-filesystem","hadoop-hcfs"],"created_at":"2024-09-24T23:03:04.887Z","updated_at":"2026-03-09T10:04:29.371Z","avatar_url":"https://github.com/GoogleCloudDataproc.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Hadoop Connectors\n\n[![GitHub release](https://img.shields.io/github/release/GoogleCloudDataproc/hadoop-connectors.svg)](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/latest)\n[![GitHub release date](https://img.shields.io/github/release-date/GoogleCloudDataproc/hadoop-connectors.svg)](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/latest)\n[![codecov](https://codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/branch/master/graph/badge.svg)](https://codecov.io/gh/GoogleCloudDataproc/hadoop-connectors)\n\nLibraries and tools for interoperability between Apache Hadoop related\nopen-source software and Google Cloud Platform.\n\n## Google Cloud Storage connector for Apache Hadoop (HCFS)\n\n[![Maven Central](https://img.shields.io/maven-central/v/com.google.cloud.bigdataoss/gcs-connector/hadoop1.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:com.google.cloud.bigdataoss%20AND%20a:gcs-connector%20AND%20v:hadoop1-*)\n[![Maven Central](https://img.shields.io/maven-central/v/com.google.cloud.bigdataoss/gcs-connector/hadoop2.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:com.google.cloud.bigdataoss%20AND%20a:gcs-connector%20AND%20v:hadoop2-*)\n[![Maven Central](https://img.shields.io/maven-central/v/com.google.cloud.bigdataoss/gcs-connector/hadoop3.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:com.google.cloud.bigdataoss%20AND%20a:gcs-connector%20AND%20v:hadoop3-*)\n\nThe Google Cloud Storage connector for Hadoop enables running MapReduce jobs\ndirectly on data in Google Cloud Storage by implementing the Hadoop FileSystem\ninterface. For details, see [the README](gcs/README.md).\n\n## Unreleased Changes\n\nThis Readme may include documentation for changes that haven't been released yet.  The latest release's documentation and source code are found here.\n\nhttps://github.com/GoogleCloudDataproc/hadoop-connectors/tree/mastr\n\n## Building the Cloud Storage connector\n\n\u003e Note that build requires Java 11+ and fails with older Java versions.\n\nTo build the connector for specific Hadoop version, run the following commands\nfrom the main directory:\n\n```bash\n./mvnw clean package\n```\n\nIn order to verify test coverage for specific Hadoop version, run the following\ncommands from the main directory:\n\n```bash\n./mvnw -P coverage clean verify\n```\n\nThe Cloud Storage connector JAR can be found in `gcs/target/` directory.\n\n## Adding the Cloud Storage connector to your build\n\nMaven group ID is `com.google.cloud.bigdataoss` and artifact ID for Cloud\nStorage connector is `gcs-connector`.\n\nTo add a dependency on Cloud Storage connector using Maven, use the following:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.google.cloud.bigdataoss\u003c/groupId\u003e\n  \u003cartifactId\u003egcs-connector\u003c/artifactId\u003e\n  \u003cversion\u003e${next-gcs-connector-release-tag}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Resources\n\nOn **Stack Overflow**, use the tag\n[`google-cloud-dataproc`](https://stackoverflow.com/tags/google-cloud-dataproc)\nfor questions about the connectors in this repository. This tag receives\nresponses from the Stack Overflow community and Google engineers, who monitor\nthe tag and offer unofficial support.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fhadoop-connectors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogleclouddataproc%2Fhadoop-connectors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fhadoop-connectors/lists"}