{"id":17720251,"url":"https://github.com/GoogleCloudDataproc/hadoop-connectors","last_synced_at":"2025-03-14T04:30:41.661Z","repository":{"id":16922943,"uuid":"19684359","full_name":"GoogleCloudDataproc/hadoop-connectors","owner":"GoogleCloudDataproc","description":"Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.","archived":false,"fork":false,"pushed_at":"2024-10-18T04:02:01.000Z","size":11492,"stargazers_count":280,"open_issues_count":80,"forks_count":237,"subscribers_count":96,"default_branch":"master","last_synced_at":"2024-10-29T15:47:49.400Z","etag":null,"topics":["bigquery","google-cloud-dataproc","hadoop","hadoop-filesystem","hadoop-hcfs"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudDataproc.png","metadata":{"files":{"readme":"README-template.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-05-12T03:11:55.000Z","updated_at":"2024-10-18T04:01:29.000Z","dependencies_parsed_at":"2023-10-20T11:43:35.908Z","dependency_job_id":"3a4764c5-d35e-4f65-a913-cfbe6119fd8f","html_url":"https://github.com/GoogleCloudDataproc/hadoop-connectors","commit_stats":{"total_commits":1459,"total_committers":132,"mean_commits":"11.053030303030303","dds":0.5243317340644277,"last_synced_commit":"22e60aa756c83b87aa35fe4a088d4336c98d1d43"},"previous_names":["googlecloudplatform/bigdata-interop"],"tags_count":90,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhadoop-connectors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudDataproc","download_url":"https://codeload.github.com/GoogleCloudDataproc/hadoop-connectors/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243526254,"owners_count":20305107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","google-cloud-dataproc","hadoop","hadoop-filesystem","hadoop-hcfs"],"created_at":"2024-10-25T15:26:35.024Z","updated_at":"2025-03-14T04:30:36.627Z","avatar_url":"https://github.com/GoogleCloudDataproc.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"# Apache Hadoop Connectors\n\n[![GitHub release](https://img.shields.io/github/release/GoogleCloudDataproc/hadoop-connectors.svg)](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/latest)\n[![GitHub release date](https://img.shields.io/github/release-date/GoogleCloudDataproc/hadoop-connectors.svg)](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/latest)\n[![codecov](https://codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/branch/master/graph/badge.svg)](https://codecov.io/gh/GoogleCloudDataproc/hadoop-connectors)\n\nLibraries and tools for interoperability between Apache Hadoop related\nopen-source software and Google Cloud Platform.\n\n## Google Cloud Storage connector for Apache Hadoop (HCFS)\n\n[![Maven Central](https://img.shields.io/maven-central/v/com.google.cloud.bigdataoss/gcs-connector/hadoop1.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:com.google.cloud.bigdataoss%20AND%20a:gcs-connector%20AND%20v:hadoop1-*)\n[![Maven Central](https://img.shields.io/maven-central/v/com.google.cloud.bigdataoss/gcs-connector/hadoop2.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:com.google.cloud.bigdataoss%20AND%20a:gcs-connector%20AND%20v:hadoop2-*)\n[![Maven Central](https://img.shields.io/maven-central/v/com.google.cloud.bigdataoss/gcs-connector/hadoop3.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:com.google.cloud.bigdataoss%20AND%20a:gcs-connector%20AND%20v:hadoop3-*)\n\nThe Google Cloud Storage connector for Hadoop enables running MapReduce jobs\ndirectly on data in Google Cloud Storage by implementing the Hadoop FileSystem\ninterface. For details, see [the README](gcs/README.md).\n\n## Unreleased Changes\n\nThis Readme may include documentation for changes that haven't been released yet.  The latest release's documentation and source code are found here.\n\nhttps://github.com/GoogleCloudDataproc/hadoop-connectors/tree/mastr\n\n## Building the Cloud Storage connector\n\n\u003e Note that build requires Java 11+ and fails with older Java versions.\n\nTo build the connector for specific Hadoop version, run the following commands\nfrom the main directory:\n\n```bash\n./mvnw clean package\n```\n\nIn order to verify test coverage for specific Hadoop version, run the following\ncommands from the main directory:\n\n```bash\n./mvnw -P coverage clean verify\n```\n\nThe Cloud Storage connector JAR can be found in `gcs/target/` directory.\n\n## Adding the Cloud Storage connector to your build\n\nMaven group ID is `com.google.cloud.bigdataoss` and artifact ID for Cloud\nStorage connector is `gcs-connector`.\n\nTo add a dependency on Cloud Storage connector using Maven, use the following:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.google.cloud.bigdataoss\u003c/groupId\u003e\n  \u003cartifactId\u003egcs-connector\u003c/artifactId\u003e\n  \u003cversion\u003e${next-gcs-connector-release-tag}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Resources\n\nOn **Stack Overflow**, use the tag\n[`google-cloud-dataproc`](https://stackoverflow.com/tags/google-cloud-dataproc)\nfor questions about the connectors in this repository. This tag receives\nresponses from the Stack Overflow community and Google engineers, who monitor\nthe tag and offer unofficial support.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGoogleCloudDataproc%2Fhadoop-connectors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGoogleCloudDataproc%2Fhadoop-connectors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGoogleCloudDataproc%2Fhadoop-connectors/lists"}