{"id":18554335,"url":"https://github.com/oracle/oci-hdfs-connector","last_synced_at":"2025-04-13T04:59:16.946Z","repository":{"id":37561684,"uuid":"72815283","full_name":"oracle/oci-hdfs-connector","owner":"oracle","description":"HDFS Connector for Oracle Cloud Infrastructure","archived":false,"fork":false,"pushed_at":"2025-02-18T04:33:36.000Z","size":682,"stargazers_count":31,"open_issues_count":8,"forks_count":26,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-04-13T04:59:10.521Z","etag":null,"topics":["cloud","hdfs","oracle-cloud"],"latest_commit_sha":null,"homepage":"https://cloud.oracle.com/cloud-infrastructure","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oracle.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-04T04:50:39.000Z","updated_at":"2025-04-04T04:32:26.000Z","dependencies_parsed_at":"2023-01-28T01:31:34.946Z","dependency_job_id":"a9bba177-da66-4d66-9c4a-1431c23bb04a","html_url":"https://github.com/oracle/oci-hdfs-connector","commit_stats":{"total_commits":74,"total_committers":16,"mean_commits":4.625,"dds":0.6351351351351351,"last_synced_commit":"350d69d40a97b07d338cc663a14ff6da89e1680c"},"previous_names":[],"tags_count":48,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Foci-hdfs-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Foci-hdfs-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Foci-hdfs-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Foci-hdfs-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oracle","download_url":"https://codeload.github.com/oracle/oci-hdfs-connector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248665758,"owners_count":21142123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","hdfs","oracle-cloud"],"created_at":"2024-11-06T21:21:21.795Z","updated_at":"2025-04-13T04:59:16.914Z","avatar_url":"https://github.com/oracle.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Oracle Cloud Infrastructure HDFS Connector for Object Storage\n\n## About\n\noci-hdfs-connector provides the public HDFS connector that integrates with the Oracle Cloud Infrastructure Object Storage Service.\n\nThe project is open source and maintained by Oracle Corp. The home page for the project is [here](https://docs.cloud.oracle.com/Content/API/SDKDocs/hdfsconnector.htm).\n\n## Installation\n\nSee [the documentation](https://docs.cloud.oracle.com/Content/API/SDKDocs/hdfsconnector.htm) for details.\n\n## Examples\n\nExamples can be found [here](https://github.com/oracle/oci-hdfs-connector/blob/master/hdfs-example/src/main/java/com/oracle/bmc/hadoop/example/SampleOracleBmcHadoopJob.java).\n\n## Documentation\n\nFull documentation, including prerequisites and installation and configuration instructions, can be found [here](https://docs.cloud.oracle.com/Content/API/SDKDocs/hdfsconnector.htm).\n\nHDFS Connector provides support for delegation token configurator that injects a delegation token on every request. This can be done by setting the delegation token file path in an environment variable `OCI_DELEGATION_TOKEN_FILE` or by setting the Hadoop property `fs.oci.delegation.token.filepath`\n\nAPI reference can be found [here](https://docs.cloud.oracle.com/tools/hdfs/latest/).\n\n## Help\n\nSee the “Questions or Feedback?” section [here](https://docs.cloud.oracle.com/Content/API/SDKDocs/hdfsconnector.htm#questions).\n\n## Version\n\nThe first three numbers of HDFS connector version represent the Hadoop version that the connector was compiled and tested against. The next three numbers are for the platform, major and minor versions respectively.\n* Minor version is incremented when there are no breaking changes\n* Major version is incremented when there are breaking changes which might affect some users\n* Platform version is incremented when there are bigger important breaking changes which would affect many users\n\nFor example, in HDFS connector version `3.3.1.x.y.z`, `3.3.1` is for the Hadoop version being used and `x.y.z` refers to the {platform}.{major}.{minor} version of the HDFS connector.\n\n## Build\n\nBuilding HDFS connector relies on Maven artifacts that are provided by the Java SDK. To obtain the artifacts, you must [download the Java SDK](https://github.com/oracle/oci-java-sdk/) and build it locally. You can then build the HDFS connector.\n \nImportant: The Java SDK file version that you download from the [Oracle Releases page](https://github.com/oracle/oci-java-sdk/releases) must match the HDFS connector version, which you can find in the [hdfs-connector/pom.xml](/blob/8cd12e68d27e1c76b01abafccb0bcc795d0a8e04/hdfs-connector/pom.xml#L110) file in the \u003cdependency\u003e tag block that has the groupId attribute com.oracle.oci.sdk.\n\n## Changes\n\nSee [CHANGELOG](/CHANGELOG.md).\n\n## Known Issues\n\nYou can find information on any known issues with the connector [here](https://docs.cloud.oracle.com/Content/knownissues.htm) and under the “Issues” tab of this GitHub repository.\n  You can find information on any known issues with the SDK [here](https://docs.cloud.oracle.com/iaas/Content/knownissues.htm) and under the [“Issues” tab of this GitHub repository](https://github.com/oracle/oci-hdfs-connector/issues).\n\n### Potential data corruption issue for OCI HDFS Connector with `RefreshableOnNotAuthenticatedProvider`\n\n**Details**: If you are using version 3.2.1.1 or earlier of the OCI HDFS Connector and you use a `RefreshableOnNotAuthenticatedProvider` (e.g. `InstancePrincipalsCustomAuthenticator`, or generally for Resource Principals or Instance Principals) you may be affected by **silent data corruption**.\n\n**Workaround**: Update the OCI HDFS Connector to version 3.2.1.3 or later. For more information about this issue and workarounds, see [Potential data corruption issue for OCI HDFS Connector with `RefreshableOnNotAuthenticatedProvider`](https://github.com/oracle/oci-hdfs-connector/issues/35).\n\n**Direct link to this issue**: [Potential data corruption issue with OCI HDFS Connector on binary data upload with `RefreshableOnNotAuthenticatedProvider`](https://docs.cloud.oracle.com/en-us/iaas/Content/knownissues.htm#knownissues_topic_Potential_data_corruption_with_OCI_Java_SDK_on_binary_data_upload_with_RefreshableOnNotAuthenticatedProvider_HDFS)\n\n## Contributing\n\noci-hdfs-connector is an open source project. This project welcomes contributions from the community. Before submitting a pull request, please [review our contribution guide](./CONTRIBUTING.md)\n\n## Security\n\nPlease consult the [security guide](./SECURITY.md) for our responsible security vulnerability disclosure process\n\n## License\n\nCopyright (c) 2016, 2024, Oracle and/or its affiliates.  All rights reserved.\nThis software is dual-licensed to you under the Universal Permissive License (UPL) 1.0 as shown at https://oss.oracle.com/licenses/upl\nor Apache License 2.0 as shown at http://www.apache.org/licenses/LICENSE-2.0. You may choose either license.\n\nSee [LICENSE](/LICENSE.txt) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle%2Foci-hdfs-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foracle%2Foci-hdfs-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle%2Foci-hdfs-connector/lists"}