{"id":20102472,"url":"https://github.com/clickhouse/libhdfs3","last_synced_at":"2025-04-05T23:09:23.873Z","repository":{"id":37748917,"uuid":"158477175","full_name":"ClickHouse/libhdfs3","owner":"ClickHouse","description":"HDFS file read access for ClickHouse","archived":false,"fork":false,"pushed_at":"2025-03-26T03:14:56.000Z","size":751,"stargazers_count":38,"open_issues_count":7,"forks_count":60,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-29T22:07:34.367Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClickHouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-21T02:06:50.000Z","updated_at":"2025-03-26T03:15:01.000Z","dependencies_parsed_at":"2023-02-19T16:31:16.809Z","dependency_job_id":"8f302d43-453a-4034-96ae-944e56aaf125","html_url":"https://github.com/ClickHouse/libhdfs3","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Flibhdfs3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Flibhdfs3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Flibhdfs3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Flibhdfs3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClickHouse","download_url":"https://codeload.github.com/ClickHouse/libhdfs3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247411235,"owners_count":20934653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T17:31:05.633Z","updated_at":"2025-04-05T23:09:23.838Z","avatar_url":"https://github.com/ClickHouse.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"libhdfs3\n========================\n[![Build Status](https://travis-ci.org/PivotalRD/libhdfs3.svg?branch=apache-rpc-9)](https://travis-ci.org/PivotalRD/libhdfs3)\n\n**A Native C/C++ HDFS Client**\n\n## Description\n\nThe Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets.\n\nHDFS is implemented in JAVA language and additionally provides a JNI based C language library *libhdfs*. To use libhdfs, users must deploy the HDFS jars on every machine. This adds operational complexity for non-Java clients that just want to integrate with HDFS.\n\n**Libhdfs3**, designed as an alternative implementation of libhdfs, is implemented based on native Hadoop RPC protocol and HDFS data transfer protocol. It gets rid of the drawbacks of JNI, and it has a lightweight, small memory footprint code base. In addition, it is easy to use and deploy.\n\nLibhdfs3 is developed by [Pivotal](http://www.pivotal.io/) and used in HAWQ, which is a massive parallel database engine in [Pivotal Hadoop Distribution](http://www.pivotal.io/big-data/pivotal-hd).\n\n========================\n## Installation\n\n### Requirement\n\nTo build libhdfs3, the following libraries are needed.\n\n    cmake (2.8+)                    http://www.cmake.org/\n    boost (tested on 1.53+)         http://www.boost.org/\n    google protobuf                 http://code.google.com/p/protobuf/\n    libxml2                         http://www.xmlsoft.org/\n    kerberos                        http://web.mit.edu/kerberos/\n    libgsasl                        http://www.gnu.org/software/gsasl/\n\nTo run code coverage test, the following tools are needed.\n\n    gcov (included in gcc distribution)\n    lcov (tested on 1.9)            http://ltp.sourceforge.net/coverage/lcov.php\n\n### Configuration\n\nAssume libhdfs3 home directory is LIBHDFS3_HOME.\n\n    cd LIBHDFS3_HOME\n    mkdir build\n    cd build\n    ../bootstrap\n\nEnvironment variable CC and CXX can be used to setup the compiler.\nScript \"bootstrap\" is basically a wrapper of cmake command, user can use cmake directly to tune the configuration. \n\nRun command \"../bootstrap --help\" for more configuration. \n\n### Build\n\nRun command to build\n    \n    make\n    \nTo build concurrently, rum make with -j option.\n\n    make -j8\n\n### Test\n\nTo do unit test, run command\n\n    make unittest\n    \nTo do function test, first start HDFS, and create the function test configure file at LIBHDFS3_HOME/test/data/function-test.xml, an example can be found at LIBHDFS3_HOME/test/data/function-test.xml.example. And run command.\n\n    make functiontest\n    \nTo show code coverage result, run command. Code coverage result can be found at BUILD_DIR/CodeCoverageReport/index.html\n\n    make ShowCoverage\n\n### Install\n\nTo install libhdfs3, run command\n\n    make install\n\n## Wiki\nhttps://github.com/PivotalRD/libhdfs3/wiki\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Flibhdfs3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclickhouse%2Flibhdfs3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Flibhdfs3/lists"}