https://github.com/Xuanwo/hdrs

HDFS Native Client in Rust via HDFS C API libhdfs
https://github.com/Xuanwo/hdrs

Last synced: about 2 months ago
JSON representation

HDFS Native Client in Rust via HDFS C API libhdfs

• Host: GitHub
• URL: https://github.com/Xuanwo/hdrs
• Owner: Xuanwo
• Created: 2022-04-25T13:32:16.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-03-21T06:42:45.000Z (6 months ago)
• Last Synced: 2024-07-28T17:42:16.074Z (about 2 months ago)
• Language: Rust
• Homepage:
• Size: 124 KB
• Stars: 33
• Watchers: 2
• Forks: 7
• Open Issues: 6
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • Funding: .github/FUNDING.yml

Awesome Lists containing this project

README

        
# hdrs   [![Build Status]][actions] [![Latest Version]][crates.io]

[Build Status]: https://img.shields.io/github/actions/workflow/status/Xuanwo/hdrs/ci.yml?branch=main

[actions]: https://github.com/Xuanwo/hdrs/actions?query=branch%3Amain

[Latest Version]: https://img.shields.io/crates/v/hdrs.svg

[crates.io]: https://crates.io/crates/hdrs

HDFS Native Client in Rust based on [hdfs-sys](https://github.com/Xuanwo/hdfs-sys).

## Quick Start

```rust

use std::io::{Read, Write};

use hdrs::Client;

fn main() -> Result<(), Box> {

    let fs = Client::connect("hdfs://127.0.0.1:9000")?;

    let mut f = fs.open_file().write(true).create(true).open("/tmp/hello.txt")?;

    let n = f.write("Hello, World!".as_bytes())?;

    let mut f = fs.open_file().read(true).open("/tmp/hello.txt")?;

    let mut buf = vec![0; 1024];

    let n = f.read(&mut buf)?;

    let _ = fs.remove_file("/tmp/hello.txt")?;

    Ok(())

}

```

## Compiletime

`hdrs` depends on [hdfs-sys](https://github.com/Xuanwo/hdfs-sys) which links `libjvm` to work.

Please make sure `JAVA_HOME` is set correctly:

```shell

export JAVA_HOME=/path/to/java

export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}

```

- Enable `vendored` feature to compile `libhdfs` and link in static.

- Specify `HDFS_LIB_DIR` or `HADOOP_HOME` to load from specified path instead of compile.

- Specify `HDFS_STATIC=1` to link `libhdfs` in static.

- And finally, we will fallback to compile `libhdfs` and link in static.

## Runtime

`hdrs` depends on [hdfs-sys](https://github.com/Xuanwo/hdfs-sys) which uses JNI to call functions provided by jars that provided by hadoop releases.

Please also make sure `HADOOP_HOME`, `LD_LIBRARY_PATH`, `CLASSPATH` is set correctly during runtime:

```shell

export HADOOP_HOME=/path/to/hadoop

export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}

export CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath --glob)

```

If `libhdfs` is configued to link dynamiclly, please also add `${HADOOP_HOME}/lib/native` in `LD_LIBRARY_PATH` to make sure linker can find `libhdfs.so`:

```shell

export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${HADOOP_HOME}/lib/native:${LD_LIBRARY_PATH}

```

## Version Requirement

`hdrs` requires at least hadoop 2.3 to work: hadoop 2.2 doesn't handle FileNotFound correctly.

`hdrs` requires at least hadoop 2.6 to work: Older version of hadoop doesn't handle errno correctly. In older versions, hadoop will set errno to `3` if input path is an empty dir.

## Contributing

Check out the [CONTRIBUTING.md](./CONTRIBUTING.md) guide for more details on getting started with contributing to this project.

## Getting help

Submit [issues](https://github.com/Xuanwo/hdrs/issues/new/choose) for bug report or asking questions in [discussion](https://github.com/Xuanwo/hdrs/discussions/new?category=q-a).

## Acknowledgment

This project is highly inspired by [clang-sys](https://github.com/KyleMayes/clang-sys)

#### License



Licensed under Apache License, Version 2.0.