Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/apache/opendal

Apache OpenDAL: access data freely.
https://github.com/apache/opendal

azblob gcs hacktoberfest redis rust s3 storage

Last synced: 4 days ago
JSON representation

Apache OpenDAL: access data freely.

Host: GitHub
URL: https://github.com/apache/opendal
Owner: apache
License: apache-2.0
Created: 2022-02-14T05:18:44.000Z (almost 3 years ago)
Default Branch: main
Last Pushed: 2024-10-29T13:49:51.000Z (about 1 month ago)
Last Synced: 2024-10-29T16:12:27.300Z (about 1 month ago)
Topics: azblob, gcs, hacktoberfest, redis, rust, s3, storage
Language: Rust
Homepage: https://opendal.apache.org
Size: 309 MB
Stars: 3,331
Watchers: 33
Forks: 469
Open Issues: 219
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Citation: CITATION.cff
- Codeowners: .github/CODEOWNERS

Awesome Lists containing this project

stars - apache/opendal
awesome-ccamel - apache/opendal - Apache OpenDAL: One Layer, All Storage. (Rust)
awesome-rust - OpenDAL - A unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. [![build](https://img.shields.io/github/actions/workflow/status/apache/opendal/ci_core.yml?branch=main)](https://github.com/apache/opendal/actions?query=branch%3Amain) (Libraries / Filesystem)
my-awesome - apache/opendal - 12 star:3.5k fork:0.5k Apache OpenDAL: One Layer, All Storage. (Rust)
fucking-awesome-rust - OpenDAL - A unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. [![build](https://img.shields.io/github/actions/workflow/status/apache/opendal/ci_core.yml?branch=main)](https://github.com/apache/opendal/actions?query=branch%3Amain) (Libraries / Filesystem)
fucking-awesome-rust - OpenDAL - A unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. [![build](https://img.shields.io/github/actions/workflow/status/apache/opendal/ci_core.yml?branch=main)](https://github.com/apache/opendal/actions?query=branch%3Amain) (Libraries / Filesystem)

README

        # Apache OpenDAL™: *One Layer, All Storage.*

[![](https://img.shields.io/github/discussions/apache/opendal)](https://github.com/apache/opendal/discussions)

[![](https://img.shields.io/discord/1081052318650339399?logo=discord&label=discord)](https://opendal.apache.org/discord)

OpenDAL is an Open Data Access Layer that enables seamless interaction with diverse storage services.

OpenDAL's development is guided by its vision of **One Layer, All Storage** and its core principles: **Open Community**, **Solid Foundation**, **Fast Access**, **Object Storage First**, and **Extensible Architecture**. Read the explained vision at [OpenDAL Vision](https://opendal.apache.org/docs/vision).



## For *ANY* languages

| Name              | Release                                          | Docs                                                                              |

| ----------------- | ------------------------------------------------ | --------------------------------------------------------------------------------- |

| [Rust Core]       | [![Rust Core Image]][Rust Core Link]             | [![Docs Release]][Rust Core Release Docs] [![Docs Dev]][Rust Core Dev Docs]       |

| [C Binding]       | -                                                | [![Docs Dev]][C Binding Dev Docs]                                                 |

| [Cpp Binding]     | -                                                | [![Docs Dev]][Cpp Binding Dev Docs]                                               |

| [D Binding]   | -                                                | -                                                                                 |

| [Dotnet Binding]  | -                                                | -                                                                                 |

| [Go Binding]      | [![Go Binding Image]][Go Binding Link]           | [![Docs Release]][Go Release Docs]                                         |

| [Haskell Binding] | -                                                | -                                                                                 |

| [Java Binding]    | [![Java Binding Image]][Java Binding Link]       | [![Docs Release]][Java Binding Release Docs] [![Docs Dev]][Java Binding Dev Docs] |

| [Lua Binding]     | -                                                | -                                                                                 |

| [Node.js Binding] | [![Node.js Binding Image]][Node.js Binding Link] | [![Docs Dev]][Node.js Binding Dev Docs]                                           |

| [Ocaml Binding]   | -                                                | -                                                                                 |

| [PHP Binding]     | -                                                | -                                                                                 |

| [Python Binding]  | [![Python Binding Image]][Python Binding Link]   | [![Docs Dev]][Python Binding Dev Docs]                                            |

| [Ruby Binding]    | -                                                | -                                                                                 |

| [Swift Binding]   | -                                                | -                                                                                 |

| [Zig Binding]     | -                                                | -                                                                                 |

[Docs Release]: https://img.shields.io/badge/docs-release-blue

[Docs Dev]: https://img.shields.io/badge/docs-dev-blue

[Rust Core]: core/README.md

[Rust Core Image]: https://img.shields.io/crates/v/opendal.svg

[Rust Core Link]: https://crates.io/crates/opendal

[Rust Core Release Docs]: https://docs.rs/opendal

[Rust Core Dev Docs]: https://opendal.apache.org/docs/rust/opendal/

[C Binding]: bindings/c/README.md

[C Binding Dev Docs]: https://opendal.apache.org/docs/c/

[Cpp Binding]: bindings/cpp/README.md

[Cpp Binding Dev Docs]: https://opendal.apache.org/docs/cpp/

[D Binding]: bindings/d/README.md

[Dotnet Binding]: bindings/dotnet/README.md

[Go Binding]: bindings/go/README.md

[Go Binding Image]: https://badge.fury.io/go/github.com%2Fapache%2Fopendal%2Fbindings%2Fgo.svg

[Go Binding Link]: https://pkg.go.dev/github.com/apache/opendal/bindings/go

[Go Release Docs]: https://pkg.go.dev/github.com/apache/opendal/bindings/go

[Haskell Binding]: bindings/haskell/README.md

[Java Binding]: bindings/java/README.md

[Java Binding Image]: https://img.shields.io/maven-central/v/org.apache.opendal/opendal-java

[Java Binding Link]: https://central.sonatype.com/artifact/org.apache.opendal/opendal-java

[Java Binding Release Docs]: https://javadoc.io/doc/org.apache.opendal/opendal-java

[Java Binding Dev Docs]: https://opendal.apache.org/docs/java/

[Lua Binding]: bindings/lua/README.md

[Node.js Binding]: bindings/nodejs/README.md

[Node.js Binding Image]: https://img.shields.io/npm/v/opendal

[Node.js Binding Link]: https://www.npmjs.com/package/opendal

[Node.js Binding Dev Docs]: https://opendal.apache.org/docs/nodejs/

[Ocaml Binding]: bindings/ocaml/README.md

[PHP Binding]: bindings/php/README.md

[Python Binding]: bindings/python/README.md

[Python Binding Image]: https://img.shields.io/pypi/v/opendal

[Python Binding Link]: https://pypi.org/project/opendal/

[Python Binding Dev Docs]: https://opendal.apache.org/docs/python/

[Ruby Binding]: bindings/ruby/README.md

[Swift Binding]: bindings/swift/README.md

[Zig Binding]: bindings/zig/README.md

## For *ANY* methods

| Name  | Description                                                        | Release                   |

| ----- | ------------------------------------------------------------------ | ------------------------- |

| [oay] | Access data via API Gateway                                        | [![oay image]][oay crate] |

| [oli] | Access data via Command Line (alternative to s3cmd, s3cli, azcopy) | [![oli image]][oli crate] |

| [ofs] | Access data via POSIX file system API (alternative to s3fs)        | [![ofs image]][ofs crate] |

[oay]: bin/oay/README.md

[oay image]: https://img.shields.io/crates/v/oay.svg

[oay crate]: https://crates.io/crates/oay

[oli]: bin/oli/README.md

[oli image]: https://img.shields.io/crates/v/oli.svg

[oli crate]: https://crates.io/crates/oli

[ofs]: bin/ofs/README.md

[ofs image]: https://img.shields.io/crates/v/ofs.svg

[ofs crate]: https://crates.io/crates/ofs

## For *ANY* integrations

| Name                   | Description                                                                   | Release                                     | Docs                                                                              |

| ---------------------- | ----------------------------------------------------------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------- |

| [dav-server-opendalfs] | a [dav-server-rs] implementation using opendal.                               | [![dav-server image]][dav-server crate]     | [![Docs Release]][dav-server release docs] [![Docs Dev]][dav-server dev docs]     |

| [object_store_opendal] | an [object_store] implementation using opendal.                               | [![object_store image]][object_store crate] | [![Docs Release]][object_store release docs] [![Docs Dev]][object_store dev docs] |

| [fuse3_opendal]        | Access data via integrations to [fuse3]                                       | [![fuse3 image]][fuse3 crate]               | [![Docs Release]][fuse3 release docs] [![Docs Dev]][fuse3 dev docs]               |

| [virtiofs_opendal]     | Access data via integrations to [vhost-user-backend]                          | [![virtiofs image]][virtiofs crate]         | [![Docs Release]][virtiofs release docs] [![Docs Dev]][virtiofs dev docs]         |

| [unftp-sbe-opendal]    | an [unftp] storage backend implementation using opendal.                      | [![unftp-sbe image]][unftp-sbe crate]       | [![Docs Release]][unftp-sbe release docs] [![Docs Dev]][unftp-sbe dev docs]       |

| [parquet_opendal]      | Provides [`parquet`](https://crates.io/crates/parquet) efficient IO utilities | [![parquet image]][parquet crate]           | [![Docs Release]][parquet release docs] [![Docs Dev]][parquet dev docs]           |

[dav-server-opendalfs]: integrations/dav-server/README.md

[dav-server-rs]: https://github.com/messense/dav-server-rs

[dav-server image]: https://img.shields.io/crates/v/dav-server-opendalfs.svg

[dav-server crate]: https://crates.io/crates/dav-server-opendalfs

[dav-server release docs]: https://docs.rs/dav-server-opendalfs/

[dav-server dev docs]: https://opendal.apache.org/docs/dav-server-opendalfs/dav_server_opendalfs/

[object_store_opendal]: integrations/object_store/README.md

[object_store]: https://docs.rs/object_store

[object_store image]: https://img.shields.io/crates/v/object_store_opendal.svg

[object_store crate]: https://crates.io/crates/object_store_opendal

[object_store release docs]: https://docs.rs/object_store_opendal/

[object_store dev docs]: https://opendal.apache.org/docs/object-store-opendal/object_store_opendal/

[fuse3_opendal]: integrations/fuse3/README.md

[fuse3]: https://docs.rs/fuse3

[fuse3 image]: https://img.shields.io/crates/v/fuse3_opendal.svg

[fuse3 crate]: https://crates.io/crates/fuse3_opendal

[fuse3 release docs]: https://docs.rs/fuse3_opendal/

[fuse3 dev docs]: https://opendal.apache.org/docs/fuse3-opendal/fuse3_opendal/

[virtiofs_opendal]: integrations/virtiofs/README.md

[vhost-user-backend]: https://docs.rs/vhost-user-backend

[virtiofs image]: https://img.shields.io/crates/v/virtiofs_opendal.svg

[virtiofs crate]: https://crates.io/crates/virtiofs_opendal

[virtiofs release docs]: https://docs.rs/virtiofs_opendal/

[virtiofs dev docs]: https://opendal.apache.org/docs/virtiofs-opendal/virtiofs_opendal/

[unftp-sbe-opendal]: integrations/unftp-sbe/README.md

[unftp]: https://crates.io/crates/unftp

[unftp-sbe image]: https://img.shields.io/crates/v/unftp-sbe-opendal.svg

[unftp-sbe crate]: https://crates.io/crates/unftp-sbe-opendal

[unftp-sbe release docs]: https://docs.rs/unftp-sbe-opendal/

[unftp-sbe dev docs]: https://opendal.apache.org/docs/unftp-sbe-opendal/unftp_sbe_opendal/

[parquet_opendal]: integrations/parquet/README.md

[parquet image]: https://img.shields.io/crates/v/parquet-opendal.svg

[parquet crate]: https://crates.io/crates/parquet-opendal

[parquet release docs]: https://docs.rs/parquet-opendal/

[parquet dev docs]: https://opendal.apache.org/docs/parquet-opendal/parquet_opendal/

## For *ANY* services

| Type                           | Services                                                                                                                                 |

| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |

| Standard Storage Protocols     | ftp http [sftp] [webdav]                                                                                                                 |

| Object Storage Services        | [azblob] [cos] [gcs] [obs] [oss] [s3] 
 [b2] [openstack_swift] [upyun] [vercel_blob]                                                  |

| File Storage Services          | fs [alluxio] [azdls] [azfile] [chainsafe] [compfs] 
 [dbfs] [gridfs] [hdfs] [hdfs_native] [ipfs] [webhdfs]                            |

| Consumer Cloud Storage Service | [aliyun_drive] [gdrive] [onedrive] [dropbox] [icloud] [koofr] 
 [pcloud] [seafile] [yandex_disk]                                      |

| Key-Value Storage Services     | [cacache] [cloudflare_kv] [dashmap] memory [etcd] 
 [foundationdb] [persy] [redis] [rocksdb] [sled] 
 [redb] [tikv] [atomicserver] |

| Database Storage Services      | [d1] [mongodb] [mysql] [postgresql] [sqlite] [surrealdb]                                                                                 |

| Cache Storage Services         | [ghac] [memcached] [mini_moka] [moka] [vercel_artifacts]                                                                                 |

| Git Based Storage Services     | [huggingface]                                                                                                                            |

[sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02

[webdav]: https://datatracker.ietf.org/doc/html/rfc4918

[azblob]: https://azure.microsoft.com/en-us/services/storage/blobs/

[cos]: https://www.tencentcloud.com/products/cos

[gcs]: https://cloud.google.com/storage

[obs]: https://www.huaweicloud.com/intl/en-us/product/obs.html

[oss]: https://www.aliyun.com/product/oss

[s3]: https://aws.amazon.com/s3/

[b2]: https://www.backblaze.com/

[openstack_swift]: https://docs.openstack.org/swift/latest/

[upyun]: https://www.upyun.com/

[vercel_blob]: https://vercel.com/docs/storage/vercel-blob

[alluxio]: https://docs.alluxio.io/os/user/stable/en/api/REST-API.html

[azdls]: https://azure.microsoft.com/en-us/products/storage/data-lake-storage/

[azfile]: https://learn.microsoft.com/en-us/rest/api/storageservices/file-service-rest-api

[chainsafe]: https://storage.chainsafe.io/

[compfs]: https://github.com/compio-rs/compio/

[dbfs]: https://docs.databricks.com/en/dbfs/index.html

[gridfs]: https://www.mongodb.com/docs/manual/core/gridfs/

[hdfs]: https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html

[hdfs_native]: https://github.com/Kimahriman/hdfs-native

[ipfs]: https://ipfs.tech/

[webhdfs]: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

[aliyun_drive]: https://www.aliyundrive.com/

[gdrive]: https://www.google.com/drive/

[onedrive]: https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage

[dropbox]: https://www.dropbox.com/

[icloud]: https://www.icloud.com/iclouddrive

[koofr]: https://koofr.eu/

[pcloud]: https://www.pcloud.com/

[seafile]: https://www.seafile.com/

[yandex_disk]: https://360.yandex.com/disk/

[cacache]: https://crates.io/crates/cacache

[cloudflare_kv]: https://developers.cloudflare.com/kv/

[dashmap]: https://github.com/xacrimon/dashmap

[etcd]: https://etcd.io/

[foundationdb]: https://www.foundationdb.org/

[persy]: https://crates.io/crates/persy

[redis]: https://redis.io/

[rocksdb]: http://rocksdb.org/

[sled]: https://crates.io/crates/sled

[redb]: https://crates.io/crates/redb

[tikv]: https://tikv.org/

[atomicserver]: https://github.com/atomicdata-dev/atomic-server

[d1]: https://developers.cloudflare.com/d1/

[mongodb]: https://www.mongodb.com/

[mysql]: https://www.mysql.com/

[postgresql]: https://www.postgresql.org/

[sqlite]: https://www.sqlite.org/

[surrealdb]: https://surrealdb.com/

[ghac]: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows

[memcached]: https://memcached.org/

[mini_moka]: https://github.com/moka-rs/mini-moka

[moka]: https://github.com/moka-rs/moka

[vercel_artifacts]: https://vercel.com/docs/concepts/monorepos/remote-caching

[huggingface]: https://huggingface.co/

## Examples

The examples are available at [here](./examples/).

## Documentation

The documentation is available at .

## Contribute

OpenDAL is an active open-source project. We are always open to people who want to use it or contribute to it. Here are some ways to go.

- Start with [Contributing Guide](CONTRIBUTING.md).

- Submit [Issues](https://github.com/apache/opendal/issues/new) for bug report or feature requests.

- Start [Discussions](https://github.com/apache/opendal/discussions/new?category=q-a) for questions or ideas.

- Talk to community directly at [Discord](https://opendal.apache.org/discord).

- Report security vulnerabilities to [private mailing list](mailto:[email protected])

## Who is using OpenDAL?

### Rust Core

- [apache/iceberg-rust](https://github.com/apache/iceberg-rust/): Native Rust implementation of [Apache Iceberg](https://iceberg.apache.org/), the open table format for analytic datasets.

- [CrateDB](https://github.com/crate/crate): An open-source, distributed SQL database  for Real-Time Analytics and Hybrid Search

- [Databend](https://github.com/datafuselabs/databend/): A modern Elasticity and Performance cloud data warehouse.

- [deepeth/mars](https://github.com/deepeth/mars): The powerful analysis platform to explore and visualize data from blockchain.

- [GreptimeDB](https://github.com/GreptimeTeam/greptimedb): An open-source, cloud-native, distributed time-series database.

- [mozilla/sccache](https://github.com/mozilla/sccache/): `sccache` is [`ccache`](https://github.com/ccache/ccache) with cloud storage

- [OctoBase](https://github.com/toeverything/OctoBase): the open-source database behind [AFFiNE](https://github.com/toeverything/affine), local-first, yet collaborative.

- [ParadeDB](https://github.com/paradedb/paradedb): Postgres for Search and Analytics - fast full-text search and analytics in Postgres and over cloud storage. Built as an extension.

- [Pants](https://github.com/pantsbuild/pants): A fast, scalable, user-friendly build system for codebases of all sizes.

- [QuestDB](https://github.com/questdb/questdb): An open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

- [RisingWave](https://github.com/risingwavelabs/risingwave): A Distributed SQL Database for Stream Processing

- [Vector](https://github.com/vectordotdev/vector): A high-performance observability data pipeline.

### C Binding

- [Milvus](https://github.com/milvus-io/milvus): A cloud-native vector database, storage for next generation AI applications

### Java Binding

- [QuestDB](https://github.com/questdb/questdb): An open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

## Branding

The first and most prominent mentions must use the full form: **Apache OpenDAL™** of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.

For more details, see the [Apache Product Name Usage Guide](https://www.apache.org/foundation/marks/guide).

## License and Trademarks

Licensed under the Apache License, Version 2.0: 

Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation.