{"id":13513919,"url":"https://github.com/leo-project/leofs","last_synced_at":"2025-04-08T12:09:44.100Z","repository":{"id":3516188,"uuid":"4574327","full_name":"leo-project/leofs","owner":"leo-project","description":"The LeoFS Storage System","archived":false,"fork":false,"pushed_at":"2020-06-02T10:33:51.000Z","size":31456,"stargazers_count":1563,"open_issues_count":255,"forks_count":155,"subscribers_count":84,"default_branch":"v1","last_synced_at":"2025-04-01T11:02:53.268Z","etag":null,"topics":["datalake","distributed-file-system","distributed-storage","erlang","leofs","nfs","nfs-server","s3","s3-storage"],"latest_commit_sha":null,"homepage":"https://leo-project.net/leofs/","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leo-project.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-06-06T15:29:16.000Z","updated_at":"2025-03-21T00:35:47.000Z","dependencies_parsed_at":"2022-08-09T13:20:41.895Z","dependency_job_id":null,"html_url":"https://github.com/leo-project/leofs","commit_stats":null,"previous_names":[],"tags_count":101,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leo-project%2Fleofs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leo-project%2Fleofs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leo-project%2Fleofs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leo-project%2Fleofs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leo-project","download_url":"https://codeload.github.com/leo-project/leofs/tar.gz/refs/heads/v1","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247838444,"owners_count":21004580,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datalake","distributed-file-system","distributed-storage","erlang","leofs","nfs","nfs-server","s3","s3-storage"],"created_at":"2024-08-01T05:00:40.139Z","updated_at":"2025-04-08T12:09:44.082Z","avatar_url":"https://github.com/leo-project.png","language":"Erlang","funding_links":[],"categories":["Erlang"],"sub_categories":[],"readme":"# \u003ca href=\"http://leo-project.net/leofs/\" target=\"_blank\"\u003eLeoFS\u003c/a\u003e - A Storage System for a \u003ca href=\"https://en.wikipedia.org/wiki/Data_lake\" target=\"_blank\"\u003eData Lake\u003c/a\u003e and the Web\n\n[![Join the chat at https://gitter.im/leo-project/leofs](https://badges.gitter.im/leo-project/leofs.svg)](https://gitter.im/leo-project/leofs?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge) [![Release](https://img.shields.io/badge/release-v1.4.4-green.svg?style=flat)](https://github.com/leo-project/leofs/releases/tag/1.4.4) [![Build Status](https://travis-ci.org/leo-project/leofs.svg?branch=v1)](http://travis-ci.org/leo-project/leofs)\n\n![LeoFS Logo](https://leo-project.net/leofs/docs-old/_static/leofs-logo-small.png)\n\n## Overview\n\nLeoFS is an **Enterprise Open Source Storage**, and it is a highly available, distributed, eventually consistent object/blob store. If you are seeking a storage system that can store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.\n\nLeoFS is supporting the following features:\n\n* **Multi Protocol**\n  * **S3-API Support**\n      * LeoFS is an Amazon S3 compatible storage system.\n      * Switch to LeoFS to decrease your cost from more expensive public-cloud solutions.\n  * **REST-API Support**\n      * To easily access LeoFS with [REST-API](https://leo-project.net/leofs/docs/admin/protocols/rest/)\n  * **NFS Support**\n      * NFS support was provided from LeoFS v1.1, the current status of which is beta.\n* **Large Object Support**\n  * LeoFS covers handling large size objects.\n* **Multi Data Center Replication**\n  * LeoFS is a highly scalable, fault-tolerant distributed file system without SPOF.\n  * LeoFS's cluster can be viewed as a huge capacity storage. It consists of a set of loosely connected nodes.\n  * We can build a global scale storage system with easy operations\n\n## Architecture\n\n![leofs-architecture-1](https://leo-project.net/leofs/docs/assets/leofs-architecture.0012.jpg)\n\nLeoFS consists of three core components - [LeoStorage](https://leo-project.net/leofs/docs/architecture/leo_storage/), [LeoGateway](https://leo-project.net/leofs/docs/architecture/leo_gateway/) and [LeoManager](https://leo-project.net/leofs/docs/architecture/leo_manager/) which depend on Erlang.\n\n[LeoGateway](https://leo-project.net/leofs/docs/architecture/leo_gateway/) handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).\n\n[LeoStorage](https://leo-project.net/leofs/docs/architecture/leo_storage/) handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.\n\n[LeoManager](https://leo-project.net/leofs/docs/architecture/leo_manager/) always monitors LeoGateway and LeoStorage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.\n\nYou can access a LeoFS system using Amazon S3 clients and the SDK\u003c/a\u003e.\n\n\n## Slide\n\nThe presentation - \u003ca href=\"https://www.slideshare.net/rakutentech/scaling-and-high-performance-storage-system-leofs\" title=\"Scaling and High Performance Storage System: LeoFS\" target=\"_blank\"\u003eScaling and High Performance Storage System: LeoFS\u003c/a\u003e  was given at Erlang User Conference 2014 in Stockholm on June 2014\n\n## GOALs\n\n* LeoFS has been aiming to provide **high reliability**, **high scalability**, and **high cost performance ratio**:\n  * HIGH Reliability\n     * Nine nines - Operating ratios is 99.9999999%\n  * High Scalability\n     * Build huge-cluster at low cost\n  * HIGH Cost Performance\n     * Fast - Over 10Gbps\n     * A lower cost than other storage\n     * Provide easy management and easy operation\n\n## Further Reference\n\n* \u003ca target=\"_blank\" href=\"https://leo-project.net/leofs/docs/\"\u003eLeoFS Documentation\u003c/a\u003e.\n\n\n## Build LeoFS with LeoFS Packages\n\nLeoFS packages have been already provided on the Web. You're able to easily install LeoFS on your environments.\n\n* LeoProject\n    * \u003ca target=\"_blank\" href=\"https://leo-project.net/leofs/download.html\"\u003eCentOS 6.x, 7.x\u003c/a\u003e\n    * \u003ca target=\"_blank\" href=\"https://leo-project.net/leofs/download.html\"\u003eUbuntu 14.04, 16.04\u003c/a\u003e\n* Community\n    * \u003ca target=\"_blank\" href=\"http://www.freshports.org/databases/leofs\"\u003eFreeBSD\u003c/a\u003e\n\n\u003ca target=\"_blank\" href=\"https://leo-project.net/leofs/docs/installation/quick/\"\u003eHere\u003c/a\u003e is the installation manual.\n\n\n## Build LeoFS From Source (For Developers)\n\nHere, we explain how to build LeoFS from source code.\nFirst, you have to install the following packages to build Erlang and LeoFS.\n\n### Build Dependencies\n```bash\n## [CentOS]\n$ sudo yum install cmake check-devel gcc gcc-c++ make\n## [Ubuntu]\n$ sudo apt-get install gcc g++ cmake make check libtool\n### For Docker\n$ apt-get install lsb-release\n```\n\n### Install Erlang\n\nYou can install Erlang with [kerl](https://github.com/kerl/kerl).\n\n* Install [kerl](https://github.com/kerl/kerl)\n\n```bash\n$ curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl\n$ chmod a+x kerl\n$ mkdir -p ~/bin\n$ mv kerl ~/bin/\n$ echo \"export PATH=$PATH:~/bin\" \u003e\u003e ~/.bashrc\n$ source ~/.bashrc\n```\n\n* Install Erlang (Erlang/OTP 19.3)\n\n```bash\n$ kerl build 19.3 19.3\n$ kerl list builds\n19.3,19.3\n\n$ kerl install 19.3 /path/to/19.3\n$  kerl list installations\n19.3 /path/to/19.3\n\n$ source /path/to/19.3/activate\n$ kerl active\nThe current active installation is:\n/path/to/19.3\n```\n\n### Install LeoFS\n\nThen, clone source of LeoFS and libraries from GitHub.\n\n```bash\n$ git clone https://github.com/leo-project/leofs.git\n$ cd leofs\n$ git checkout -b develop remotes/origin/develop\n$ ./rebar get-deps\n$ ./git_checkout.sh develop\n```\n\nThen, build LeoFS with the following commands.\n\n```bash\n$ make \u0026\u0026 make release_for_test\n```\n\nNow, you can find the LeoFS package as follow.\n\n```bash\n$ ls package/\nleo_gateway/  leo_manager_0/  leo_manager_1/  leo_storage/  README.md\n```\n\nThen, we can start and access LeoFS with the following commands. Also, you're able to easily operate LeoFS with [leofs-adm](https://github.com/leo-project/leofs/blob/master/leofs-adm) script.\n\n```bash\n$ package/leo_manager_0/bin/leo_manager start\n$ package/leo_manager_1/bin/leo_manager start\n$ package/leo_storage/bin/leo_storage start\n$ package/leo_gateway/bin/leo_gateway start\n$ ./leofs-adm status\n [System Confiuration]\n-----------------------------------+----------\n Item                              | Value\n-----------------------------------+----------\n Basic/Consistency level\n-----------------------------------+----------\n                    system version | 1.3.4\n                        cluster Id | leofs_1\n                             DC Id | dc_1\n                    Total replicas | 1\n          number of successes of R | 1\n          number of successes of W | 1\n          number of successes of D | 1\n number of rack-awareness replicas | 0\n                         ring size | 2^128\n-----------------------------------+----------\n Multi DC replication settings\n-----------------------------------+----------\n [mdcr] max number of joinable DCs | 2\n [mdcr] total replicas per a DC    | 1\n [mdcr] number of successes of R   | 1\n [mdcr] number of successes of W   | 1\n [mdcr] number of successes of D   | 1\n-----------------------------------+----------\n Manager RING hash\n-----------------------------------+----------\n                 current ring-hash |\n                previous ring-hash |\n-----------------------------------+----------\n\n [State of Node(s)]\n-------+--------------------------+--------------+----------------+----------------+----------------------------\n type  |           node           |    state     |  current ring  |   prev ring    |          updated at\n-------+--------------------------+--------------+----------------+----------------+----------------------------\n  S    | storage_0@127.0.0.1      | attached     |                |                | 2017-06-02 14:59:20 +0900\n-------+--------------------------+--------------+----------------+----------------+----------------------------\n\n$ ./leofs-adm start\nOK\n\n$ ./leofs-adm status\n [System Confiuration]\n-----------------------------------+----------\n Item                              | Value\n-----------------------------------+----------\n Basic/Consistency level\n-----------------------------------+----------\n                    system version | 1.3.4\n                        cluster Id | leofs_1\n                             DC Id | dc_1\n                    Total replicas | 1\n          number of successes of R | 1\n          number of successes of W | 1\n          number of successes of D | 1\n number of rack-awareness replicas | 0\n                         ring size | 2^128\n-----------------------------------+----------\n Multi DC replication settings\n-----------------------------------+----------\n [mdcr] max number of joinable DCs | 2\n [mdcr] total replicas per a DC    | 1\n [mdcr] number of successes of R   | 1\n [mdcr] number of successes of W   | 1\n [mdcr] number of successes of D   | 1\n-----------------------------------+----------\n Manager RING hash\n-----------------------------------+----------\n                 current ring-hash | 433fe365\n                previous ring-hash | 433fe365\n-----------------------------------+----------\n\n [State of Node(s)]\n-------+--------------------------+--------------+----------------+----------------+----------------------------\n type  |           node           |    state     |  current ring  |   prev ring    |          updated at\n-------+--------------------------+--------------+----------------+----------------+----------------------------\n  S    | storage_0@127.0.0.1      | running      | 433fe365       | 433fe365       | 2017-06-02 15:00:10 +0900\n  G    | gateway_0@127.0.0.1      | running      | 433fe365       | 433fe365       | 2017-06-02 15:00:12 +0900\n-------+--------------------------+--------------+----------------+----------------+----------------------------\n```\n\n## Build a LeoFS Cluster\n\nYou can easily build a LeoFS cluster. See \u003ca target=\"_blank\" href=\"https://leo-project.net/leofs/docs/installation/cluster/\"\u003ehere\u003c/a\u003e.\n\n## Configure LeoFS\n\nAbout the configuration of LeoFS, See \u003ca target=\"_blank\" href=\"https://leo-project.net/leofs/docs/admin/settings/cluster/\"\u003ehere\u003c/a\u003e.\n\n## Benchmarking\n\nYou can benchmark LeoFS with \u003ca target=\"_blank\" href=\"https://github.com/basho/basho_bench\"\u003eBasho Bench\u003c/a\u003e, and \u003ca target=\"_blank\" href=\"https://leo-project.net/leofs/docs/benchmark/README/\"\u003ehere\u003c/a\u003e is a documentation to benchmark LeoFS.\n\n\n## Integration Test\n\nYou can test LeoFS with \u003ca target=\"_blank\" href=\"https://github.com/leo-project/leofs_test2\"\u003eleofs_test\u003c/a\u003e whether LeoFS has issues or not before getting installed LeoFS in your dev/staging/production environment(s).\n\n\n## Milestones\n### Version 1\n\n* *DONE* - [v1.0](https://github.com/leo-project/leofs/releases/tag/1.0.2)\n    * Multi Data Center Replication\n    * Increase compatibility S3-APIs#5\n        * Other bucket operations\n* *DONE* - [v1.1](https://github.com/leo-project/leofs/releases/tag/1.1.5)\n    * NFS v3 Support *(alpha)*\n    * Improve Web GUI Console (Option)\n* *DONE* - [v1.2](https://github.com/leo-project/leofs/releases/tag/1.2.22)\n    * NFS v3 Support *(beta)*\n    * Watchdog\n    * Automated data-compaction\n* *DONE* - v1.3\n    * NFS v3 Support *(stable)*\n    * Improve compatibility S3-APIs#6\n        * [AWS Signature v4](http://docs.aws.amazon.com/general/latest/gr/sigv4_changes.html) support (v1.3.0)\n        * Custom Metadata Support (v1.3.1)\n* *DONE* - v1.4\n    * Improvement of the core features\n    * Integration with distributed computing frameworks#1\n        * Hadoop integration\n        * Spark integration\n\n### Version 2\n* *WIP* - v2.0\n    * Erasure Code\n    * Improve Data Security for [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation) and Enterprise Storages\n    * Improve compatibility S3-APIs#7\n        * [Server Side Object Encryption](http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html)\n    * NFS v3 Support *(stable)*\n        * Improve performance of the list objects, `the ls command`\n    * Improvement of the Multi Data Center Replication\n    * Searching objects by a custom-metadata\n* v2.1\n    * Hinted Hand-off\n    * Improve compatibility S3-APIs#8\n        * Objects Expiration into a Bucket\n        * Object Versioning\n    * [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) Support\n    * Integration with distributed computing frameworks#2\n        * Hadoop integration\n        * Spark integration\n    * Improve Web GUI console, \u003ca href=\"https://github.com/leo-project/leo_center\" target=\"_blank\"\u003eLeoFS Center\u003c/a\u003e *(option)*\n* v2.2\n    * Data Deduplication\n    * Improve compatibility S3-APIs#9\n\n\n## Versioning Policy\n\nLeoFS adheres to [the versioning policy](http://semver.org/) from v1.3.3.\n\n\n## Licensing\n\nLeoFS is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.\n\n\n## Sponsors\n\n* January, 2019 - : [Lions Data, Ltd.](https://lions-data.com)\n* Februay, 2010 - December, 2018: [Rakuten, Inc.](http://global.rakuten.com/corp/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleo-project%2Fleofs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleo-project%2Fleofs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleo-project%2Fleofs/lists"}