{"id":16656585,"url":"https://github.com/sheharyarn/dbfs","last_synced_at":"2025-03-21T16:32:09.679Z","repository":{"id":70981671,"uuid":"107188703","full_name":"sheharyarn/dbfs","owner":"sheharyarn","description":"Distributed Blockchain-based File Storage 📡 ","archived":false,"fork":false,"pushed_at":"2018-08-28T04:59:38.000Z","size":1369,"stargazers_count":64,"open_issues_count":0,"forks_count":14,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-18T03:12:51.589Z","etag":null,"topics":["blockchain","cloud-native","cluster","cryptography","distributed-systems","elixir","encryption","erlang","file-storage","file-system","hacktoberfest","mnesia","security"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sheharyarn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-16T22:14:10.000Z","updated_at":"2025-02-17T01:50:37.000Z","dependencies_parsed_at":"2024-01-29T02:45:40.524Z","dependency_job_id":null,"html_url":"https://github.com/sheharyarn/dbfs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fdbfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fdbfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fdbfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sheharyarn%2Fdbfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sheharyarn","download_url":"https://codeload.github.com/sheharyarn/dbfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244829560,"owners_count":20517329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain","cloud-native","cluster","cryptography","distributed-systems","elixir","encryption","erlang","file-storage","file-system","hacktoberfest","mnesia","security"],"created_at":"2024-10-12T09:57:53.813Z","updated_at":"2025-03-21T16:32:09.669Z","avatar_url":"https://github.com/sheharyarn.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"DBFS\n====\n\n\u003e Distributed Blockchain-based File Storage in Elixir 📡\n\n`DBFS` is an experiment to implement a (very naïve) distributed file storage service using a blockchain\nin Elixir. With growing interest in Blockchains and Cryptocurrencies, I wanted to understand the complex\ninner workings of Blockchains and their feasibility in the context of business applications. I started\nby writing a [very simple Blockchain][blog-blockchain] to get started, and then wrote `DBFS` as a\nslightly more complex version that actually did something.\n\n\n\n\n## Screenshots, Talk and Paper\n\nI gave a lightning talk about the project at the [ElixirConf 2018 in Europe][elixirconf-eu], which you\ncan check out [here][talk-2018].\n\n[\u003cimg src=\"./media/elixirconf-talk.png\" height=\"300px\" /\u003e][talk-2018]\n\nI also wrote a [basic whitepaper][whitepaper] detailing the application as part of my semester project,\nand you can also see some screenshots of the application here:\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"./media/screenshot-1.png\" height=\"200px\" /\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"./media/screenshot-2.png\" height=\"200px\" /\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"./media/screenshot-3.png\" height=\"200px\" /\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n\n## Architecture\n\nThe application is divided into two parts:\n\n - [Elixir Backend][dbfs]: The core application, responsible for storing the data, performing consensus\n   among nodes and exposing an API for clients\n - [React Frontend][dbfs-web]: A Javascript based web-client that connects to the backend, provides a UI\n   to perform operations and displays statistics about all the connected nodes.\n\nThe application is designed as a \"Private Blockchain\" after weighing the pros and cons of different\napproaches, and even though it should _absolutely not be used in production_, it is meant to be run on\nprivate infrastructure instead of being made publicly available so any node can connect.\n\nThe `sys.config` file defines 3 nodes. You can change this to add or remove nodes, but you'll have to\nperform the setup on each node (if you're running it on different machines). You'll also have to make\nsure that your `hosts` file points these domains to their correct IPs.\n\n - `node_1@dbfs.newyork`\n - `node_2@dbfs.london`\n - `node_3@dbfs.singapore`\n\n\n\n\n## Setup\n\nYou need to have these dependencies installed at minimum:\n\n - **Erlang/OTP 20.1**\n - **Elixir 1.5.2**\n - **Postgres 9.4**\n - **Node.js 9.6.1** for Assets and Yarn\n - **Ruby 2.3.3** for Sass\n\nSet up the Backend:\n\n```bash\n$ git clone to.shyr.io/dbfs ~/dbfs\n$ cd ~/dbfs\n$ mix deps.get\n$ mix compile\n```\n\nSet up the Frontend:\n\n```bash\n$ git clone to.shyr.io/dbfs-web ~/dbfs-web\n$ cd ~/dbfs-web\n$ yarn install\n```\n\nFor the first node of the backend, we need to create its instance and initialize the blockchain database.\nFor other nodes, we can simply create their instances without creating the blockchain (They will\nautomatically be synchronized when we start them). Assuming, you're starting all nodes on the same machine:\n\n```bash\n$ cd ~/dbfs\n$ NODE=newyork   PRIMARY=1 mix do ecto.create, ecto.migrate, ecto.seed\n$ NODE=london    PRIMARY=0 mix do ecto.create, ecto.migrate, ecto.seed\n$ NODE=singapore PRIMARY=0 mix do ecto.create, ecto.migrate, ecto.seed\n```\n\n\n\n\n## Get it Running\n\nTo finally start the backend nodes, you need to pass them a port, node name and boot-up config. You can\nstart one node or multiple, depending on the requirements.\n\n```bash\n$ NODE=newyork PORT=3000 elixir --name node_1@dbfs.newyork --erl \"-config sys.config\" –S mix phoenix.server\n```\n\nIf you decide to run multiple nodes together, you also need to initialize the distributed Mnesia tables\nwhich are responsible for performing consensus across the network. First enter the REPL command-line of one\nof the running nodes, and enter the following:\n\n```elixir\niex\u003e DBFS.Consensus.Global.setup\n```\n\nFinally, to start the Web-Client:\n\n```bash\n$ cd ~/dbfs-web\n$ PORT=4000 yarn start\n```\n\n\n\n\n## Usage\n\nTo use the application and upload/download files, they need to be encrypted using your private key. You can\ngenerate one using `ssh-keygen`:\n\n```bash\n$ ssh-keygen -b 1024 -t rsa -f key -q -N \"\"\n```\n\n\n\n\n## What's Next?\n\nNothing. This was just an experiment and a learning experience. I'm not planning on doing any more work on\nthis, but you're free to play around and learn from my mistakes (You should absolutely not make the same\ndesign decisions I made).\n\n\n\n\n## License\n\nAll code here is available as open source under the terms of the [MIT License][license].\n\n\n\n\n  [dbfs]:             https://github.com/sheharyarn/dbfs\n  [dbfs-web]:         https://github.com/sheharyarn/dbfs-web\n  [blog-blockchain]:  https://sheharyar.me/blog/writing-blockchain-elixir/\n  [talk-2018]:        https://speakerdeck.com/sheharyar/dbfs-elixirconf-eu-2018-lightning-talk\n  [whitepaper]:       https://jmp.sh/83yHQva\n  [elixirconf-eu]:    http://elixirconf.eu/\n\n\n  [license]:          ./LICENSE\n  [img-talk]:         ./media/elixirconf-talk.png\n  [img-shot-1]:       ./media/screenshot-1.png\n  [img-shot-2]:       ./media/screenshot-2.png\n  [img-shot-3]:       ./media/screenshot-3.png\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsheharyarn%2Fdbfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsheharyarn%2Fdbfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsheharyarn%2Fdbfs/lists"}