{"id":13584325,"url":"https://github.com/technomada/cloud-from-scratch","last_synced_at":"2025-04-07T01:32:02.255Z","repository":{"id":40785661,"uuid":"266924553","full_name":"technomada/cloud-from-scratch","owner":"technomada","description":"Build yourself a private self-hosted full-stack personal cloud system.","archived":false,"fork":false,"pushed_at":"2021-01-18T20:47:10.000Z","size":384,"stargazers_count":254,"open_issues_count":10,"forks_count":17,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-11-06T01:39:57.568Z","etag":null,"topics":["caddy","cloud","docker","private-cloud","raspberry-pi","wireguard"],"latest_commit_sha":null,"homepage":"https://cloudfromscratch.com","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/technomada.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-26T02:21:17.000Z","updated_at":"2024-10-18T00:36:05.000Z","dependencies_parsed_at":"2022-07-18T03:30:44.570Z","dependency_job_id":null,"html_url":"https://github.com/technomada/cloud-from-scratch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technomada%2Fcloud-from-scratch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technomada%2Fcloud-from-scratch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technomada%2Fcloud-from-scratch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/technomada%2Fcloud-from-scratch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/technomada","download_url":"https://codeload.github.com/technomada/cloud-from-scratch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247578020,"owners_count":20961218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["caddy","cloud","docker","private-cloud","raspberry-pi","wireguard"],"created_at":"2024-08-01T15:04:10.324Z","updated_at":"2025-04-07T01:31:57.246Z","avatar_url":"https://github.com/technomada.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"[ Tested 2020.05.28 ] [ Updated 2020.06.03 ] [ Design Version 0.1 (Beta) ]\n\nForward\n\nThis project is very early in, it's mission is to maintain contemporary patterns and resources including specific instruction to create and operate personal cloud systems.  If you find a 'bug' or have ideas please contribute with a new issue or by making a pull request.  Thanks!  Let's make this a great way to build our own cloud.\n\n\n# Cloud From Scratch\nBuild a self-hosted personal private cloud system from scratch.\n\nYou may be surprised to find that building and maintaining your own private cloud system, using your own equipment, hosted in your own home, is not only possible, it's relatively easy and is really fun and rewarding. The following instructions should provide everything you need to build a multi-host capable personal cloud computing platform from scratch. Using primarily off the shelf technologies and inspired by the \"Arch Way\" (**Simplicity, Modernity, Pragmatism, User centrality, Versatility**) this project aims to require only a small amount of technical attention and allow a wide range of expression and functionality. After completing the following steps you'll have yourself a fully functional personal private cloud system ready to be shaped into the services you wish.  I don't know about you, but I know I'm excited, so let's get started.\n\n\n# Overview\nWhen completed the system looks something like this.\n\n```\n[ EDGE NODE VPS ]                     |               [ LAN NODE ]\n                                      |               \n    [Wireguard] \u003c---------------------+-------------- [Wireguard]\n         ^                            |                   ^\n         |                            |                   |\n         v                            |                   v\n    [Caddy Reverse Proxy]             |         +----------------------------------+\n                                      |         | DOCKER                           |\n                                      |         |\t\t\t\t   |\n\t\t\t\t      |\t\t+----------------------------------+\n\t\t\t\t      |\t\t.\t\t\t\t   .\n                                      |         .    [ ------ Caddy ---------]     .\n                                      |         .        |       |        |        . \n                                      |         .        v       |        v        .\n                                      |         .       [APP]    |      [APP]      . \n                                      |         .                v                 . \n                                      |         .              [APP]               . \n                                      |         .                                  . \n                                                +. . . . . . . . . . . . . . . . . +\n                                               \n \n         ** Internet  **                               ** Home Network Cloud **\n\n```\n\nFrom this platform you'll be able to install, own, operate and access from anywhere, from any device, any cloudable software you choose to install.\n\nChat * Photos * Calls * File Storage * Music * Notes * Weather * News * Etc\n\nYou'll have your own public domain name like https://bobspersonalwebsite.com resolving data and apps from a cloud system you built from scratch running on equipment inside your own home network. \n\nCool, let's do it!\n\n## Requirements\nFor this particular setup we'll use a...\n* **Domain Name** - Public Address\n* **Cloudflare Account** - Bandwidth Assistance\n* **VPS Account** - Privacy and NAT Mitigation (Linode/Digital Ocean/Vultr/Etc)\n* **Raspberry PI** - Local Host (v3 or greater, or adopt the instructions to a PC or Virtual Machine)\n\nNote the requirements are needed for following the directions specifically, if you're comfortable doing something different feel free to adapt the instruction to suit your situation.  For example, depending on your drivers you may or may not feel comfortable using Cloudflare.  Some advantages of Cloudflare are privacy (hiding your actual server IP (if you proxy all the addresses,)) caching (if you expect a lot of traffic,) nice API control, and it's free.  But, you trade that for unencrypted man in the middle, certainly an understandable deal-breaker for certain situations.  If you find yourself in this situation Cloudflare can be skipped and your domain mapped directly to your edge node IP using your registrars dns panel.    \n\nIf you're not already familiar with [Wireguard](https://www.wireguard.com/) and [Docker](https://www.docker.com/) you may want to first familiarize yourself with these as they play core roles in this project. These instructions assume you are comfortable with command-line based installation and configuration.  For text editing we'll use vim, feel free to replace vim with the text editor of your choice.\n\nIf this sounds like your cup of tea dear reader read on. Or if you prefer something a little more automated consider one of [these](#web-gui-automated-run-your-own-cloud-system).\n\n## Sections\n* **Provision Edge Node**\n  * VPS Instance\n  * Wireguard\n  * Docker\n  * Caddy\n\n* **Setup Domain**\n  * Register Domain (or use an existing one)\n  * Cloudflare Setup\n\n* **Build a Local Node**\n  * Configure OS\n  * Wireguard\n  * Docker\n  * Caddy\n\n* **Example Applications**\n  * Ghost\n  * GOGS\n  * Express\n\n* Troubleshooting\n* Optional Configurations\n* Discussion\n* Links\n\n\n## Provision Edge Node\nThe edge node functions as a lightweight, always online, public access gateway, mainly routes traffic, provides a layer of privacy and mitigates NAT issues.\n\nCreate a VPS instance at your favorite VPS service, like Digital Ocean or Vultr.  (use these affiliate links to support this project: [Digital Ocean](https://digitalocean.com) | [Vultr $100 free credit for 30 days](https://www.vultr.com/?ref=8580218-6G).)\n\nAny tier level with at least **512MB RAM** should be enough.\n\nCreate a new instance using **Debian 10** (Buster)\n\nFor the purpose of this tutorial we'll assume your edge node ip address is `198.51.100.1`.\n\nLog in via SSH to your new server\n```\n$ ssh root@your-new-server-ip\n```\n\nand update the system\n```\n$ apt update\n$ apt upgrade\n```\n\n### Wireguard\nWe'll use wireguard to route traffic through your NAT and provide a layer of privacy to keep your home IP address private.  You can think of this kinda like a reverse VPN.\n\n[Install and configure wireguard](edge-node/edge-node-debian-10-wireguard.md) on edge node.\n\n### Docker\nRunning services in docker keeps things tidy and manageable, we'll setup docker to contain our edge services starting with a reverse proxy web server.\n\n[Install and configure docker](edge-node/edge-node-debian-10-docker.md) on edge node.\n\n### Caddy\nCaddy is super easy to use, automatically supports Let's Encrypt https certs and will be used to route our domain requests into our home node network.\n\n[Install and configure caddy](edge-node/edge-node-debian-10-caddy.md) on edge node.\n\n\n## Setup Domain\n### Domain Name\nUse your own existing domain or register a new one.  | [namecheap](https://namecheap.com) -- support this project by using this affiliate link.\n\t\n### Cloudflare\nUse [CloudFlare](https://cloudflare.com) as your name server (set your domain name name servers to the nameserver names your cloudflare account instructs.)\n\n**DNS**\nConfigure an A record to point to the IP of your VPS.  **WITH cloudflare proxy** enabled. eg `A @ 198.51.100.1`\n(replace `example.com` with your domain name and `198.51.100.1` is an example address, use the ip address of your edge node whenever you see the `198.51.100.1` address.)\n\nConfigure cloudflare a CNAME record `edge.example.com` (replace `.example.com` with your domain name eg `edge.yourdomain.com`) point it to your domain name **WITHOUT cloudflare proxy (click to make a grey cloud)**. eg `CNAME edge example.com`\n\n**SSL/TLS**\nSet to **Full (strict)** Option  (otherwise you may get a 'too many redirects' error.)\n  \n[Configure your domain name](edge-node/edge-node-debian-10-configure-domain-name.md) on the edge node.\n\nGOOD JOB!  At this point you have a working edge node with a publicly assigned domain name ready to accept and forward web traffic to your at home local network.  Now might be a good moment to take a break and go for a little walk.  Next up... local node!\n\n\n## Local Node\nLocal nodes live within your home network.  In this system local nodes are pretty much where everything lives and happens.  Cloud systems can be built from one or more hosts, but to keep things simple we'll start out with just one node, a Raspberry Pi.\n\n### Get Pi\n\nPi [Kit $100](https://www.amazon.com/CanaKit-Raspberry-4GB-Starter-Kit/dp/B07V5JTMV9) This is a kit, feel free to get any Pi setup v3 or better.\n\n### Install Raspbian\nNote: if you don't have a Pi, you could use a virtual machine, or laptop if so note you may need to adapt these instructions to your situation.\n\n[Install and configure raspbian](local-node/local-node-raspbian-setup.md) on your local node.\n\n### Wireguard\nHere we'll setup the local side wireguard.  When the local node boots and gets an internet connection it will automatically connect to your edge node and become virtually accessible to external requests.\n\n[Install and configure wireguard](local-node/local-node-raspbian-wireguard.md) on your local node.\n\n### Docker\nLocal node services are setup within docker containers.  Web requests are routed through caddy (running in a container) to other containerized processes but are all mapped into urls within your domain name. \n\n[Install and configure docker](local-node/local-node-raspbian-docker.md) on your local node.\n\n### Caddy \nWhen a web request arrives at your edge node, it hits your edge node caddy instance, which routes the request through wireguard to your local caddy which then routes it to the docker container running the service at that url.\n\n[Install and configure caddy](local-node/local-node-raspbian-caddy.md) on your local node.\n\n\nSUPER!! Everything is connected!  You now have a publicly addressable domain name which serving content from a device within your local network.  Cool.  Next let get that node doing things.\n\n## Applications\nA cloud needs to do things, you'll be able to build your cloud into whatever suits you. Here you'll find a few examples to get you started, we'll start with a simple caddy static file server then move on to dynamic container based applications.\n\n### Caddy static file serving\n\n(note from here out you can assume we're dealing with the **LOCAL NODE**)\n\nEdit Caddyfile\n```\n$ vim ~/Caddyfile\n```\n\nComment out the `respond` line and add `file_server` section.\n```\nhttp://example.com {\n\t\n\tfile_server {\n\t\troot /usr/share/caddy/example.com\n\t\t}\n\t#respond \"Yay!  It really works!\"\t\n\t}\n```\n\nEdit ~/www/example.com/index.html\n```\n$ vim ~/www/example.com/index.html\n```\n\nSay something witty.\n```\nStatic works too!!\n```\n\n\nRestart Caddy\n```\n$ sudo docker restart caddy_web_server\n```\n\n\nTest\n```\n$ curl -v https://example.com\n```\nTry testing from your browser on your phone or desktop.\n\nBrowse to [https://example.com]\n\nYou should see your wit!!\n\nHere you might notice too that if you enter http://example.com you'll be forwarded to the secure link https://example.com.\n\n\nFine work.  Now for something more appy.\n\n\n### Dynamic (sky's the limit) applications.\n\nHere's few apps setups to try, click one of these or jump to the [more](apps/README.md) page for expanded lists.\n\n* [Ghost](apps/ghost.md)\n* [GOGS](apps/gogs.md)\n* [Express](apps/express.md)\n\nThere's plenty [more](apps/README.md) one can install, and more becomes available all the time.\n\nThere you have it.  Your own cloud.  Let us know what you do with yours!\n\n## Troubleshooting\nSometimes things change or there might be a misstyped command.  Here are some hints and tips to get the wrong back in line.\n\n## Doing More\n* [Using subdomains](more/subdomains.md)\n\n## Security Adjustments\n\n## Discussion\n\nWhy run a personal cloud? \n\n**It's better to own your platform.**\n\n* Ad Free (Calmer/Cleaner/Faster)\n* Control\n* Convenience\n* Persistence\n* Personalization\n* Privacy\n* Options\n* Speed\n* \"Unlimited\" Storage\n\n## Where to go next\n* add TC (traffic control)\n* multiple nodes\n* email hosting\n* https access lan side\n* advanced management\n\t* automated application distribution\n\t* system health monitoring\n\t* network boot and automated dynamic provisioning\n* remote service sharing and integration\n* firewall\n* cloud suite\n\n\n### Related Links\n* https://github.com/cncf/landscape#trail-map\n* https://www.reddit.com/r/selfhosted/\n* https://redecentralize.org/\n* https://www.cncf.io/projects/\n\n### Similar Projects\n* https://github.com/ahmadsayed/cloud-from-scratch\n* https://github.com/awesome-selfhosted/awesome-selfhosted#self-hosting-solutions\n* https://github.com/funkypenguin/geek-cookbook\n* https://github.com/progmaticltd/homebox\n* https://github.com/sovereign/sovereign\n* https://freedombox.org/\n\n### Additional Reading\n* https://medium.com/better-programming/running-a-container-with-a-non-root-user-e35830d1f42a\n* https://neustadt.fr/essays/against-a-user-hostile-web/\n* https://www.howtoforge.com/tutorial/build-your-own-cloud-on-debian-wheezy/\n* https://www.inkandswitch.com/local-first.html\n* https://www.rechberger.io/tutorial-how-to-build-your-own-server-infrastructure-using-ansible/\n\n\n### Web GUI automated run your own cloud systems\nPrefer turnkey web gui based management?  Take a look at these.\n* https://caprover.com/\n* https://cloudron.io/\n* https://cozy.io/\n* https://nextcloud.com/\n* https://openmediavault.org/\n* https://owncloud.org/\n* https://sandstorm.io/\n* https://yunohost.org/\n\n### Useful tools and services\n* https://argoproj.github.io/projects/argo-cd\n* https://www.ansible.com/overview/how-ansible-works\n* https://www.backblaze.com/\n* https://github.com/sshuttle/sshuttle\n* https://grafana.com/\n* https://github.com/longhorn/longhorn\n* https://landscape.cncf.io/\n* https://localhost.run/\n* https://microk8s.io/\n* http://play-with-docker.com/\n* https://ngrok.com/\n* https://openebs.io/\n* https://www.openfaas.com/\n* https://www.portainer.io/\n* https://prometheus.io/\n* https://traefik.io/\n* https://www.zerotier.com/\n\n\n### Related tech to consider\n* https://docs.inlets.dev/\n* https://github.com/lithdew/flatend\n\n### About Cloud From Scratch technology\nThe cloud from scratch project favors using established technologies over building new tech.\n* Easier to adopt\n* Familiarity\n* If something exists that works, don't build something new\n* Longevity\n* Leverage billions of dollars worth of existing energy\n* Well understood\n* Tested\n\n\n**Learn more about the tech**\n* https://github.com/pirate/wireguard-docs\n\n**Containerization and digging deeper**\n* https://ericchiang.github.io/post/containers-from-scratch/\n* https://medium.com/swlh/build-containers-from-scratch-in-go-part-1-namespaces-c07d2291038b\n\n### How to support this project\n* contribute feedback/ideas/recommendations/discoveries\n* report issues (this project contains moving targets)\n* create online videos\n* share on social media\n* sponsor\n\nHave a suggestion, question, comment, or request?  Would like to see instructions for a particular topic? Submit a [NEW ISSUE](https://github.com/technomada/cloud-from-scratch/issues/new).\n\nContact: cfs2006@textyio.com\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnomada%2Fcloud-from-scratch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechnomada%2Fcloud-from-scratch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnomada%2Fcloud-from-scratch/lists"}