{"id":25876116,"url":"https://github.com/andrewarrow/devops","last_synced_at":"2025-07-07T00:06:31.143Z","repository":{"id":179136903,"uuid":"663019055","full_name":"andrewarrow/devops","owner":"andrewarrow","description":"This code will setup a new FREE VM on google cloud with postgres, a load balancer, and a web app that can query from the postgres running on localhost.","archived":false,"fork":false,"pushed_at":"2024-07-06T15:25:56.000Z","size":3171,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-05T01:41:19.843Z","etag":null,"topics":["automation","free","google-cloud","hosting","https","letsencrypt","load-balancer","postgres","ssh-keys","vm"],"latest_commit_sha":null,"homepage":"https://many.pw","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andrewarrow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-06T11:31:38.000Z","updated_at":"2024-08-22T15:24:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"9f2a781d-880f-48d1-bfd4-3f4ea4013eed","html_url":"https://github.com/andrewarrow/devops","commit_stats":{"total_commits":48,"total_committers":1,"mean_commits":48.0,"dds":0.0,"last_synced_commit":"db0d5d36656c38742cfef2734fdd8b02e7c28796"},"previous_names":["andrewarrow/devops"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/andrewarrow/devops","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewarrow%2Fdevops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewarrow%2Fdevops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewarrow%2Fdevops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewarrow%2Fdevops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andrewarrow","download_url":"https://codeload.github.com/andrewarrow/devops/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewarrow%2Fdevops/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263991444,"owners_count":23540665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","free","google-cloud","hosting","https","letsencrypt","load-balancer","postgres","ssh-keys","vm"],"created_at":"2025-03-02T10:20:53.909Z","updated_at":"2025-07-07T00:06:31.122Z","avatar_url":"https://github.com/andrewarrow.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"CREATE USER fred WITH SUPERUSER PASSWORD 'fred'\npsql --user=postgres -c \"CREATE database feedback\"\nCREATE EXTENSION IF NOT EXISTS citext\n\n\n\n# devops\nThis code will setup a new VM with postgres, a load balancer, and a web app that can query from the postgres running on localhost.\n\nI use it with [google cloud](https://cloud.google.com/) and select the `e2-micro` VM so it's in the free tier.\n\nYou have to put it in `us-west1` (Oregon) or `us-central1` (Iowa).\n\nYour boot disk has to be `standard persistent disk` up to 30GB.\n\nSo you get a 30GB hard drive, 1GB ram, and two `AMD EPYC 7B12 2250 MHz processors`. \nThat's enough to run a nice little site with plenty of traffic. \nI've had links on the front page of hacker news and it never went down, didn't\neven max out the cpu.\n\nUse Ubuntu 22.04 LTS x86/64, amd64 jammy image\n\n# balancer\nThe load balancer serves `one` main purpose: you can deploy a new version of the web app\nwith zero down time.\n\nI don't use google's real load balancers or their real postgres because, free!\n\nWithout the balancer if you deployed the web app, for a second or two, the\nreverse proxy would give the user a 500 error. That might not seem that bad but for\na production site, I want to be able to deploy many times a day and not affect\nusers ever.\n\nThe other purpose is to run on port 443 and handle SSL and the certs from\n[letsencrypt](https://letsencrypt.org/). It also runs on port 80 and just \nforwards any request on 80 to 443. i.e. you can't make an http request.\nEverything forwards to https.\n\nSo it runs two golang [httputil.ReverseProxy](https://pkg.go.dev/net/http/httputil#NewSingleHostReverseProxy) things. One on port 3000 and another on 3001.\n\nThe balancer sends 100% of traffic to either 3000 or 3001. It never splits up\nthe traffic 50% to each because that's not the point. The point is to be able to\ndo this:\n\n```\nscp web aa@YOUR-IP:\nssh aa@YOUR-IP 'bash -s' \u003c script-3001.sh\n```\n\nand script-3001.sh is:\n\n```\nsystemctl stop web-3001.service\nmv /home/aa/web /home/aa/web-3001\nsystemctl start web-3001.service\n```\n\nIt's safe to run `systemctl stop web-3001.service` because 0% of traffic is\ngoing it to. 100% is going to 3000. That's the default and how it starts.\n\nThe logic to do a deploy hits your site at a special url. The url has a guid\nso no one will be able to guess this url and use it but you.\n\nLike this:\n\n```\nhttps://many.pw/f0e3267a-376c-4a21-8f53-f4b5192357c6/3000\n```\n\nNote `f0e3267a-376c-4a21-8f53-f4b5192357c6` is not my real guid! Keep your\nguid secret. Anyway that route of /guid/3000 makes the balancer change to\nthe other one it's not using. If it's using 3000 it goes to 3001. If it's\non 3001 that command makes it go back to 3000.\n\nThis is done with:\n\n```\nif WebPort == 3000 {\n  WebPort++\n} else {\n  WebPort--\n}\nReverseProxyWeb = makeReverseProxy(WebPort, false)\n```\n\nSo there is a little script to query the current WebPort value and know\nwhich one is safe to run `systemctl stop web-%s.service` on where %s gets\nfilled in as either 3000 or 3001\n\nThere are two systemd service files:\n\n```\nExecStart=/home/aa/web-3000 run 3000\n```\n```\nExecStart=/home/aa/web-3001 run 3001\n```\n\nNotice they use a different binary. This allows us to scp a file called `web`\nto /home/aa/ and then call `systemctl stop` on the right service. THEN\nyou can `mv web web-3000` or `mv web web-3001` because if the service is\nrunning you CANNOT replace the binary.\n\nYou build on your local machine:\n\n```\nGOOS=linux GOARCH=amd64 go build\n```\n\nAnd boom, upload new version, stop the right service, rename the file,\nstart back up the new service, and then hit that special URL and all\nof a sudden users get the new version!\n\n# key gen\nYou need to run:\n\n```\n# save as ~/.ssh/aa\nssh-keygen -t ed25519 -C aa@devops\n# save as ~/.ssh/root\nssh-keygen -t ed25519 -C root@devops\n```\n\nYou could change `aa` to whatever username you like, but I say might as\nwell just use aa it's a nice username.\n\nThe public version `aa.pub` and `root.pub` need to be added to your google\nVM's list of SSH keys.\n\n# env vars\n\n```\nexport BALANCER_GUID=?\nexport VM_IP=YOUR-IP\n```\n\nYou can get your IP from the google VM (it's free!). Isn't that amazing. A free IP in\ntoday's world.\n\nYour BALANCER_GUID value you will set after running a command.\n\n# How to run\n\nFirst make sure you run `./build.sh` in the `balancer` dir and then the `web` \ndirectory so your binary files are ready. Then run each of commands below one by one.\n\nThe `./vm env youremail yourdomains` one is very special. Your email is the email\nyou want to use for the letsencrypt cert. And yourdomains is a comma separated list\nof domains. For example:\n\n```\n./vm env fred@gmail.com good.com,great.co,other-domain.org\n```\n\nAs long as your have an A record pointing to the IP of your google VM in\neach domain,\nletsencrypt will be able to make a cert.\n\nSo I like to run this list one by one but you could also place these in\na file and run it all at once!\n\nIt also picks a random guid for you and outputs:\n\n```\nfmt.Println(\"export BALANCER_GUID=\" + guid)\n```\n\nSo that's how you get that value.\n\n```\n./vm psql\n./vm cp ../aa.conf /etc/systemd/system/ root\n./vm env youremail yourdomains\n./vm cp ../balancer/balancer.service /etc/systemd/system/ root\n./vm cp ../balancer/balancer /home/aa/ aa\n./vm reload balancer\n./vm cp ../web/web /home/aa/web-3000 aa\n./vm cp ../web/web /home/aa/web-3001 aa\n./vm web\n./vm reload web-3000\n./vm reload web-3001\n```\n\n# deploy\n\n```\n./vm deploy-web yourdomain\n```\n\nThat's all you need to run to hit that special url after it uploads the new binary.\n\nAnd you should not need to do this often because it will require downtime but\nif you want to make logic changes to the balancer you can deploy it with:\n\n```\n./vm deploy-balancer\n```\n\n```\n\nfunc Serve() {\n\tdomainList := os.Getenv(\"BALANCER_DOMAINS\")\n\tReverseProxyBackend = makeReverseProxy(BackendPort, false)\n\tReverseProxyWeb = makeReverseProxy(WebPort, false)\n\n\tcfg := simplecert.Default\n\tcfg.Domains = strings.Split(domainList, \",\")\n\tcfg.CacheDir = \"/certs\"\n\tcfg.SSLEmail = os.Getenv(\"BALANCER_EMAIL\")\n\tcertReloader, err := simplecert.Init(cfg, nil)\n\tfmt.Println(\"err\", err)\n\n\tgo http.ListenAndServe(\":80\", http.HandlerFunc(simplecert.Redirect))\n\tgo http.ListenAndServe(\":8082\", http.HandlerFunc(handleLocal))\n\n\ttlsconf := tlsconfig.NewServerTLSConfig(tlsconfig.TLSModeServerStrict)\n\ttlsconf.GetCertificate = certReloader.GetCertificateFunc()\n\n\thandler := http.HandlerFunc(handleRequest)\n\n\ts := \u0026http.Server{\n\t\tAddr:      \":443\",\n\t\tHandler:   handler,\n\t\tTLSConfig: tlsconf,\n\t}\n\n\ts.ListenAndServeTLS(\"\", \"\")\n\n\tfor {\n\t\ttime.Sleep(time.Second)\n\t}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewarrow%2Fdevops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrewarrow%2Fdevops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewarrow%2Fdevops/lists"}