{"id":16573713,"url":"https://github.com/dkarter/neptune","last_synced_at":"2025-09-10T05:43:56.661Z","repository":{"id":36965656,"uuid":"494330660","full_name":"dkarter/neptune","owner":"dkarter","description":"☔ Elixir Cluster on K8s","archived":false,"fork":false,"pushed_at":"2024-01-09T04:13:20.000Z","size":200,"stargazers_count":3,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-02T23:31:17.934Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://k8s.console.lol/","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dkarter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-20T05:20:22.000Z","updated_at":"2024-07-03T01:21:56.000Z","dependencies_parsed_at":"2023-01-17T09:03:06.213Z","dependency_job_id":null,"html_url":"https://github.com/dkarter/neptune","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkarter%2Fneptune","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkarter%2Fneptune/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkarter%2Fneptune/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkarter%2Fneptune/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dkarter","download_url":"https://codeload.github.com/dkarter/neptune/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247256805,"owners_count":20909361,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T21:42:53.951Z","updated_at":"2025-04-04T22:23:10.302Z","avatar_url":"https://github.com/dkarter.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg\n    src=\"./assets/images/logo.svg\"\n    alt=\"Neptune Logo\"\n    width=\"220\"\n  /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eNeptune\u003c/b\u003e\n  \u003cbr /\u003e\n  Example Elixir Cluster on K8s\n\u003c/p\u003e\n\n![screenshot](./screenshot.png)\n\n## Setup Instructions\n\n## Set up the app for running it locally\n\n```bash\nmix setup\n```\n\n## Run locally\n\n```bash\nmix phx.server\n```\n\nYou should now be able to visit the server in your browser on http://localhost:4000\n\n## Run **Production** Release Locally\n\n### Set environment variables\nMake sure you have Direnv installed for this to work.\n\n```bash\ncp .envrc.example .envrc\n```\n\nEdit the `SECRET_KEY_BASE` to a stable value, which you can generate using `mix phx.gen.secret`. Here's a handy script\n\n```bash\nsed 's@SECRET_KEY_BASE=.*@SECRET_KEY_BASE='\"$(mix phx.gen.secret)\"'@' .envrc\n```\n\nCheck that it changed, and then enable it with:\n\n```bash\ndirenv allow\n```\n\n### Build the Production Release\n\n```bash\nMIX_ENV=prod mix release --overwrite\n```\n\n### Build Docker Image\n\n```bash\ndocker build -t neptune .\n```\n\n### Run in Docker Container\n\nThis command will pass env vars from the terminal session to the container and will run it in daemon mode\n\n```bash\ndocker run --rm -d -p $PORT:$PORT -e PORT -e SECRET_KEY_BASE -e DATABASE_URL --name neptune neptune\n```\n\nYou should now be able to visit the server in your browser on http://localhost:4000\n\n### Stop Docker Container\n\n```bash\ndocker stop neptune\n```\n\n## Run more nodes\n\nMake sure the `.hosts.erlang` file has the right hostname for your machine.\n\nGet the hostname this way:\n\n```bash\nhostname\n```\n\nRun in two different terminals:\n\n```bash\nMIX_ENV=prod PORT=4000 iex --sname node4000 -S mix phx.server\n```\n\n```bash\nMIX_ENV=prod PORT=4001 iex --sname node4001 -S mix phx.server\n```\n\nThe nodes should automatically connect via libcluster :sparkles:\n\n# Kubernetes\nThis app can be run locally using Minikube\n\n## Setup\nFirst set up the cluster:\n\n```bash\nminikube start -p neptune-mk -n 2\n```\n\n(this will create a 2 node cluster)\n\n## Secrets\nYou will need to create a secrets file\n\n```bash\nkubectl create secret generic neptune-secret\n```\n\nAnd then edit it to include a few secrets:\n\n```bash\nkubectl edit secrets neptune-secret\n```\n\nIt should look something like this:\n\n```yaml\napiVersion: v1\ndata:\n  database_password: \u003cbase64 encoded password\u003e\n  database_url: \u003cbase64 encoded database url e.g. ecto://postgres:mysecretpassword@postgres-service/neptune_prod\u003e\n  secret_key_base: \u003cbase64 encoded generated secret e.g. from mix phx.gen.secret\u003e\nkind: Secret\nmetadata:\n  creationTimestamp: \"2022-06-17T04:08:33Z\"\n  name: neptune-secret\n  namespace: default\n  resourceVersion: \"27981\"\n  uid: b9d65217-4fa0-42a7-b230-b611de14ad10\ntype: Opaque\n```\n\n## Applying configurations\n\n```bash\nkubectl apply -f k8s\n```\n\nThis should create all necessary pods and services\n\n## Accessing the tunnel\nTo access the cluster you can use\n\n```bash\nminikube tunnel\n```\n\nThen to find the IP address of the cluster:\n\n```bash\nkubectl get pods -o wide\n```\n\nAnd look for the external IP of the neptune-service, which you can plug into your browser e.g. `http://10.106.243.99:4000`\n\n## Connecting to Postgres DB\n\nFirst list the pods to find the Cluster IP of the postgres service\n\n```bash\nkubectl get pods\n```\n\n```\nNAME               TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE\nkubernetes         ClusterIP      10.96.0.1        \u003cnone\u003e        443/TCP          27d\nneptune-headless   ClusterIP      None             \u003cnone\u003e        \u003cnone\u003e           6s\nneptune-service    LoadBalancer   10.99.44.84      10.99.44.84   4000:31793/TCP   6s\npostgres-service   ClusterIP      10.110.129.226   \u003cnone\u003e        5432/TCP         5s\n```\n\nThen use that IP to connect to the database via psql\n\n```bash\npsql -U postgres -h 10.110.129.226\n```\n\n\n# Deployment\n\n## Semi-Auto\n\nThis project uses Pulumi to deploy the K8s cluster to DigitalOcean.\n\nFirst install Pulumi on your machine, then cd into the `infra` directory and set all the required secrets for the stack (dev):\n\n```bash\npulumi config set --secret digitalocean:token \u003cDO_API_TOKEN\u003e\npulumi config set domain example.com # make sure you have the nameservers pointed at digitalocean since this is how we manage the DNS\npulumi config set subdomain k8s # can be anything\npulumi config set --secret database_password \u003cpass\u003e\npulumi config set --secret database_url 'ecto://\u003cuser\u003e:\u003cpass\u003e@postgres-service/neptune_prod'\npulumi config set --secret secret_key_base '\u003cgenerate via mix phx.gen.secret\u003e'\n```\n\nThen run\n\n```bash\npulumi up\n```\n\n## Full Auto (Continuous Deployment + Preview Envs)\n\n### Preview Envs\n\nEvery PR gets its own preview environment on an entirely separate cluster, which\nis deployed to dedicated subdomain. This allows testing changes in isolation and\nsignificantly streamlines the steps a developer has to make to get from code change to UAT and subsequently to production.\n\nThe subdomain is generated based on the branch name using\n[codenamize.js](https://github.com/stemail23/codenamize-js). This allows using\nany branch name including names that are not subdomain friendly such as\n`feature/do-a-thing`.\n\nThe generated subdomain will look something like this\n`http://vengeful-professor.console.lol`\n\n### Production Deploy\nComing soon...!\n\n# TODO\nThis project achieved its main goal of demonstrating how to create Preview Envs\nfor PRs such as the ones popularized by Heroku, Render, Netlify, Vercel etc... But do it with Kubernetes as part of the CI/CD workflow.\n\nI intend to continue to iterate on this and make it a template for future\nprojects requiring hand rolled infrastructure (e.g. massive scale). Also this\nserves as a great learning opportunity for Elixir clustering and work\ndistribution/state sharing in a cluster.\n\nThere are a few more optimizations I am still working on\n## Must Have\n- [ ] Production deploy to canonical domain console.lol\n  - Currently production deploys are not automated via CD (but can be deployed\n    from the commandline using Pulumi)\n- [ ] automatic SSL cert for the load balancer (via Digital Ocean annotations on the k8s config)\n  - https://docs.digitalocean.com/products/kubernetes/how-to/configure-load-balancers/\n\n## Optimizations (Nice to have)\n- [ ] Single staging cluster with namespaces for preview envs\n  - Potentially save money (since each 2 node cluster is ~$24, not including\n    load balancer and volumes)\n  - Potentially speed up initial deploy time\n  - Could be a rabbit hole though\n- [ ] Use labels to trigger build on PRs (e.g. Deploy label)\n  - [ ] label removal should destroy the environment\n- [ ] Change the destroy env workflow to be triggered only by closed PRs (with\n      the labels)\n  - To avoid workflow failures when non PR branches are closed\n- [ ] separate secrets for production and staging (perhaps use a stack file as a\n      template so there's less to define and use the \"copy from\" flag when running\n      `pulumi stack create`)\n- [ ] break Pulumi file into smaller files\n- [ ] attempt to re-use k8s yaml folder\n  - [ ] will require passing the docker image tag as a variable\n- [ ] automatically delete volumes for preview environments\n  - the volumes are created by the stateful set automatically, but are not\n    destroyed when the environment is spun down. Maybe there's a way to create\n    the volume via Pulumi and attach it to the cluster — this way it will get\n    deleted on destroy\n\n## Learning opportunities\n- [ ] Cluster Dynamic Supervisor with Horde\n- [ ] Distributed Task Supervisor for spreading work across the cluster\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkarter%2Fneptune","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdkarter%2Fneptune","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkarter%2Fneptune/lists"}