{"id":17019273,"url":"https://github.com/schnoddelbotz/cloud-vm-docker","last_synced_at":"2026-05-18T00:32:58.691Z","repository":{"id":52710745,"uuid":"254594023","full_name":"schnoddelbotz/cloud-vm-docker","owner":"schnoddelbotz","description":"*WIP* a fully serverless GCE based docker swarm alternative...? dunno yet :)","archived":false,"fork":false,"pushed_at":"2021-04-20T14:15:32.000Z","size":243,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-08T12:05:45.050Z","etag":null,"topics":["compute-engine","docker","gce","golang","google-cloud","vm"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/schnoddelbotz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-10T09:20:11.000Z","updated_at":"2024-04-10T11:09:45.000Z","dependencies_parsed_at":"2022-09-19T10:40:42.351Z","dependency_job_id":null,"html_url":"https://github.com/schnoddelbotz/cloud-vm-docker","commit_stats":null,"previous_names":["schnoddelbotz/cloud-task-zip-zap"],"tags_count":27,"template":false,"template_full_name":null,"purl":"pkg:github/schnoddelbotz/cloud-vm-docker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schnoddelbotz%2Fcloud-vm-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schnoddelbotz%2Fcloud-vm-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schnoddelbotz%2Fcloud-vm-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schnoddelbotz%2Fcloud-vm-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/schnoddelbotz","download_url":"https://codeload.github.com/schnoddelbotz/cloud-vm-docker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schnoddelbotz%2Fcloud-vm-docker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33160481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T22:39:12.733Z","status":"ssl_error","status_checked_at":"2026-05-17T22:39:10.741Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compute-engine","docker","gce","golang","google-cloud","vm"],"created_at":"2024-10-14T06:48:41.487Z","updated_at":"2026-05-18T00:32:58.670Z","avatar_url":"https://github.com/schnoddelbotz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cloud-vm-docker -- !!WIP!!\n\nRunning dockerized one-shot workloads on Google ComputeEngine has never been easier.\nAt least this might do as a valid advertisement slogan for cloud-vm-docker, given:\n\n```bash\n# run a task locally, using plain, local Docker\n$ docker run busybox echo foo\n\n# the same, but run task on Docker on a ComputeEngine VM\n$ cloud-vm-docker run busybox echo foo\n```\n\n## (intended) usage\n\nOK. That looked too simple, as it was using all the defaults from environment.\nSo, a more complex example:\n\n```bash\n# run command from custom GCR-hosted image, using a VM with 16 cores\n$ cloud-vm-docker run \\\n    -d \\\n    --vm-type n1-standard-16 \\\n    eu.gcr.io/my-project-6afd9bfb/my-compute-task-image:latest \\\n    bash -c \"echo started \u0026\u0026 sleep 3600 \u0026\u0026 echo done\"\n\n# Like `docker run -d`, the above command will not wait for task to complete\n# and will not print logs. Of course, they're accessible +/- as if it was plain Docker.\n$ cloud-vm-docker ps\nVM_ID       IMAGE                   COMMAND                                  CREATED        STATUS\nfb0f979473  busybox                 echo foo                                 5 min ago      created\n\n# Containers running on VMs will forward logs to StackDriver. To read those logs, like in Docker, do:\n$ cloud-vm-docker logs fb0f979473\n2020/04/12 10:20:05 started\n\n# Compute tasks are best run in forground (e.g. in Airflow DAGs), as this will implicitly wait\n# for container command completion.\n# But if you decided  to run a task 'detached' (-d), then you can wait for completion:\n$ cloud-vm-docker wait fb0f979473\n2020/04/12 10:30:15 started waiting for completion of task 6af7db3a\n2020/04/12 10:35:10 task 6af7db3a completed - setting wait's exit status to the task's one: EXIT_STATUS_OK\n```\n\nNote that, unless disabled by using `--no-ssh`, cloud-vm-docker will automatically \"upload\"\n(using cloud_init) your SSH public keys from `~/.ssh/*.pub`, which will authorize you for SSH VM logins.\nTo come: `cloud-vm-docker ssh \u003cmy-vm\u003e [\u003cCMD\u003e]`.\n\n## why?\n\nWhy bother with cloud-vm-docker? Because ...\n- CloudFunctions have a maximum runtime of 9 minutes and are limited to 1 or 2 cores and 2 GB of RAM.\n  You can only execute code within specific runtimes (Python, Node, Go, ...).\n- CloudRun enables running arbitrary Docker images in CloudFunctions-style, but are also limited (2 CPU/2 GB)\n- We just want to run on our Dockerized code on ephemeral VMs, without neither worrying about provisioning\n  the VMs nor setting up Docker, nor starting processes in those VMs -- cloud-vm-docker does it all!\n- To circumvent some limitations listed in https://cloud.google.com/compute/docs/containers/deploying-containers\n\n### example use cases\n\n- use cloud-vm-docker eg. in your Airflow workflows, to off-load resource hungry compute tasks to the cloud\n- use cloud-vm-docker eg. to run some jMeter benchmarks on capable cloud VMs ... against your own site\n- use cloud-vm-docker eg. data-heavy processing tasks, which benefit from cloud data \"locality\"\n- use cloud-vm-docker eg. to spin up an VM instance for further operations to be carried out via SSH\n- playing with Go, Docker and GoogleCloud APIs\n\n## how does it work(flow)?\n\n- Task submission (via curl against HTTP CFN or `cloud-vm-docker run`)\n- Alternatively, `cloud-vm-docker task-vm create ...` will bypass above http cfn and spin GCE VM + save FireStore record\n- VM is set up with a `cloudservice` systemd service, which will ...\n  - PreStart: curl-CFN to update task status in firestore to BOOTED\n  - Start: run your container!\n  - Post: curl-CFN to [update task status to EXITED] and DELETE the VM itself\n- Furthermore, VM is set up to ...\n - forward container logs to stackdriver\n - allow SSH access using your local ~/.ssh/*.pub keys\n - run your container ...\n    - have a `CVD_CFN_URL` and `MGMT_TOKEN` in environment, so status updates can be sent from within your containerized app\n    - bound to VM's host docker.sock\n    - bound to your VM's host GCR credentials \n\n## setting up google cloud for cloud-vm-docker usage\n\nfor deployment, ensure you did this once:\n```bash\ngcloud auth login\ngcloud projects list\ngcloud config set project ...\n\n# to let run cloud-vm-docker locally and interact with google services, create a svc account as in\n# https://cloud.google.com/datastore/docs/reference/libraries\n\n# best in your .bashrc\nexport GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/svc-account.json\n\n# one day, `cloud-vm-docker setup` should do, but for now ... rely on gcloud. could use docker image...\nmake gcp_deploy\n\n# the above command will deploy one cloud function:\n# A HTTP endpoint, intended for submission of VM tasks and status/management requests\n```\n\n## test-drive -- what works now?\n\nGit clone this repo, and adjust `testenv.inc.sh` to your needs. Then ...\n\n```bash\n# especially adjust command for getting auto-completion :-)\nsource testenv.inc.sh\nmake deploy_gcp\nmake clean test build\n\n# this creates a VM directly (via Google Compute API + FireStore, using local GOOGLE_APPLICATION_CREDENTIALS)\n./cloud-vm-docker task-vm create busybox sh -c 'echo hello world ; sleep 120 ; echo goodnight'\n\n# ^^ notice:\n# - ComputeEngine console UI should show the VM within a few secs\n# - If nothing goes wrong (TM), the VM should self-destruct upon completion, just leaving logs\n\n# the same, but using \"official\" way via CloudFunction / HTTP endpoint\n# NOTE: Does NOT spawn the VM atm, just logs what it will do soon...\n./cloud-vm-docker run busybox sh -c 'echo hello world ; sleep 120 ; echo goodnight'\n# the same as ^ ... but using plain curl (no need for cloud-vm-docker or docker to run container!)\ncurl -H'X-Authorization: YOUR_TOKEN' https://your-cfn-endpoint.cloudfunctions.net/CloudVMDocker/run \\\n  -H'Content-type: application/json' \\\n  -d@'{\"image\":\"busybox\", \"command\":[\"sh\", \"-c\", \"echo\", \"hello\", \"world\"]}'\n\n# list VMs as stored in FireStore\n./cloud-vm-docker ps\n\n# this should be ./cloud-vm-docker ssh ... but, for now, look up IP in console[FIXME].\n# If this fails ... there's a bug with more than 1 ssh keys in home. +1 fixme...\nssh cloud-vm-docker@IP \n\n# delete the VM -- [TODO: autocomplete!]\n./cloud-vm-docker task-vm kill ...ID_as_shown_in_ps_output...\n``` \n\n## links\n\nGoogle Cloud general\n\n- https://cloud.google.com/compute/docs/regions-zones#available\n- https://github.com/googleapis/google-cloud-go\n\nVMs\n\n- https://cloud.google.com/compute/docs/machine-types\n- https://github.com/googleapis/google-api-go-client/blob/master/examples/compute.go\n- https://cloud.google.com/compute/docs/reference/rest/v1/instances/insert\n- https://godoc.org/google.golang.org/api/compute/v1\n- https://cloudinit.readthedocs.io/en/latest/index.html\n- https://www.freedesktop.org/software/systemd/man/systemd.service.html\n\nFireStore\n\n- https://pkg.go.dev/cloud.google.com/go/firestore?tab=doc missing:\n- https://github.com/googleapis/google-cloud-go/blob/master/firestore/watch.go\n- https://github.com/GoogleCloudPlatform/golang-samples/blob/master/firestore/firestore_snippets/query.go\n\nOperations\n\n- https://github.com/googleapis/google-api-go-client/blob/master/examples/operation_progress.go\n\n## TODO\n\n- tests, tests, tests\n- FireStore: updates -- status updates via CFN (see cloud_init), or after `task-vm kill ...`\n- have some monitoring dashboard web endpoint using `status` data + google monitoring/logs links ...\n- or update some google-hosted dashboard to add/remove machines as they come/run/go(history)\n- https://cloud.google.com/compute/docs/storing-retrieving-metadata --\u003e put VM meta in FireStore / partially?\n  ```bash\n  curl -H'Metadata-Flavor:Google' \"http://metadata.google.internal/computeMetadata/v1/instance/\"curl -H'Metadata-Flavor:Google' \"http://metadata.google.internal/computeMetadata/v1/instance/\"\n  curl -H'Metadata-Flavor:Google' \"http://metadata.google.internal/computeMetadata/v1/instance/attributes/user-data\"\n  ```\n  \n- deployment: let user disable HTTP endpoint if not needed\n- list which commands work as 100% \"drop-in\" replacement for docker commands -- goal: as-much-as-possible\n- coool! can I use this for interactive containers as well? no, not yet, maybe never. you can ssh to vm though.\n- allow alternate VM disk images? custom cloud_init? custom network? labels? svcAccount (or roles to add to default)?\n- have some simple dashboard ('docker ps++') served via http cfn?\n- in theory, even with failed mgmt request, VM should be shut down via `shutdown` command, but service is run as non-root user ...\n- for `--wait`, do not poll data store! https://firebase.google.com/docs/firestore/query-data/listen\n- for `--wait`, it should pull/print logs automatically if non-zero exit code\n- so much err handling/responses\n- any non-zero Docker run exit code will result in exit(1) - it should return Docker exit code [cobra]","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschnoddelbotz%2Fcloud-vm-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fschnoddelbotz%2Fcloud-vm-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschnoddelbotz%2Fcloud-vm-docker/lists"}