{"id":18941484,"url":"https://github.com/mchmarny/long-running-job","last_synced_at":"2025-07-31T00:42:56.151Z","repository":{"id":77051646,"uuid":"203426029","full_name":"mchmarny/long-running-job","owner":"mchmarny","description":"Demo showcasing the use container execution feature in GCE to effectively run long running jobs","archived":false,"fork":false,"pushed_at":"2020-02-08T20:28:46.000Z","size":2882,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-12T12:36:50.616Z","etag":null,"topics":["api","container","gce","gcp","gcs","golang","metadata","vm"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mchmarny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-20T17:47:45.000Z","updated_at":"2024-05-26T05:39:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"86a1324a-d506-49a6-b9fe-f21d1fe38735","html_url":"https://github.com/mchmarny/long-running-job","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Flong-running-job","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Flong-running-job/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Flong-running-job/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Flong-running-job/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mchmarny","download_url":"https://codeload.github.com/mchmarny/long-running-job/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249148031,"owners_count":21220465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","container","gce","gcp","gcs","golang","metadata","vm"],"created_at":"2024-11-08T12:28:15.460Z","updated_at":"2025-04-15T20:32:05.621Z","avatar_url":"https://github.com/mchmarny.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# long-running-job\n\n\u003e More detailed blog post on this including full gcloud commands is available at [m.chmarny.com](https://m.chmarny.com/how-to-run-containerized-workloads-in-gce-vm-87c7c3e952f3)\n\nWhile the idea of a serverless platform and long running workloads does seem somewhat \"unnatural\" at first, smart people are already working on that (looking at you @Knative community). In the meantime, a simple approach is sometimes all you may need.\n\nIn this demo I will illustrating how to use Google Compute Engine (GCE) container execution option to run variable duration jobs. This approach supports custom VMs, GPU/TPU accelerators and VPC networks… so may be handy alternative to other compute options on GCP. I'll also demo how to auto-terminate the created VM on container completion, so you won't have to pay for idle VM time. \n\nFinally, in this example I'll parses small gzip file from Google Cloud Storage (GCS), but since this approach is not limited by client timeouts you can use it to do pretty much anything… transformations on bigger files, lightweight ETL, or media format encoding. You can even combine it with GCP Task Queue for more complex pipelines.\n\n## Pre-requirements\n\nIf you don't have one already, start by creating new project and configuring [Google Cloud SDK](https://cloud.google.com/sdk/docs/).\n\n## Setup\n\nTo start, clone this repo, and navigate into that directory:\n\n```shell\ngit clone https://github.com/mchmarny/long-running-job.git\ncd long-running-job\n```\n\n## Run in GCE\n\n\u003e Note, to keep this readme short, I prepared series of scripts that you can execute rather than listing the complete commands. You should absolutely review each one of these scripts for content before executing it. This will help you understand the individual commands and allow you use them in the future.\n\n### Service Account\n\nTo execute this sample you will need a GCP service account. You can do that either in UI or using `gcloud` SDK. To find out more read [creating and managing service accounts](https://cloud.google.com/iam/docs/creating-managing-service-accounts).\n\nYou can enable the necessary API and create the specific service account for this demo using the [bin/setup](bin/setup) script. This script will also assign the new account all the necessary IAM roles and provision a service account key which will be saved in the `~/.gcp-keys` folder in your home directory. You should protect that key or just delete it after this demo.\n\n```shell\nbin/setup\n```\n\n### Container Image\n\nThe unit of code delivery to to the GCE VM will be container image. To create an image from this demo, you can use the [bin/image](bin/image) script\n\n```shell\nbin/image\n```\n\n### Deploy Container to VM\n\nTo create a new GCE VM and configure it to run the above built image, execute the [bin/deploy](bin/deploy) script\n\n```shell\nbin/deploy\n```\n\n### Container Logs\n\nOnce the VM started you can monitor the logs output from the VM to Stackdriver using the [bin/monitor](bin/monitor) script\n\n\u003e Note, this command will print only the logs that are output by the user code in the container. You can see the complete list of log entries by removing the `jsonPayload.message:\"[LRJ]\"` filter.\n\n```shell\nbin/monitor\n```\n\nAfter the container exists, the VM will be shutdown but the logs should be still available in Stackdriver for forensic analyses\n\n## Run Locally\n\nYou can run this code also locally by executing the [bin/run](bin/run) script\n\n```shell\nbin/run\n```\n\n## Disclaimer\n\nThis is my personal project and it does not represent my employer. I take no responsibility for issues caused by this code. I do my best to ensure that everything works, but if something goes wrong, my apologies is all you will get.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Flong-running-job","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmchmarny%2Flong-running-job","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Flong-running-job/lists"}