{"id":18432120,"url":"https://github.com/dustinspecker/linux-job-worker-service","last_synced_at":"2025-04-14T01:20:38.953Z","repository":{"id":249898334,"uuid":"832770341","full_name":"dustinspecker/linux-job-worker-service","owner":"dustinspecker","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-07T17:27:28.000Z","size":65,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-16T09:26:34.261Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dustinspecker.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-23T17:31:51.000Z","updated_at":"2024-08-07T17:27:32.000Z","dependencies_parsed_at":"2024-11-06T05:28:14.709Z","dependency_job_id":"5b9ef254-8609-447f-9da5-cb979294f66f","html_url":"https://github.com/dustinspecker/linux-job-worker-service","commit_stats":null,"previous_names":["dustinspecker/linux-job-worker-service"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustinspecker%2Flinux-job-worker-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustinspecker%2Flinux-job-worker-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustinspecker%2Flinux-job-worker-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustinspecker%2Flinux-job-worker-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dustinspecker","download_url":"https://codeload.github.com/dustinspecker/linux-job-worker-service/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248805177,"owners_count":21164232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T05:27:29.897Z","updated_at":"2025-04-14T01:20:38.920Z","avatar_url":"https://github.com/dustinspecker.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Linux Job Worker Service\n\n\u003e A library to run jobs on Linux 64-bit systems in an isolated manner.\n\n## Purpose\n\nThis library is intended to be used to run arbitrary commands on Linux 64-bit systems. To help prevent disruptions\nto other processes running on the system, the job is run in an isolated manner. This isolation includes:\n\n- new PID namespace to prevent killing other processes on the host\n- new mount namespace to mount a new proc filesystem so that the job can't see other processes on the host\n- new network namespace to prevent the job from accessing the local network and internet\n- configurable CPU, memory, and IO limits to throttle the job using cgroups v2\n\n## Install\n\n```bash\ngo get github.com/dustinspecker/linux-job-worker-service\n```\n\n## Running the server and client\n\n1. Install [Go](https://golang.org/doc/install)\n1. Build the server and client binaries via:\n\n   ```bash\n   make build\n   ```\n\n1. Generate certificates via:\n\n   ```bash\n   make certs\n   ```\n\n1. Find the physical device Major/Minor of where `/` is mounted by running:\n\n   ```bash\n   lsblk\n   ```\n\n   example output:\n\n   ```\n   NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS\n   loop0         7:0    0     4K  1 loop /snap/bare/5\n   loop1         7:1    0  63.9M  1 loop /snap/core20/2318\n   loop2         7:2    0  74.2M  1 loop /snap/core22/1380\n   loop3         7:3    0 268.9M  1 loop /snap/firefox/4630\n   loop4         7:4    0 268.4M  1 loop /snap/firefox/4650\n   loop5         7:5    0  10.7M  1 loop /snap/firmware-updater/127\n   loop6         7:6    0 349.7M  1 loop /snap/gnome-3-38-2004/143\n   loop7         7:7    0 505.1M  1 loop /snap/gnome-42-2204/176\n   loop8         7:8    0  91.7M  1 loop /snap/gtk-common-themes/1535\n   loop9         7:9    0  10.4M  1 loop /snap/snap-store/1147\n   loop10        7:10   0  10.5M  1 loop /snap/snap-store/1173\n   loop11        7:11   0  38.7M  1 loop /snap/snapd/21465\n   loop12        7:12   0  38.8M  1 loop /snap/snapd/21759\n   loop13        7:13   0   476K  1 loop /snap/snapd-desktop-integration/157\n   loop14        7:14   0 183.7M  1 loop /snap/spotify/77\n   loop15        7:15   0 181.8M  1 loop /snap/spotify/78\n   sr0          11:0    1  86.3M  0 rom\n   nvme0n1     259:0    0 931.5G  0 disk\n   ├─nvme0n1p1 259:2    0   100M  0 part\n   ├─nvme0n1p2 259:3    0    16M  0 part\n   ├─nvme0n1p3 259:4    0 930.7G  0 part\n   └─nvme0n1p4 259:5    0   749M  0 part\n   nvme1n1     259:1    0 931.5G  0 disk\n   ├─nvme1n1p1 259:6    0     1G  0 part /boot/efi\n   └─nvme1n1p2 259:7    0 930.5G  0 part /var/snap/firefox/common/host-hunspell\n                                         /\n   nvme2n1     259:8    0   1.8T  0 disk\n   ├─nvme2n1p1 259:9    0    16M  0 part\n   └─nvme2n1p2 259:10   0   1.8T  0 part\n   ```\n\n   Find the MAJ:MIN of the disk type that `/` is mounted on. In the above output, `259:1` is the MAJ:MIN of the disk that `/` is mounted on.\n\n1. Start the server via:\n\n   ```bash\n   sudo ./bin/server -root-physical 259:1\n   ```\n\n1. Start a job by running:\n\n   ```bash\n   ./bin/client start -cpu 1 -memory 1000000000 -io 10000000 echo Hello World!\n   ```\n\n   The output will look something like:\n\n   ```\n   job id: \u003cuuid\u003e\n   ```\n\n   Take note of the UUID as it can be used to get the job's status and output. As well as stopping the job.\n\n1. Get the job's combined stderr/stdout via:\n\n   ```bash\n   ./bin/client stream \u003cuuid\u003e\n   ```\n\n   The output will look something like:\n\n   ```\n   Hello World!\n   ```\n\n   Note: `stream` will print until the job completes or the user interrupts the command.\n\n1. Get the job's status via:\n\n   ```bash\n   ./bin/client status \u003cuuid\u003e\n   ```\n\n   The output will look something like:\n\n   ```\n   status: completed\n   exit code: 0\n   exit reason:\n   ```\n\n1. Long running jobs can be stopped via:\n\n   ```bash\n   ./bin/client stop \u003cuuid\u003e\n   ```\n\n## Library usage\n\nAn example to run a job that echoes \"Hello, World!\":\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"os\"\n\n\t\"github.com/dustinspecker/linux-job-worker-service/pkg/executor\"\n\t\"github.com/dustinspecker/linux-job-worker-service/pkg/job\"\n)\n\nfunc main() {\n\tif len(os.Args) \u003e 3 \u0026\u0026 os.Args[1] == \"run\" {\n\t\terr := executor.Run(os.Args[2], os.Args[3:])\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"error running command: %v\", err)\n\t\t}\n\t\treturn\n\t}\n\n\t// create a job config\n\tconfig := job.Config{\n\t\tRootPhysicalDeviceMajMin: \"259:1\",          // example of how to find this value is in the Development section\n\t\tJobExecutorPath:          \"/proc/self/exe\", // this will re-run the same executable but as `\u003cexe\u003e run command arguments\n\t\tCommand:                  \"echo\",\n\t\tArguments:                []string{\"Hello, World!\"},\n\t\tIOInBytes:                10_000_000,  // 10 MB read/write limit on the physical RootDeviceMajorMinor device\n\t\tMemoryInBytes:            100_000_000, // 100 MB memory limit for the command to use\n\t\tCPU:                      0.5,         // 50% of a single CPU core for the command to be throttled to\n\t}\n\n\t// create job\n\techoJob := job.New(config)\n\n\t// start the job\n\tif err := echoJob.Start(); err != nil {\n\t\tpanic(err)\n\t}\n\n\t// for long-lived jobs, they may optionally be stopped by running:\n\t// if err := job.Stop(); err != nil {\n\t//     panic(err)\n\t// }\n\n\t// collect the job's combined stdout and stderr until the job completes\n\toutput, err := io.ReadAll(echoJob.Stream())\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tfmt.Println(string(output))\n\n\t// get the job's exit code\n\tstatus := echoJob.Status()\n\tfmt.Printf(\"exit code: %d\", status.ExitCode)\n}\n```\n\nNote: it is important that the application supports running `executor.Run` when the program is executed\nas `\u003cexe\u003e run command arguments`. The job library will re-execute the program with `run` when the job\nconfiguration is told to use `/proc/self/exe` as the `JobExecutorPath`.\n\n## Development\n\n1. Install [Go](https://golang.org/doc/install)\n1. Install [protoc and Go plugins](https://grpc.io/docs/languages/go/quickstart/)\n1. Clone repository\n1. Run linting via (be sure to install [golangci-lint 1.59.1](https://github.com/golangci/golangci-lint/releases/tag/v1.59.1):\n\n   ```bash\n   make lint\n   ```\n\n1. Run unit tests via:\n\n   ```bash\n   make test-unit\n   ```\n\n1. Build binaries via:\n\n   ```bash\n   make build\n   ```\n\n1. Find the physical device Major/Minor of where `/` is mounted by running:\n\n   ```bash\n   lsblk\n   ```\n\n   example output:\n\n   ```\n   NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS\n   loop0         7:0    0     4K  1 loop /snap/bare/5\n   loop1         7:1    0  63.9M  1 loop /snap/core20/2318\n   loop2         7:2    0  74.2M  1 loop /snap/core22/1380\n   loop3         7:3    0 268.9M  1 loop /snap/firefox/4630\n   loop4         7:4    0 268.4M  1 loop /snap/firefox/4650\n   loop5         7:5    0  10.7M  1 loop /snap/firmware-updater/127\n   loop6         7:6    0 349.7M  1 loop /snap/gnome-3-38-2004/143\n   loop7         7:7    0 505.1M  1 loop /snap/gnome-42-2204/176\n   loop8         7:8    0  91.7M  1 loop /snap/gtk-common-themes/1535\n   loop9         7:9    0  10.4M  1 loop /snap/snap-store/1147\n   loop10        7:10   0  10.5M  1 loop /snap/snap-store/1173\n   loop11        7:11   0  38.7M  1 loop /snap/snapd/21465\n   loop12        7:12   0  38.8M  1 loop /snap/snapd/21759\n   loop13        7:13   0   476K  1 loop /snap/snapd-desktop-integration/157\n   loop14        7:14   0 183.7M  1 loop /snap/spotify/77\n   loop15        7:15   0 181.8M  1 loop /snap/spotify/78\n   sr0          11:0    1  86.3M  0 rom\n   nvme0n1     259:0    0 931.5G  0 disk\n   ├─nvme0n1p1 259:2    0   100M  0 part\n   ├─nvme0n1p2 259:3    0    16M  0 part\n   ├─nvme0n1p3 259:4    0 930.7G  0 part\n   └─nvme0n1p4 259:5    0   749M  0 part\n   nvme1n1     259:1    0 931.5G  0 disk\n   ├─nvme1n1p1 259:6    0     1G  0 part /boot/efi\n   └─nvme1n1p2 259:7    0 930.5G  0 part /var/snap/firefox/common/host-hunspell\n                                         /\n   nvme2n1     259:8    0   1.8T  0 disk\n   ├─nvme2n1p1 259:9    0    16M  0 part\n   └─nvme2n1p2 259:10   0   1.8T  0 part\n   ```\n\n   Find the MAJ:MIN of the disk type that `/` is mounted on. In the above output, `259:1` is the MAJ:MIN of the disk that `/` is mounted on.\n\n1. Generate certificates via:\n\n   ```bash\n   make certs\n   ```\n\n1. Run the server via:\n\n   ```bash\n   sudo ./bin/server -root-physical 259:1\n   ```\n\n1. Run integration tests via\n\n   ```bash\n   sudo env PATH=$PATH ROOT_DEVICE_MAJ_MIN=259:1 make test-integration\n   ```\n\n   Note: `sudo` is required since the integration tests create new namespaces and control groups (cgroups).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdustinspecker%2Flinux-job-worker-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdustinspecker%2Flinux-job-worker-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdustinspecker%2Flinux-job-worker-service/lists"}