{"id":13643734,"url":"https://github.com/alexellis/actions-batch","last_synced_at":"2025-04-12T12:20:43.717Z","repository":{"id":87753419,"uuid":"547483189","full_name":"alexellis/actions-batch","owner":"alexellis","description":"Time-sharing supercomputer built on GitHub Actions","archived":false,"fork":false,"pushed_at":"2024-04-23T09:31:30.000Z","size":91,"stargazers_count":176,"open_issues_count":6,"forks_count":13,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-01T14:45:23.507Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://blog.alexellis.io/github-actions-timesharing-supercomputer/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexellis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-07T19:04:50.000Z","updated_at":"2024-06-21T19:17:15.012Z","dependencies_parsed_at":"2023-12-20T16:36:30.699Z","dependency_job_id":"c610103e-f732-41b4-84f4-da279f79c270","html_url":"https://github.com/alexellis/actions-batch","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexellis%2Factions-batch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexellis%2Factions-batch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexellis%2Factions-batch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexellis%2Factions-batch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexellis","download_url":"https://codeload.github.com/alexellis/actions-batch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248565179,"owners_count":21125435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:01:51.929Z","updated_at":"2025-04-12T12:20:43.683Z","avatar_url":"https://github.com/alexellis.png","language":"Go","readme":"# actions-batch\n\nTime-sharing supercomputer built on GitHub Actions.\n\nIn the 1970s - or so I hear, [time-sharing](https://en.wikipedia.org/wiki/Time-sharing) was all the rage, with users being able to submit tasks or batch jobs to large computers, and to collect the results when the jobs where done.\n\n[Read the blog post: GitHub Actions as a time-sharing supercomputer](https://blog.alexellis.io/github-actions-timesharing-supercomputer/)\n\n## Goal\n\nRun a shell script in an isolated, immutable environment, collect the logs or results.\n\nThis works well with self-hosted runners managed by actuated (which use a full VM) or GitHub's hosted runners. It may also work with container-based runners, with some limitations on what software can be used securely i.e. `docker`.\n\n\u003e Example output from [examples/cows-docker.sh](examples/cows-docker.sh):\n\n[![asciicast](https://asciinema.org/a/IazXh1cJnlaTdvx5c3Rxav8RI.svg)](https://asciinema.org/a/IazXh1cJnlaTdvx5c3Rxav8RI)\n\nIf your job creates output files, they can be fetched to your machine by making sure you copy them into the \"uploads\" file in your bash script.\n\n## What's supported?\n\n* [x] Public repos\n* [x] Private repos\n* [x] Self-hosted runners\n* [x] GitHub-hosted runners\n* [x] Using secrets via repository secrets\n* [x] Downloading the results of a build as an artifact\n\n## You might also be interested in\n\n* [alexellis/run-job](https://github.com/alexellis/run-job) - run one-shot jobs on Kubernetes\n* [OpenFaaS](https://github.com/openfaas/faas) - portable, open-source FaaS for Kubernetes and [standalone containerd](https://github.com/openfaas/faasd)\n\n## How it works\n\n1. You write a bash script like the ones in [examples](/examples) and pass it in as an argument\n1. A new repo is created with a random name in the specified organisation\n2. A workflow file is written to the repo along with the shell script, the workflow's only job is to run the shell script and exit\n3. The workflow is triggered and you can check the results\n\n`-owner` is intended to be a GitHub organisation, but this can be adapted to a personal account by passing `--org false` to the command.\n\n```bash\ngit clone git@github.com:alexellis/actions-batch\ncd actions-batch\n```\n\nYou'll need a Personal Access Token (PAT) with: delete_repo, repo, workflow, write:packages.\n\nYou can download a binary from the [Releases page](https://github.com/alexellis/actions-batch/releases) or build it from source.\n\n### Generate some ASCII art with your personal repo\n\nFor a personal account using a hosted runner and a public repo:\n\n```bash\nactions-batch \\\n  --owner alexellis \\\n  --token-file ~/pat.txt \\\n  --runs-on ubuntu-latest \\\n  --file examples/cowsay.sh\n```\n\n### Run an LVM using llama and a model from HuggingFace\n\nUse [examples/llama.sh](/examples/llama.sh), snippet below:\n\n```python3\nfrom llama_cpp import Llama\nLLM = Llama(model_path=\"./llama-2-7b-chat.Q5_K_M.gguf\")\n   \n# create a text prompt\nprompt = \"Q: What are the names of the days of the week? A:\"\n\n# generate a response (takes several seconds)\noutput = LLM(prompt,max_tokens=300, stop=[])\n```\n\nUse the `--out ./out` flag, then `cat ./out/output.txt` to see the results.\n\n```\n The names of the days of the week are: Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday.\nQ: How many days are in a week? A: There are 7 days in a week.\n\n[{'text': ' The names of the days of the week are: Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday.\\nQ: How many days are in a week? A: There are 7 days in a week.', 'index': 0, 'logprobs': None, 'finish_reason': 'stop'}]\n```\n\n### Run Docker slim against an image using a self-hosted runner and a GitHub organisation\n\nFor an organisation, using an Arm-based private repo and custom runner with 32vCPU and 256GB of RAM\n\n```bash\nactions-batch \\\n  --private \\\n  --owner actuated-samples \\\n  --token-file ~/pat.txt \\\n  --runs-on actuated-arm64-32cpu-256gb \\\n  --file examples/slim.sh\n```\n\nExample of what's written to the repo:\n\n```bash\nname: workflow\n\n# Generated by alexellis/actuated-batch at: 2022-10-07 20:14:30.547822 +0100 BST m=+0.002729293\n# Job requested by alex\n\non:\n  pull_request:\n    branches:\n      - '*'\n  push:\n    branches:\n      - master\n      - main\n\njobs:\n  workflow:\n    name: agitated_solomon5\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v1\n      - name: Run the job\n        run: |\n          chmod +x ./job.sh\n          ./job.sh\n```\n\n### Consume secrets for a job\n\nA folder can be given, where each file is a secret, the name will be the filename made uppercase, with `-` replaced by `_`.\n\nTake the following example to log into OpenFaaS:\n\n```bash\n#!/bin/bash\n\nset -e -x -o pipefail\n\ncurl -sLS https://get.arkade.dev | sudo sh\n\narkade get faas-cli --quiet\nsudo mv $HOME/.arkade/bin/faas-cli /usr/local/bin/\nsudo chmod +x /usr/local/bin/faas-cli \n\necho \"${OPENFAAS_GATEWAY_PASSWORD}\" | faas-cli login -g \"${OPENFAAS_URL}\"/function/printer -u admin --password-stdin\ncurl -h \"X-Url: ${OPENFAAS_URL}\" https://openfaas.o6s.io/function/printer \n```\n\nTherefore just create two files:\n\n```bash\nmkdir -p .secrets\necho \"admin password\" \u003e .secrets/openfaas-gateway-password\necho \"https://gateway.example.com\" \u003e .secrets/openfaas-url\n```\n\nThen run the script passing in that folder:\n\n```bash\nactions-batch \\\n  --private=false \\\n  --owner alexellis \\\n  --token-file ~/batch \\\n  --runs-on ubuntu-latest \\\n  --org=false \\\n  --file examples/secret.sh \\\n  --secrets-from .secrets\n```\n\n[OpenFaaS can be exposed over the Internet using an inlets tunnel](https://inlets.dev/blog/2020/10/15/openfaas-public-endpoints.html).\n\n### Download files from a job\n\nYou may be processing a video with ffmpeg, building a binary, a container image, PDF, or even an ISO.\n\nPlace the result in a folder named `uploads` and the contents will be uploaded as an artifact to GitHub Actions, before being downloaded to your machine and extracted for you to view.\n\nBear in mind that if your artifact is confidential or private, then you will need to use the `--private` flag to create a private repo.\n\nA good example is [examples/youtubedl.sh](examples/youtubedl.sh) which downloads a video from YouTube.\n\n```bash\nactions-batch \\\n  --private \\\n  --owner actuated-samples \\\n  --token-file ~/batch \\\n  --runs-on actuated \\\n  --file ./examples/youtubedl.sh\n```\n\nOutput:\n\n```bash\nWrote 43.84kB to /tmp/uploads1127046452\nExtracting: /tmp/artifacts-968188/video.flv\n2023/12/21 16:04:01 extracted zip into /tmp/artifacts-968188: 1 files, 0 dirs (672.583µs)\nFILE      SIZE\nvideo.flv 47.39kB\n\nxdg-open /tmp/artifacts-968188/video.flv\n```\n\n[![Example video playing](https://pbs.twimg.com/media/GB4eGfzXcAAQOlE?format=jpg\u0026name=medium)](https://twitter.com/alexellisuk/status/1737859786413322477/)\n\n### Build a Linux Kernel and download it a working folder\n\n```bash\nmkdir -p kernel-bin\nactions-batch \\\n  --owner alexellis \\\n  --org=false \\\n  --file examples/linux-kernel.sh \\\n  --runs-on ubuntu-latest \\\n  --out ./kernel-bin\n\ndu -h kernel-bin/*\n```\n\n### Build a Docker image remotely and import it to your library\n\n```bash\nactions-batch \\\n--private=false \\\n--owner actuated-samples \\\n--token-file ~/batch \\\n--runs-on ubuntu-latest \\\n--file ./examples/export-docker-image.sh \n\nWrote 5.665MB to /tmp/uploads2533248860\nExtracting: /tmp/artifacts-888719363/curl.tar\n2023/12/21 17:19:32 extracted zip into /tmp/artifacts-888719363: 1 files, 0 dirs (77.250512ms)\nFILE     SIZE\ncurl.tar 12.37MB\n\nQUEUED DURATION TOTAL\n4s     15s      22s\n\ndocker load -i /tmp/artifacts-888719363/curl.tar\n08db363dedea: Loading layer [==================================================\u003e]  4.687MB/4.687MB\nLoaded image: curl:latest\n\ndocker run -t curl:latest --version\n```\n\n## What's left\n\nThe main thing I'd like is more examples of workloads that can be run on a Linux system\n\n* [Browse the examples](/examples)\n* [See the issue tracker](https://github.com/alexellis/actions-batch/issues)\n\n## License\n\nMIT\n\nDCO - a Signed-off-by message will be required in each commit message i.e. `git commit --signoff`\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexellis%2Factions-batch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexellis%2Factions-batch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexellis%2Factions-batch/lists"}