{"id":19740621,"url":"https://github.com/juliagpu/buildkite","last_synced_at":"2025-04-30T05:33:14.795Z","repository":{"id":39064626,"uuid":"310036872","full_name":"JuliaGPU/buildkite","owner":"JuliaGPU","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-24T12:07:04.000Z","size":144,"stargazers_count":21,"open_issues_count":2,"forks_count":6,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-25T10:42:58.525Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JuliaGPU.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-04T15:06:00.000Z","updated_at":"2024-10-24T12:07:08.000Z","dependencies_parsed_at":"2024-03-08T17:25:25.461Z","dependency_job_id":"fd83974b-8f9a-461e-854f-816628f0423e","html_url":"https://github.com/JuliaGPU/buildkite","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaGPU%2Fbuildkite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaGPU%2Fbuildkite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaGPU%2Fbuildkite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaGPU%2Fbuildkite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JuliaGPU","download_url":"https://codeload.github.com/JuliaGPU/buildkite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224199811,"owners_count":17272341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T01:22:35.254Z","updated_at":"2024-11-12T01:22:35.826Z","avatar_url":"https://github.com/JuliaGPU.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JuliaGPU Buildkite\n\nThis repository contains resources related to the JuliaGPU Buildkite CI infrastructure:\n\n- a custom Ubuntu-based image that can be based off of another image (e.g.\n  CUDA's images);\n- support for encrypted environment variables in the pipeline, and an\n  environment hook to decode them;\n- Docker Compose templates and systemd service files to tie everything together,\n  and give each job a safe and reproducible execution environment.\n\nMany of these features come from the fact that the JuliaGPU CI is intended to be\nrun on a variety of repositories (and external PRs to those repositories),\nwhereas Buildkite is typically used within the (trusted) boundaries of a single\norganization.\n\nThis README documents how to use the JuliaGPU Buildkite CI. For details on the available\nagents, or how to add one, see the [agents README](agents/README.md). **Note that this\nBuildkite set-up is fairly new, and the set-up may still change. Keep an eye on this README\nif anything breaks.**\n\n\n## Adding a repository\n\nIf your Julia package can use a GPU, you may use the JuliaGPU CI infrastructure to run tests\non a system with a GPU. This is **not a general-purpose CI service**, and only intended to\nrun GPU-related tests of Julia packages. For all other testing, use public infrastructure\n(Travis, Github Actions, etc).\n\n### Steps for administrators\n\nBefore everything else, the [the Buildkite\napp](https://github.com/settings/connections/applications/Iv1.112bf4be3e5ecdeb) needs to be\navailable for the GitHub organization that hosts your repository. If the organization is not\nlisted, a user with administrative privileges on both the JuliaLang BuildKite and the GitHub\norganization in question should [set that\nup](https://buildkite.com/organizations/julialang/repository-providers/new) and make sure\nthe organization is then listed as a [repository\nprovider](https://buildkite.com/organizations/julialang/repository-providers). You should\ncoordinate this on the #gpu channel of the JuliaLang slack, and ping BuildKite admins\n(@maleadt, @vchuravy, @DilumAluthge, ...).\n\nNext, a BuildKite admin should set-up a pipeline for your repository:\n\n1. [Create a new pipeline](https://buildkite.com/organizations/julialang/pipelines/new),\n   using the package's name.\n\n   a. Under `Git Repository`, select the GitHub organization (if it isn't listed, the\n   BuildKite app isn't properly set up) and the repository from the drop-down. Make sure\n   check-out uses HTTPS.\n\n   b. Check `Auto-create webhooks`.\n\n   c. Grant permission to the JuliaGPU team.\n\n2. Use the following steps, and click `Save and Close`:\n\n   ```yaml\n   steps:\n     - label: \":buildkite: Pipeline upload\"\n       command: buildkite-agent pipeline upload\n       branches: \"!gh-pages\"\n       agents:\n         queue: \"juliagpu\"\n   ```\n\n3. Navigate to the Pipeline Settings.\n\n   a. Under general settings, make the pipeline public by clicking the big green button.\n\n   b. Under GitHub settings\n   \n      - check the box to `Build pull requests from third-party forked repositories`\n      - check the box to `Build tags`\n      - set the branch filter to `master v*` (and other branches you want to run CI for,\n      e.g., `release-*`)\n\n### Steps for developers\n\nFinally, you should create `.buildkite/pipeline.yml` in your repository with the steps to\nperform GPU CI. Start from the following template:\n\n```yaml\nenv:\n  SECRET_CODECOV_TOKEN: \"...\"\n\nsteps:\n  - label: \"Julia v1\"\n    plugins:\n      - JuliaCI/julia#v1:\n          version: \"1\"\n      - JuliaCI/julia-test#v1: ~\n      - JuliaCI/julia-coverage#v1:\n          codecov: true\n    agents:\n      queue: \"juliagpu\"\n      cuda: \"*\"\n    if: build.message !~ /\\[skip tests\\]/\n    timeout_in_minutes: 60\n```\n\nIf you want to perform several tests, or introduce test phases, you may want to add\nadditional steps. Note that each step is run in a fresh container (as opposed to how\nBuildkite normally works), so you need to repeat the set-up that provides Julia (here done\nby the `julia` plugin) and/or instantiates your project (done by the `julia-test` plugin).\nIf you need to send resources across steps, use\n[artifacts](https://buildkite.com/docs/pipelines/artifacts).\n\nFor coverage submission to Codecov to work, you need to encrypt your `CODECOV_TOKEN` and\nspecify it as a global `SECRET_CODECOV_TOKEN` (see below).\n\n\n\n## Using secrets\n\nDuring start-up, agents will scan for `SECRET_` environment variables and decrypt their\ncontents for use in the rest of the pipeline. If you want to use this mechanism to provide,\nsay, a secret `CODECOV_TOKEN`, run the `encrypt` script in this repository and follow its\nprompts:\n\n\n```\n$ ./tools/encrypt\nVariable name: CODECOV_TOKEN\nSecret value:\n\nUse the following snippet in your pipeline.yml:\n\nenv:\n  SECRET_CODECOV_TOKEN: \"kaIXEN51HinaQ4JGclQcIgxeMMtXDb5uvnP3E2eKrH4Eruf2pKd5QwUGcIVL8+rcWeo5FWj883rNxRQEH3YeCWs6/i7vzs+ORvG51QeCNYQgNqFzPsWRcq5qJYc+JPFbisS7q9nghqWTwr52cnjarD4Xx3ceGorMyS5NvFpCNxMgqHNyGkLvipxcTTJfKZK61bpnbntoIjiIO1XSZKjcxnXFGFnolV9BHCr5v8f7F42n2tUH7X3nDHmTBr1AbO2lFAU9ra/KezHcIf0wg2HcV8LZD0+mj8q/SBPjQZSH7cxwx4Q2eTjT4Sw7xnrBGuySVm8ZPCAV7nRNEHo+VqR+GQ==\"\n```\n\nIf your version of OpenSSL is too old, the `./tools/encrypt` script may fail.\nIn that case, you can run it inside Docker:\n```bash\ndocker run --rm -it -v $(pwd):/root ubuntu bash -c 'apt update \u0026\u0026 apt install -y openssl \u0026\u0026 /root/tools/encrypt'\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliagpu%2Fbuildkite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuliagpu%2Fbuildkite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliagpu%2Fbuildkite/lists"}