{"id":15157698,"url":"https://github.com/pytorch/ossci-job-dsl","last_synced_at":"2025-09-30T07:30:42.435Z","repository":{"id":45137155,"uuid":"122256864","full_name":"pytorch/ossci-job-dsl","owner":"pytorch","description":"Jenkins job definitions for OSSCI","archived":true,"fork":false,"pushed_at":"2022-01-05T21:51:38.000Z","size":1042,"stargazers_count":10,"open_issues_count":3,"forks_count":33,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-09-27T20:03:21.578Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Groovy","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pytorch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-20T21:11:23.000Z","updated_at":"2024-08-26T00:13:26.000Z","dependencies_parsed_at":"2022-08-26T08:50:54.862Z","dependency_job_id":null,"html_url":"https://github.com/pytorch/ossci-job-dsl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Fossci-job-dsl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Fossci-job-dsl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Fossci-job-dsl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Fossci-job-dsl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pytorch","download_url":"https://codeload.github.com/pytorch/ossci-job-dsl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234712854,"owners_count":18875523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T20:01:40.855Z","updated_at":"2025-09-30T07:30:42.040Z","avatar_url":"https://github.com/pytorch.png","language":"Groovy","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OSSCI Job DSL repository\n\nThis is a [Job DSL](https://github.com/jenkinsci/job-dsl-plugin) project\nfor FAIR/AML's OSSCI infrastructure.\n\nLooking for information about the actual machines the jobs run on?\nSee [fairinternal/ossci-infra](https://github.com/fairinternal/ossci-infra).\n\n## CI failed, but my local build is fine. What should I do?!\n\n### Cheat sheet\n\n- Search for `DOCKER_IMAGE:` line at the top of the `Test` phase; it should have a line\n  like `308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-trusty-py3.6-gcc5.4:tmp-173-5910`;\n  this is your docker image.  (If the tag is `tmp-###-####`, it comes with\n  a build of your source; if it's `###` that's the stock image.)  If you can't see it,\n  you might need to download the full log and look for it.\n- Run `aws configure` and set the default region to `us-east-1`.\n  If the aws command is not installed, install it via the instructions\n  in https://aws.amazon.com/cli/ (usually, you can use `pip install awscli` to install AWS CLI.)\n- Get the public access key and secret access key at https://fb.quip.com/oAX3ApaV35jU\n  (Facebook employees only).  If you're a non-Facebook employee, talk\n  to @ezyang about getting access.\n- If you have AWS CLI v1 run `aws ecr get-login` with your AWS credentials to get your Docker\n  login command.  Run this command to login.\n    + If you get the error `unknown shorthand flag: 'e' in -e`, \n      delete `-e none` from the command line.\n    + If you can't connect to the Docker daemon, you need to `sudo addgroup $username docker`\n      and then log out and then re-login\n- If you have AWS CLI v2 run `aws ecr get-login-password | docker login --username AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com`\n- Run `docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it $DOCKER_IMAGE /bin/bash`\n\nTry prepending sudo if you get the `permission denied` error for the docker commands\n(and later figure out why your user doesn't have permissions to connect\nto the Docker socket; maybe you need to add yourself to the docker\ngroup and reboot).\n\nWant to run a Docker image on a GPU?  Standard issue devgpus don't\nallow use of Docker, so you will have to either (1) run docker\non devfair, (2) get a GPU-enabled AWS instance (the OSS CI\nteam has a few allocated, get in touch with them to see how\nto connect), (3) find a GPU machine that you're managing yourself.\nAll of these will require some time to provision, so don't try to\ndo this last minute.\n\nWant to know more about what Docker images are available? See\n\"Available docker images.\"\n\n### Available Docker images\n\nIf you just want to reproduce a test error, there is the particular\nDocker image for your job which you should pull and test.  But if you're\ninterested in repurposing our CI Docker images for other purposes,\nit helps to know about the general structure of the Docker images our\nCI exposes and how they are built (so you can find the URL for a base\nimage you might be interested in.)\n\nFor historical reasons, there are two sets of Docker images, one for\nPyTorch and one for Caffe2 (we intend to merge these at some point, but\nwe haven't finished yet.\n\nPyTorch Dockerfiles source lives at https://github.com/pytorch/pytorch-ci-dockerfiles\nand are built every week at https://ci.pytorch.org/jenkins/job/pytorch-docker-master/\n\nCaffe2 Dockerfiles source lives at https://github.com/pytorch/pytorch/tree/master/docker/caffe2/jenkins\nand are built upon request at https://ci.pytorch.org/jenkins/job/caffe2-docker-trigger/\n\n### Advanced tricks\n\n**Summary for gdb-enabled CPU:**\n\n    ssh ubuntu@$CPU_HOST\n    docker run --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -u jenkins -i $DOCKER_IMAGE /bin/bash\n\n**Summary for ASAN builds (jobs like pytorch_linux_xenial_py3_clang5_asan_test)\n\n    ssh ubuntu@$CPU_HOST\n    docker run --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -u jenkins -i $DOCKER_IMAGE /bin/bash\n    export LD_PRELOAD=/usr/lib/llvm-5.0/lib/clang/5.0.0/lib/linux/libclang_rt.asan-x86_64.so\n    cd ~/workspace\n    # run your test repro\n\n**Summary for gdb-enabled NVIDIA/CUDA GPU**\n\n    ssh ubuntu@$GPU_HOST\n    docker run --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -u jenkins -i --runtime=nvidia -e CUDA_VERSION=8 -e NVIDIA_VISIBLE_DEVICES=all $DOCKER_IMAGE /bin/bash\n\n**Summary for AMD/ROCM GPU**\n\n    ssh -p $AMD_PORT $AMD_USERNAME@$AMD_HOST\n    docker run --device=/dev/kfd --device=/dev/dri --group-add video -it $DOCKER_IMAGE /bin/bash\n\nGet credentials for an AMD GPU machine at https://fb.quip.com/Luj5AQjlH11U\nThis should only be necessary if you actually\nplan to run tests on an AMD GPU; if you are debugging build failures,\nany old host is OK, though make sure you have 16G of RAM (at least).\n\n**What is my CPU/GPU HOST?**\n\n- If you don't need the exact same hardware, you can run these commands\n  on any machine that has Docker\n- There are some AWS dev machines which can be used.  This particular\n  author likes to use ec2-52-90-201-109.compute-1.amazonaws.com\n  The canonical information about all our running instances can be found\n  on [AWS console](https://console.aws.amazon.com/ec2/v2/home?\u0026region=us-east-1#Instances:sort=desc:tag:Name);\n  you'll need a login under the 'caffe2' account, ask @pietern for\n  access.\n\n**What is my Docker image?**\n\nYour Docker image will look something like\n`registry.pytorch.org/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py2:69-3002`.\n\nThis image is:\n- `COMMIT_DOCKER_IMAGE` in the build log, and\n- `DOCKER_IMAGE` in the test log\n\nYou can tell you've got the right one because Jenkins homedir\nwill have a `workspace` directory.\n\n**What do I do once I'm in?**\n\nRead the actual [jobs](https://github.com/pytorch/ossci-job-dsl/tree/master/jobs)\ndirectory to see how to actually build/test (at the very least, you will\nneed to set `PATH` to pick up the correct Python executable.)\n\nYou **DO NOT** need to build PyTorch; it will already be installed. But\nif you want to inject debugging code - feel free to and use regular `python setup.py develop` instructions in `~/workspace`.\n\n#### Appendix\n\n**What do all the flags in the docker run command mean?**\n\n* The `--rm` argument ensures that the Docker image gets immediately\n  deleted when you exit.  If you don't want this, delete `--rm`...\n  but don't forget to `docker rm` the image when you are done\n  (stopping it is not sufficient.)\n\n* By default Docker does not enable ptrace, which means that you will\n  have a hard time running gdb inside the docker image.\n  `--cap-add=SYS_PTRACE --security-opt seccomp=unconfined` ensures\n  this capability is allowed.\n\n* The docker image will have a pretty bare set of installed packages.\n  To install more, run `sudo apt update` and then `sudo apt install`\n  for the packages you want.  `gdb` and `vim` can be quite useful.\n\n**The CUDA docker command didn't work.**\n\nYou need to install [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker/tree/2.0)\nwhich knows how to expose CUDA devices inside Docker.\n\n**Where is my source?**\n\nCaffe2 builds don't currently store their source code in test images;\nyou will need to git clone a copy of the source and checkout the correct\none.\n\n### Mac OS X\n\nOS X builds are not containerized.  You probably have a Macbook;\nfirst try reproducing locally.  Otherwise, see https://fb.quip.com/FIDAOAi7r2A\nfor canonical information about our OS X workers. (Facebook employees\nonly).\n\nChanges you make to these machines affect everyone, so please be careful.\n\n### Windows\n\nYou must Remote Desktop into the Windows machines; see [this\nQuip](https://fb.quip.com/5itGAsp277me) for information how to access\n(you may have to ask for access.)\n\nChanges you make to these machines affect everyone, so please be careful.\n\n## Job structure\n\nTaking PyTorch as an example (much of the same applies to Caffe2),\nhere is how we structure our jobs:\n\n* The pytorch-docker builds are responsible for taking the Dockerfiles\n  from [pytorch-dockerfiles](https://github.com/pytorch/pytorch-ci-dockerfiles) and building\n  Docker images.  These images are uploaded to\n  `registry.pytorch.org/pytorch`.  Every new built Docker image gets a new\n  tag, which is a sequentially incrementing number.\n\n    * After a Docker build completes, we test and make sure that master\n      of PyTorch builds with the new image (in case changes in the image\n      introduced a regression.)  If it passes, the Docker build process\n      will deploy the image, by making a commit \"Update PyTorch DockerVersion\"\n      which updates the latest PyTorch DockerVersion (the tag, really)\n      in `./src/main/groovy/ossci/pytorch/DockerVersion.groovy`.\n      (You can also manually update the active DockerVersion by editing\n      this file.)\n\n* PyTorch builds are intermediated by a top level trigger build for\n  commits to master, and pull requests (using Jenkins GitHub Pull\n  Request Builder).\n\n    * This kicks off parallel builds for each system configuration\n      we are interested in (at the moment, only Python 2 and Python 3,\n      but we will be adding CUDA 8 and CUDA 9 permutations as well.)\n\n    * The configuration build itself is split into two phases.  The\n      first phase *only* builds and installs PyTorch into the Docker\n      image.  We then push the Docker image to the registry.  This\n      build is done on a CPU-only machine.  The second phase tests\n      PyTorch on a GPU provisioned machine by loading the Docker image.\n\n* There are also miscellaneous cronjobs for cleaning old Docker images\n  from the registry and the local builders.\n\n## File structure\n\n    .\n    ├── jobs                    # DSL script files\n    ├── resources               # resources for DSL scripts\n    ├── src\n    │   ├── main\n    │   │   ├── groovy          # support classes\n    │   │   └── resources\n    │   │       └── idea.gdsl   # IDE support for IDEA\n    │   └── test\n    │       └── groovy          # specs\n    └── build.gradle            # build file\n\n## Testing\n\n`./gradlew test` runs the specs.\n\n[JobScriptsSpec](src/test/groovy/com/dslexample/JobScriptsSpec.groovy) \nwill loop through all DSL files and make sure they don't throw any exceptions when processed. All XML output files are written to `build/debug-xml`. \nThis can be useful if you want to inspect the generated XML before check-in.\n\n## Seed Job\n\nYou can create the example seed job via the Rest API Runner (see below) using the pattern `jobs/seed.groovy`.\n\nOr manually create a job with the same structure:\n\n* Invoke Gradle script\n   * Use Gradle Wrapper: `true`\n   * Tasks: `clean test`\n* Process Job DSLs\n   * DSL Scripts: `jobs/**/*Jobs.groovy`\n   * Additional classpath: `src/main/groovy`\n* Publish JUnit test result report\n   * Test report XMLs: `build/test-results/**/*.xml`\n\nNote that starting with Job DSL 1.60 the \"Additional classpath\" setting is not available when\n[Job DSL script security](https://github.com/jenkinsci/job-dsl-plugin/wiki/Script-Security) is enabled.\n\n## REST API Runner\n\nNote: the REST API Runner does not work with [Automatically Generated DSL](https://github.com/jenkinsci/job-dsl-plugin/wiki/Automatically-Generated-DSL). \n\nA gradle task is configured that can be used to create/update jobs via the Jenkins REST API, if desired. Normally\na seed job is used to keep jobs in sync with the DSL, but this runner might be useful if you'd rather process the\nDSL outside of the Jenkins environment or if you want to create the seed job from a DSL script.\n\n```./gradlew rest -Dpattern=\u003cpattern\u003e -DbaseUrl=\u003cbaseUrl\u003e [-Dusername=\u003cusername\u003e] [-Dpassword=\u003cpassword\u003e]```\n\n* `pattern` - ant-style path pattern of files to include\n* `baseUrl` - base URL of Jenkins server\n* `username` - Jenkins username, if secured\n* `password` - Jenkins password or token, if secured\n\n## Miscellaneous tips\n\nSometimes, you will be looking for a function in the Jenkins\nJob DSL, and it will simply not exist.  DO NOT DESPAIR.\nRead this instead: http://www.devexp.eu/2014/10/26/use-unsupported-jenkins-plugins-with-jenkins-dsl/\n\nIn particular, http://job-dsl.herokuapp.com/ is really helpful, even\nif you're not necessarily working on a custom DSL function.\n\nWhen you do this, you might want to edit the web UI, and then\nsee the Jenkins? Click on \"REST API\" at the bottom of the job\npage and click the link for \"config.xml\", which will give you the\nconfig.xml of the job.  Example: https://ci.pytorch.org/jenkins/job/skeleton-pull-request/config.xml\n\n## Useful Groovy scripts\n\nYou can navigate to https://ci.pytorch.org/jenkins/script and run Groovy scripts to run ad hoc management tasks.\nThis can be very useful for tasks that are tedious to execute manually.\n\n**Beware: with great power comes great responsibility!!**\n\n### Mass removal of stale jobs\n\n```groovy\nimport jenkins.model.*\n  \ndef folder = Jenkins.instance.items.find { job -\u003e\n  job.name == \"caffe2-builds\"\n}\n\ndef jobs = folder.items.findAll { job -\u003e \n  job.name =~ /^caffe2-linux-/\n}\n\njobs.each { job -\u003e \n  println(\"Planning to remove ${job.name}\") \n  //job.delete()\n}\n\nnull\n```\n\n### Listing the number of queued items waiting on what labels\n\n```\ndef map = [:]\n\nJenkins.instance.queue.items.each {\n    i = map.get(it.assignedLabel, 0);\n    map[it.assignedLabel] = i + 1;\n}\n\nsorted = map.sort { a, b -\u003e b.value \u003c=\u003e a.value }\n\nsorted.each { label, count -\u003e\n    println(\"${label}: ${count}\");\n}\n\nprintln \"---\"\n\nJenkins.instance.slaves.each {\n  println \"${it.name} (${it.getComputer().countBusy()}/${it.getNumExecutors()}): ${it.getLabelString()}\"\n}\n\nnull\n```\n\n### Pruning stale queued jobs\n\n```groovy\nimport hudson.model.*\n  \ndef queue = Hudson.instance.queue\n  \ndef cancel = queue.items.findAll {\n  if (it.task.name.startsWith('name-of-job-to-cleanup')) {\n    return true;\n  }\n  return false;\n}\n\ncancel.each {\n  queue.cancel(it.task)\n}\n```\n\n### Developing using IntelliJ\n\nA more pleasant Java development experience can be attained by working\non ossci-job-dsl inside a real Java IDE.  Here's how to set it up using\nIntelliJ:\n\n1. In the opening splash screen, select \"Import Project\"\n2. Select the directory of ossci-job-dsl\n3. Import project from external model: Gradle\n4. Click through the last screen, finishing the import\n5. To test, click \"Run\" and \"Edit configurations\"\n6. Create a new run configuration based on Gradle\n7. Select the current project as the Gradle project, and put \"test\" in\n   Tasks.\n\nYou now have running tests!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Fossci-job-dsl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpytorch%2Fossci-job-dsl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Fossci-job-dsl/lists"}