{"id":23201648,"url":"https://github.com/mitodl/odl_devstack_tools","last_synced_at":"2026-05-02T19:32:01.475Z","repository":{"id":145114267,"uuid":"129302634","full_name":"mitodl/odl_devstack_tools","owner":"mitodl","description":"Helpful tools for running edX's docker-based devstack","archived":false,"fork":false,"pushed_at":"2023-10-20T13:38:10.000Z","size":30,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-05T09:41:48.872Z","etag":null,"topics":["devstack","docker","edx"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mitodl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-12T19:40:05.000Z","updated_at":"2024-04-26T15:38:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"c09051c6-48ed-450b-8438-cb2ec98a5812","html_url":"https://github.com/mitodl/odl_devstack_tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mitodl/odl_devstack_tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fodl_devstack_tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fodl_devstack_tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fodl_devstack_tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fodl_devstack_tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mitodl","download_url":"https://codeload.github.com/mitodl/odl_devstack_tools/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fodl_devstack_tools/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32547645,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T19:18:06.202Z","status":"ssl_error","status_checked_at":"2026-05-02T19:16:21.335Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devstack","docker","edx"],"created_at":"2024-12-18T15:16:26.179Z","updated_at":"2026-05-02T19:32:01.459Z","avatar_url":"https://github.com/mitodl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"**⚠️ WARNING (12/3/2021) ⚠️** –  _This project is not set up to work with the `maple` release of `edx-platform`, and possibly some earlier releases. If no one has the time to add support for later releases to this project, it's best to set up your local devstack the old-fashioned way (running a shell in LMS/Studio, then changing config settings and installing packages in there)_\n\n# ODL Devstack Tools\n\nThis repo provides some helpful tools for configuring and running [devstack](https://github.com/edx/devstack) (edX's Docker-based solution for running Open edX services).\n\n### Features\n\n- Create a custom docker image (which is just a layer on top of edX's devstack image) with some helpful packages \n  already installed (e.g.: `pdbpp`)\n- Automatically apply changes to the JSON config files in devstack when running the containers\n  (`lms.yml` or `studio.yml` for Juniper or later; `lms.env.json`, `cms.env.json`, `lms.auth.json`, or `cms.auth.json`\n  for every release before Juniper).\n- Via docker-compose configuration, specify local repos that you want mounted \n  into the devstack containers and installed. \n  This is useful for testing changes in repos that devstack depends on (e.g.: `XBlock`, \n  `xblock-utils`, `edx-sga`). If the packages use setuptools, they'll be installed via pip with the `-e` flag, so the \n  server will restart automatically when you make changes to those packages locally.\n- Create any number of docker-compose files to take advantage of the features above for different\n  purposes (e.g.: one compose file for testing changes to `XBlock`, another one for testing changes to `edx-sga`, \n  another one for testing some LMS feature that requires a bunch of config changes). You can also combine them as \n  desired.\n- If desired, sets up your container to enable exporting courses to Github (specifically the \n  github.mit.edu private repos) so you don't need to create SSH keys and add them to Github\n  every single time you restart your containers. \n \n\n### Usage\n\n##### 1) Set environment variables\n \nThese environment variables will need to be set in your host machine (they can be added to `~/.bash_profile`, et. al.):\n\n```bash\n# The name of the custom devstack image that will be built as a layer on top of the devstack image.\nCUSTOM_DEVSTACK_IMG_NAME=\"edxops/edxapp:odlcustom\"\n# The name of the existing devstack image on top of which your custom image will be based. Defaults to 'edxops/edxapp:latest'\nCUSTOM_DEVSTACK_BASE_IMG=\"edxops/edxapp:latest\"\n# Path to this repo on your machine.\nCUSTOM_DEVSTACK_PATH=\"/path/to/odl_devstack_tools\"\n\n# The path to helper files in the container. ***Do not change this value***\nDEVSTACK_CONTAINER_HELPER_DIR=\"/edx/app/edxapp/helper\"\n# The path to the directory in the container where local repos will be mounted. ***Do not change this value***\nDEVSTACK_CONTAINER_MOUNT_DIR=\"/edx/app/edxapp/venvs/edxapp/src\"\n```\n\n##### 2) Build custom image\n\nWhen you have the latest images from edX (via `make dev.pull` - more details \n[here](https://github.com/edx/devstack#using-the-latest-images)), you can run the following command to \ncreate the new image based on edX's image.\n \n```bash\ndocker build $CUSTOM_DEVSTACK_PATH -t $CUSTOM_DEVSTACK_IMG_NAME --build-arg BASE_IMG=$CUSTOM_DEVSTACK_BASE_IMG --no-cache\n```\n\n##### 3) Run devstack with the added compose file(s)\n\nTo spin up the containers using the custom image and take advantage of the features described above,\nrun `docker-compose up` like the commands below. For the sake of familiarity and consistency, these \ncommands mimic the ['make dev.up' command in devstack](https://github.com/edx/devstack/blob/master/Makefile).\n\n```bash\n# Run LMS with the basic custom docker-compose file\ndocker-compose -f docker-compose.yml -f docker-compose-host.yml -f $CUSTOM_DEVSTACK_PATH/docker-compose-custom.yml up -d lms\n# Run LMS with the basic custom docker-compose file and an additional one that you created\ndocker-compose -f docker-compose.yml -f docker-compose-host.yml \\\n  -f $CUSTOM_DEVSTACK_PATH/docker-compose-custom.yml -f $CUSTOM_DEVSTACK_PATH/docker-compose-mine.yml up -d lms\n```\n\n### Applying config changes\n\n##### Summary\n\nChanges to YML/JSON config values can be automatically applied when the container starts by doing the following:\n\n- Create a JSON patch file in this repo's `./configpatch` directory _(NOTE: The files are written as JSON patches, but \n  they will work as expected with YAML config files)_\n- In your custom docker-compose file, mount the JSON patch file into the \n  `${DEVSTACK_CONTAINER_HELPER_DIR}/configpatch/` directory in the container\n\n##### Detail\n\nYou can automatically apply changes to any of the devstack JSON config files \n(`lms.yml` or `studio.yml` for Juniper or later; `lms.env.json`, `cms.env.json`, `lms.auth.json`, or `cms.auth.json`\nfor every release before Juniper) by creating a patch file in the `./configpatch` directory. \nThese patch files should be `.json` type and in \n[jsonpatch](http://jsonpatch.com/) format. The file name doesn't matter. See the `.example` files in the `./configpatch/` \ndirectory for example usage.\n\nAfter the patch file is created, the patch file needs to be mounted into the container by adding to the `volume`\nsection of your docker-compose file.\n\n```yml\nservices:\n  lms:\n    volumes:\n      - ${CUSTOM_DEVSTACK_PATH}/configpatch/my_patch.json:${DEVSTACK_CONTAINER_HELPER_DIR}/configpatch/my_patch.json\n```\n\n### Specifying local repos to be mounted and installed automatically\n\n##### Summary\n\nLocal repos can be mounted and installed in devstack containers by doing the following:\n\n- In your docker-compose file, add a environment variable with a name that starts with `ADDED_REQ_`, with the value\n  being the package name (e.g.: `ADDED_REQ_XBLOCK=XBlock`) \n- In your docker-compose file, mount your local repo directory alongside the `edx-platform` virtualenv packages \n  (e.g.: /path/to/repo/XBlock:${DEVSTACK_CONTAINER_MOUNT_DIR}/XBlock)\n\n##### Detail\n\nWhen hacking on or testing changes to some package that edX depends on, it's very helpful to mount and install your \nlocal repo into the devstack containers. Among the benefits: (1) changes you make to those repos trigger a server\nrestart automatically for LMS/Studio (if your package can be installed as [\"editable\"](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)); \n(2) when you create a migration for one of those repos in the devstack container, \nthe migration files are immediately available in your local repo.\n\nRunning `docker-compose up` with the example docker-compose file below would automatically mount the `XBlock` repo \nand install it as a local package in the `lms` container.\n   \n```yml\nservices:\n  lms:\n    environment:\n      - ADDED_REQ_XBLOCK=XBlock\n    volumes:\n      - /path/to/repo/XBlock:${DEVSTACK_CONTAINER_MOUNT_DIR}/XBlock\n```\n\n\n### Enabling course content exporting to Github in Studio\n\nIf you want to be able to export course content to github.mit.edu, follow these steps:\n\n1. On your **host machine**, generate an SSH key ([Github guide](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#generating-a-new-ssh-key))\n  and move the keys (`id_rsa` and `id_rsa.pub`) to `path/to/odl_devstack_tools/ssh`.\n  **NOTE**: Use your github.mit.edu email address and NO PASSPHRASE.\n1. Add that SSH key to your Github account ([guide](https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/)).\n1. Apply all the necessary JSON settings values and advanced course settings related\n  to course content exporting.\n\n### Extras\n\nIt's helpful to have a shorthand for running the containers using certain compose files explicitly or by default.\nThe variables and bash function below provide that.  \n\n```bash\nexport USE_CUSTOM_DEVSTACK=true\nexport DEFAULT_CUSTOM_DEVSTACK_COMPOSE_FILE=\"docker-compose-custom.yml\"\n\nfunction dedxup() {\n  local compose_file_args=()\n  \n  # Add custom compose file(s) to docker-compose params if it's set and the file exists\n  if [ \"$USE_CUSTOM_DEVSTACK\" = true ]; then\n    compose_file_args+=( '-f' \"$CUSTOM_DEVSTACK_PATH/$DEFAULT_CUSTOM_DEVSTACK_COMPOSE_FILE\" )\n    if [ ! -z $ADDED_DEVSTACK_COMPOSE_FILE ]; then\n      compose_file_args+=( '-f' \"$CUSTOM_DEVSTACK_PATH/$ADDED_DEVSTACK_COMPOSE_FILE\" )\n    fi\n    for compose_file_arg in \"${compose_file_args[@]}\"\n    do\n      if [ $compose_file_arg != '-f' ]; then\n        echo -e \"Using additional compose file (\\033[1;92m$compose_file_arg\\e[0m) ...\"\n      fi\n    done\n    echo ''\n  fi\n  \n  docker-compose -f docker-compose.yml -f docker-compose-host.yml ${compose_file_args[@]} up -d $@\n}\n```\n\nExample usages:\n```bash\n# Run LMS using the default custom compose file (docker-compose-custom.yml)\ndedxup lms\n# Run LMS without any custom compose files\nUSE_CUSTOM_DEVSTACK=false dedxup lms\n# Run LMS using the default custom compose file AND an additional custom compose file\nADDED_DEVSTACK_COMPOSE_FILE='docker-compose-xblock-dev.yml' dedxup lms\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitodl%2Fodl_devstack_tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmitodl%2Fodl_devstack_tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitodl%2Fodl_devstack_tools/lists"}