{"id":13588816,"url":"https://github.com/DataBiosphere/job-manager","last_synced_at":"2025-04-08T06:32:50.763Z","repository":{"id":27261489,"uuid":"101917710","full_name":"DataBiosphere/job-manager","owner":"DataBiosphere","description":"Job Manager API and UI for interacting with asynchronous batch jobs and workflows.","archived":false,"fork":false,"pushed_at":"2023-10-04T16:58:53.000Z","size":3480,"stargazers_count":25,"open_issues_count":133,"forks_count":5,"subscribers_count":30,"default_branch":"master","last_synced_at":"2024-04-14T13:18:17.236Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-30T18:51:22.000Z","updated_at":"2024-04-29T18:57:11.249Z","dependencies_parsed_at":"2024-04-29T19:07:11.120Z","dependency_job_id":null,"html_url":"https://github.com/DataBiosphere/job-manager","commit_stats":null,"previous_names":[],"tags_count":60,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fjob-manager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fjob-manager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fjob-manager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fjob-manager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/job-manager/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247792202,"owners_count":20996878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:06:57.398Z","updated_at":"2025-04-08T06:32:45.748Z","avatar_url":"https://github.com/DataBiosphere.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# Job Manager\n\nIf you're looking for basics for Terra developers, see [here](TERRA_QUICKSTART.md).\n\n[![CircleCI branch](https://img.shields.io/circleci/project/github/DataBiosphere/job-manager/master.svg?label=Tests%20on%20Circle%20CI\u0026style=flat-square)](https://circleci.com/gh/DataBiosphere/job-manager/tree/master)\n![Github](https://img.shields.io/badge/Supported%20Backends-cromwell-blue.svg?\u0026style=flat-square)\n![GitHub](https://img.shields.io/github/license/DataBiosphere/job-manager.svg?color=orange\u0026style=flat-square)\n[![GitHub release](https://img.shields.io/github/release/DataBiosphere/job-manager.svg?label=Latest%20Release\u0026style=flat-square\u0026colorB=green)](https://github.com/DataBiosphere/job-manager/releases)\n[![Github](https://img.shields.io/badge/Docker%20Image-UI-blue.svg?style=flat-square)](https://console.cloud.google.com/gcr/images/broad-dsp-gcr-public/US/job-manager-ui)\n[![Github](https://img.shields.io/badge/Docker%20Image-cromwell%20shim%20API-blue.svg?style=flat-square)](https://console.cloud.google.com/gcr/images/broad-dsp-gcr-public/US/job-manager-api-cromwell)\n\n## Lifecycle notice\n\nJob Manager is in maintenance mode and is not recommended for new projects.\n\ndsub support is deprecated and has been removed.\n\nThe last release of Job Manager to support dsub without compromise is [1.5.7](https://github.com/DataBiosphere/job-manager/releases/tag/v1.5.7).\n\n## User facing documentation\n\nWelcome to the Job Manager repository! If you're a developer you're in the right place.\n\nHowever, if you just want to try out or deploy Job Manager, you will probably find our user and deployment focused content in\nour ReadTheDocs pages: https://data-biosphere-job-manager.readthedocs.io/en/latest/\n\n## Welcome\n\nSee the [development guide](#development) below.\n\nThe Job Manager is an API and UI for monitoring and managing jobs in a backend execution engine.\n\nThe Broad, Verily, and many other organizations in the life sciences execute enormous numbers of scientific workflows and need to manage those operations. Job Manager was born out of the experiences of producing data for some of the world’s largest sequencing projects such as The Cancer Genome Atlas, Baseline, and the Thousand Genomes Project.\n\nThe Job Manager aspires to bring ease and efficiency to developing and debugging workflows while seamlessly scaling to production operations management.\n\n## Key Features\n* Supports visualization over [Cromwell](https://github.com/broadinstitute/cromwell) backend\n* Service provider interface can be extended to support other engines\n* Rich search capabilities across current and historic workflows\n* Aborting workflows\n* Clean, intuitive UX based on material design principles\n\n## Architecture Overview\n\nThe Job Manager [defines an API](api/jobs.yaml) via OpenAPI. An Angular2 UI is provided over the autogenerated Typescript bindings for this API. The UI is configurable at compilation time to support various deployment environments (see [environment.ts](ui/src/environments/environment.ts)), including auth, cloud projects, and label columns.\n\nThe UI must be deployed along with a backend implementation of the API, two such implementations are provided here:\n\n### Cromwell\nMonitors jobs launched by the [Cromwell workflow engine](https://github.com/broadinstitute/cromwell). The Python Flask wrapper was created using Swagger Codegen and can be configured to pull data from a specific Cromwell instance. _At this time, to utilize all job manager features, please consider using Cromwell v32 or newer._\n\n## Development\n\n### Prerequisites\n\n- Install docker and docker-compose\n- Check out the repository and navigate to the directory:\n  ```sh\n    git clone https://github.com/DataBiosphere/job-manager.git\n    cd job-manager\n  ```\n- Setup git-secrets on the repository:\n  - On Mac:\n  ```\n  brew install git-secrets\n  ```\n\n  - On Linux:\n  ```\n  rm -rf git-secrets\n  git clone https://github.com/awslabs/git-secrets.git\n  cd git-secrets\n  sudo make install \u0026\u0026 sudo chmod o+rx /usr/local/bin/git-secrets\n  cd ..\n  rm -rf git-secrets\n  ```\n\n- Configure the `git secrets` hook:\n  ```sh\n    git secrets --install\n  ```\n\n### Server Setup\n\n#### Cromwell\n\n- Link your preferred backend docker compose file as `docker-compose.yml`:\n\n  - Cromwell (local): `ln -sf cromwell-instance-compose.yml docker-compose.yml`\n  - Cromwell (CaaS): `ln -sf cromwell-caas-compose.yml docker-compose.yml`\n- Follow [servers/cromwell](servers/cromwell/README.md#Development) for Cromwell server setup then return here to continue.\n\n\n### Run Locally\n- Run `docker-compose up` from the root of the repository:\n  - If this is the first time running `docker-compose up`  this might take a few minutes.\n  - Eventually you should see a compilation success message like this: \n  ```\n  jmui_1        | webpack: Compiled successfully.\n  ```\n- Make sure that your Cromwell backend is ready to receive query requests. \n- Navigate to http://localhost:4200.\n\n#### Notes\n1. Websocket reload on code change does not work in docker-compose (see\nhttps://github.com/angular/angular-cli/issues/6349).\n2. Changes to `package.json`, `requirements.txt`, or `constraints.txt` [regenerating the API](#updating-the-api-using-swagger-codegen) require a rebuild with:\n  ```\n  docker-compose up --build\n  ```\n  Alternatively, rebuild a single component:\n  ```\n  docker-compose build ui\n  ```\n3. As of 03/31/2023 you'll need to register a `redirectURI` for your application if you've enabled Google OAuth for your JM instance. This is a result of Google deprecating the `Google Sign-In JavaScript Platform Library`.\n  * To do so, log in to Google Cloud Platform and go to the `Credentials` page under the `API \u0026 Services` section of the sidebar.\n  * Under `OAuth 2.0 Client IDs`, find the ID used for your JM domain and click the name to go to the edit page.\n  * Under `Authorized redirect URIs`, add the new `redirectURI` as `https://{ORIGIN_NAME}/redirect-from-oauth` (make sure to click the save button after).\n    * Example URI: `https://test-domain.org/redirect-from-oauth`\n    * You'll need to repeat this process if you use a different client ID for each of your domains/environments.\n  * **NOTE**: As a result of the Platform Library deprecation Google is no longer keeping track of active sessions. The authorization flow JM supports (known as the implicit flow) will provide an access_token that lasts for an hour.\n    * Upon expiration, the applciation will redirect users to the Sign In page again. Simply log back in and you'll be redirected to the page you were previously viewing.\n\n### Updating the API using swagger-codegen\n\n* We use [swagger-codegen](https://github.com/swagger-api/swagger-codegen) to transform the API \ndefined in `api/jobs.yaml` into appropriate classes for the servers and the UI to use.\n* Whenever the API is updated, run this to trigger a rebuild: \n```sh\ndocker-compose up rebuild-swagger\n```\n \n#### Swagger codegen notes\n\n* The `rebuild-swagger` job does nothing if the file `api/jobs.yaml` has not changed since the\nlast time it was run.\n* The `rebuild-swagger` job will run by default during `docker-compose up` to generate the swagger for the other \nservices if necessary. The other services will not start until their swagger classes exist.\n* After regenerating the model files, you'll need to test and update the server implementations to \nresolve any broken dependencies on old API definitions or implement additional functionality to match the new specs. \n\n## Job Manager UI Server\nFor UI server documentation, see [ui](ui/).\n\n## Job Manager `cromwell` Server\nFor `cromwell` server documentation, see [servers/cromwell](servers/cromwell/README.md).\n\n## Build docker images and releases\n\n### How to build\n\nStarting with release v1.6.0, Job Manager docker images are on [GCR](https://console.cloud.google.com/gcr/images/broad-dsp-gcr-public).\n\n- Configure Docker to authenticate with GCR\n    ```\n    gcloud config set account username@broadinstitute.org\n    gcloud auth configure-docker us.gcr.io\n    ```\n\n- Set the Docker tag first in bash, e.g. `TAG=\"v0.1.0\"`\n\n- To publish the `job-manager-ui` image with `$TAG` from the root of this Github repository:\n    ```\n    docker build -t us.gcr.io/broad-dsp-gcr-public/job-manager-ui:$TAG . -f ui/Dockerfile\n    docker push us.gcr.io/broad-dsp-gcr-public/job-manager-ui:$TAG\n    ```\n    \n- To publish the `job-manager-api-cromwell` image with `$TAG` from the root of this Github repository:\n    ```\n    docker build -t us.gcr.io/broad-dsp-gcr-public/job-manager-api-cromwell:$TAG . -f servers/cromwell/Dockerfile\n    docker push us.gcr.io/broad-dsp-gcr-public/job-manager-api-cromwell:$TAG\n    ```\n\n### Add a Github release pointing to the GCR images\n\nFrom v1.6.0, each release in Github will also release 2 corresponding Docker images on GCR:\n\n- [job-manager-ui](https://console.cloud.google.com/gcr/images/broad-dsp-gcr-public/US/job-manager-ui)\n- [job-manager-api-cromwell](https://console.cloud.google.com/gcr/images/broad-dsp-gcr-public/US/job-manager-api-cromwell)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDataBiosphere%2Fjob-manager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDataBiosphere%2Fjob-manager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDataBiosphere%2Fjob-manager/lists"}