{"id":13582772,"url":"https://github.com/treasure-data/digdag","last_synced_at":"2025-05-15T02:07:14.219Z","repository":{"id":9179927,"uuid":"42751387","full_name":"treasure-data/digdag","owner":"treasure-data","description":"Workload Automation System","archived":false,"fork":false,"pushed_at":"2024-08-27T02:42:56.000Z","size":84997,"stargazers_count":1316,"open_issues_count":120,"forks_count":229,"subscribers_count":128,"default_branch":"master","last_synced_at":"2025-04-10T10:08:06.792Z","etag":null,"topics":["digdag"],"latest_commit_sha":null,"homepage":"https://www.digdag.io/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/treasure-data.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-18T23:18:08.000Z","updated_at":"2025-03-07T21:36:46.000Z","dependencies_parsed_at":"2023-01-16T21:01:01.123Z","dependency_job_id":"ec16c731-6105-4f07-9b9a-1857e952807b","html_url":"https://github.com/treasure-data/digdag","commit_stats":null,"previous_names":[],"tags_count":111,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fdigdag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fdigdag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fdigdag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fdigdag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/treasure-data","download_url":"https://codeload.github.com/treasure-data/digdag/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254259383,"owners_count":22040820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["digdag"],"created_at":"2024-08-01T15:03:00.851Z","updated_at":"2025-05-15T02:07:14.137Z","avatar_url":"https://github.com/treasure-data.png","language":"Java","funding_links":[],"categories":["Java","others"],"sub_categories":[],"readme":"# Digdag\n\n[![CircleCI](https://dl.circleci.com/status-badge/img/gh/treasure-data/digdag/tree/master.svg?style=svg\u0026circle-token=5a93079551888e4dc43ad75fe6e2bd312153995c)](https://dl.circleci.com/status-badge/redirect/gh/treasure-data/digdag/tree/master)\n\n[![CI](https://github.com/treasure-data/digdag/workflows/CI/badge.svg)](https://github.com/treasure-data/digdag/actions)\n\n\n## [Documentation](https://docs.digdag.io)\n\nPlease check [digdag.io](https://digdag.io) and [docs.digdag.io](https://docs.digdag.io) for installation \u0026 user manual.\n\nREST API document is available at [docs.digdag.io/api](http://docs.digdag.io/api/).\n\n## Release Notes\n\nThe list of release note is [here](https://github.com/treasure-data/digdag/tree/master/digdag-docs/src/releases).\n\n\n## Development\n\n### Prerequirements\n\n* JDK 8\n* Node.js 12.x\n\nInstalling Node.js using nodebrew:\n\n```\n$ curl -L git.io/nodebrew | perl - setup\n$ echo 'export PATH=$HOME/.nodebrew/current/bin:$PATH' \u003e\u003e ~/.bashrc\n$ source ~/.bashrc\n$ nodebrew install-binary v12.x\n$ nodebrew use v12.x\n```\n\nInstalling Node.js using Homebrew on Mac OS X:\n\n```\n$ brew install node\n```\n\n* Python 3\n  * sphinx\n  * sphinx_rtd_theme\n  * recommonmark\n\n### Running tests\n\n```\n$ ./gradlew check\n```\n\nTest coverage report is generated at `didgag-*/build/reports/jacoco/test/html/index.html`.\nFindbugs report is generated at `digdag-*/build/reports/findbugs/main.html`.\n\n```\n$ CI_ACCEPTANCE_TEST=true ./gradlew digdag-tests:test --info --tests acceptance.BuiltInVariablesIT\n```\n\nTo execute tests in digdag-tests subproject locally, `tests` option that is provided by Gradle is useful.\nEnvironment variable `CI_ACCEPTANCE_TEST=true` is needed to execute digdag-tests.\n\n### Testing with PostgreSQL\n\nTest uses in-memory H2 database by default. To use PostgreSQL, set following environment variables:\n\n```\n$ export DIGDAG_TEST_POSTGRESQL=\"$(cat config/test_postgresql.properties)\"\n```\n\n### Building CLI executables\n\n```\n$ ./gradlew cli\n$ ./gradlew cli -PwithoutUi  # build without integrated UI\n```\n\n(If the command fails during building UI due to errors from `node` command, you can try to add `-PwithoutUi` argument to exclude the UI from the package).\n\nIt makes an executable in `pkg/`, e.g. `pkg/digdag-$VERSION.jar`.\n\n### Develop digdag-ui\n\nNode.js development server is useful because it reloads changes of digdag-ui source code automatically.\n\nFirst, put following lines to ~/.config/digdag/config and start digdag server:\n\n```\nserver.http.headers.access-control-allow-origin = http://localhost:9000\nserver.http.headers.access-control-allow-headers = origin, content-type, accept, authorization, x-td-account-override, x-xsrf-token, cookie\nserver.http.headers.access-control-allow-credentials = true\nserver.http.headers.access-control-allow-methods = GET, POST, PUT, DELETE, OPTIONS, HEAD\nserver.http.headers.access-control-max-age = 1209600\n```\n\nThen, start digdag-ui development server:\n\n```\n$ cd digdag-ui/\n$ npm install\n$ npm run dev    # starts dev server on http://localhost:9000/\n```\n\n### Updating REST API document\n\nRun this command to update REST API document file at digdag-docs/src/api/swagger.yaml.\n\n```\n./gradlew swaggerYaml  # dump swagger.yaml file\n```\n\nUse `--enable-swagger` option to check the current Digdag REST API.\n\n```\n$ ./gradlew cli\n$ ./pkg/digdag-\u003ccurrent version\u003e.jar server --memory --enable-swagger # Run server with --enable-swagger option\n\n$ docker run -dp 8080:8080 swaggerapi/swagger-ui # Run Swagger-UI on different console\n$ open http://localhost:8080/?url=http://localhost:65432/api/swagger.json # Open api/swagger.json on Swagger-UI\n```\n\n### Updating documents\n\nDocuments are in digdag-docs/src directory. They're built using Sphinx.\n\nWebsite is hosted on [www.digdag.io](http://www.digdag.io) using Github Pages. Pages are built using deployment step of circle.yml and automatically pushed to [gh-pages branch of digdag-docs repository](https://github.com/treasure-data/digdag-docs/tree/gh-pages).\n\nTo build the pages and check them locally, follow this instruction.\n\nCreate a virtual environment of Python and install dependent Python libraries including Sphinx.\n\n```\n$ python3 -m venv .venv\n$ source .venv/bin/activate\n(.venv)$ pip install -r digdag-docs/requirements.txt -c digdag-docs/constraints.txt\n```\n\nAfter installation of Python libraries, You can build with running the following command:\n\n```\n(.venv)$ ./gradlew site\n```\n\nThis might not always update all necessary files (Sphinx doesn't manage update dependencies well). In this case, run `./gradlew clean` first.\n\nIt builds index.html at digdag-docs/build/html/index.html.\n\n### Development on IDEs\n\n#### IntelliJ IDEA\n\nDigdag is using a Java annotation processor `org.immutables:value.` The combination of Java annotation processing and Gradle on IntelliJ IDEA sometimes introduces some troubles. In Digdag's case, you may run into some compile errors like `cannot find symbol: class ImmutableRestWorkflowDefinitionCollection.`\nSo we'd recommend the followings to avoid those compile errors if you want to develop Digdag one the IDE.\n\n1. There's an important configuration option to be enabled to fully have IntelliJ be fully integrated with an existing gradle build configuration: `Delegate IDE build/run actions to gradle` needs to be enabled.\n\n![](https://user-images.githubusercontent.com/17990895/48221255-9706be80-e35f-11e8-8283-1ca6d713e31c.png)\n\n## Releasing a new version\nThis is for committers only.\n### Prerequisite: Sonatype OSSRH\nYou need an account in Sonatype OSSRH, and configure it in your `~/.gradle/gradle.properties`.\n\nossrhUsername=(your Sonatype OSSRH username)\nossrhPassword=(your Sonatype OSSRH password)\n\n### Prerequisite: PGP signatures\nYou need your PGP signatures to release artifacts into Maven Central, and configure Gradle to use your key to sign.\nConfigure it in your `~/.gradle/gradle.properties`.\n\n```\nsigning.gnupg.executable=gpg\nsigning.gnupg.useLegacyGpg=false\nsigning.gnupg.keyName=(the last 8 symbols of your keyId)\nsigning.gnupg.passphrase=(the passphrase used to protect your private key)\n```\n\n### Release procedure\n\nAs mentioned in the prerequirements, we need to build with JDK 8 in this procedure.\n\n1. run `git pull upstream master --tags`.\n1. run `./gradlew setVersion -Pto=\u003cversion\u003e` command.\n1. write release notes to `releases/release-\u003cversion\u003e.rst` file. It must include at least version (the first line) and release date (the last line).\n1. run `./gradlew clean cli site check releaseCheck`.\n1. make a release branch. `git checkout -b release_v\u003cversion\u003e` and commit.\n1. push the release branch to origin and create a PR.\n1. after the PR is merged to master, checkout master and pull latest upstream/master.\n1. run `./gradlew clean cli site check releaseCheck` again.\n1. if it succeeded, run `./gradlew release`.\n1. create a tag `git tag -a v\u003cversion\u003e` and push `git push upstream v\u003cversion\u003e`\n1. create a release in [GitHub releases](https://github.com/treasure-data/digdag/releases).\n1. upload `pkg/digdag-\u003cversion\u003e.jar` to the release\n1. a few minutes later, run `digdag selfupdate` and confirm the version.\n\nIf major version is incremented, also update `version =` and `release =` at [digdag-docs/src/conf.py](digdag-docs/src/conf.py).\n\nIf you are expert, skip 5. to 7. and directly update master branch.\n\n### Post-process of new release\n\nYou also need following steps after new version has been released.\n\n1. create next snapshot version, run `./gradlew setVersion -Pto=\u003cnext-version\u003e-SNAPSHOT`.\n1. push to master.\n\n### Releasing a SNAPSHOT version\n\n```\n./gradlew releaseSnapshot\n```\n**Note**\nSnapshot release is not supported currently.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fdigdag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftreasure-data%2Fdigdag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fdigdag/lists"}