Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/treasure-data/digdag

Workload Automation System
https://github.com/treasure-data/digdag

digdag

Last synced: 5 days ago
JSON representation

Workload Automation System

Awesome Lists containing this project

README 

        
# Digdag



[![CircleCI](https://dl.circleci.com/status-badge/img/gh/treasure-data/digdag/tree/master.svg?style=svg&circle-token=5a93079551888e4dc43ad75fe6e2bd312153995c)](https://dl.circleci.com/status-badge/redirect/gh/treasure-data/digdag/tree/master)



[![CI](https://github.com/treasure-data/digdag/workflows/CI/badge.svg)](https://github.com/treasure-data/digdag/actions)



## [Documentation](https://docs.digdag.io)



Please check [digdag.io](https://digdag.io) and [docs.digdag.io](https://docs.digdag.io) for installation & user manual.



REST API document is available at [docs.digdag.io/api](http://docs.digdag.io/api/).



## Release Notes



The list of release note is [here](https://github.com/treasure-data/digdag/tree/master/digdag-docs/src/releases).



## Development



### Prerequirements



* JDK 8

* Node.js 12.x



Installing Node.js using nodebrew:



```

$ curl -L git.io/nodebrew | perl - setup

$ echo 'export PATH=$HOME/.nodebrew/current/bin:$PATH' >> ~/.bashrc

$ source ~/.bashrc

$ nodebrew install-binary v12.x

$ nodebrew use v12.x

```



Installing Node.js using Homebrew on Mac OS X:



```

$ brew install node

```



* Python 3

  * sphinx

  * sphinx_rtd_theme

  * recommonmark



### Running tests



```

$ ./gradlew check

```



Test coverage report is generated at `didgag-*/build/reports/jacoco/test/html/index.html`.

Findbugs report is generated at `digdag-*/build/reports/findbugs/main.html`.



```

$ CI_ACCEPTANCE_TEST=true ./gradlew digdag-tests:test --info --tests acceptance.BuiltInVariablesIT

```



To execute tests in digdag-tests subproject locally, `tests` option that is provided by Gradle is useful.

Environment variable `CI_ACCEPTANCE_TEST=true` is needed to execute digdag-tests.



### Testing with PostgreSQL



Test uses in-memory H2 database by default. To use PostgreSQL, set following environment variables:



```

$ export DIGDAG_TEST_POSTGRESQL="$(cat config/test_postgresql.properties)"

```



### Building CLI executables



```

$ ./gradlew cli

$ ./gradlew cli -PwithoutUi  # build without integrated UI

```



(If the command fails during building UI due to errors from `node` command, you can try to add `-PwithoutUi` argument to exclude the UI from the package).



It makes an executable in `pkg/`, e.g. `pkg/digdag-$VERSION.jar`.



### Develop digdag-ui



Node.js development server is useful because it reloads changes of digdag-ui source code automatically.



First, put following lines to ~/.config/digdag/config and start digdag server:



```

server.http.headers.access-control-allow-origin = http://localhost:9000

server.http.headers.access-control-allow-headers = origin, content-type, accept, authorization, x-td-account-override, x-xsrf-token, cookie

server.http.headers.access-control-allow-credentials = true

server.http.headers.access-control-allow-methods = GET, POST, PUT, DELETE, OPTIONS, HEAD

server.http.headers.access-control-max-age = 1209600

```



Then, start digdag-ui development server:



```

$ cd digdag-ui/

$ npm install

$ npm run dev    # starts dev server on http://localhost:9000/

```



### Updating REST API document



Run this command to update REST API document file at digdag-docs/src/api/swagger.yaml.



```

./gradlew swaggerYaml  # dump swagger.yaml file

```



Use `--enable-swagger` option to check the current Digdag REST API.



```

$ ./gradlew cli

$ ./pkg/digdag-.jar server --memory --enable-swagger # Run server with --enable-swagger option



$ docker run -dp 8080:8080 swaggerapi/swagger-ui # Run Swagger-UI on different console

$ open http://localhost:8080/?url=http://localhost:65432/api/swagger.json # Open api/swagger.json on Swagger-UI

```



### Updating documents



Documents are in digdag-docs/src directory. They're built using Sphinx.



Website is hosted on [www.digdag.io](http://www.digdag.io) using Github Pages. Pages are built using deployment step of circle.yml and automatically pushed to [gh-pages branch of digdag-docs repository](https://github.com/treasure-data/digdag-docs/tree/gh-pages).



To build the pages and check them locally, follow this instruction.



Create a virtual environment of Python and install dependent Python libraries including Sphinx.



```

$ python3 -m venv .venv

$ source .venv/bin/activate

(.venv)$ pip install -r digdag-docs/requirements.txt -c digdag-docs/constraints.txt

```



After installation of Python libraries, You can build with running the following command:



```

(.venv)$ ./gradlew site

```



This might not always update all necessary files (Sphinx doesn't manage update dependencies well). In this case, run `./gradlew clean` first.



It builds index.html at digdag-docs/build/html/index.html.



### Development on IDEs



#### IntelliJ IDEA



Digdag is using a Java annotation processor `org.immutables:value.` The combination of Java annotation processing and Gradle on IntelliJ IDEA sometimes introduces some troubles. In Digdag's case, you may run into some compile errors like `cannot find symbol: class ImmutableRestWorkflowDefinitionCollection.`

So we'd recommend the followings to avoid those compile errors if you want to develop Digdag one the IDE.



1. There's an important configuration option to be enabled to fully have IntelliJ be fully integrated with an existing gradle build configuration: `Delegate IDE build/run actions to gradle` needs to be enabled.



![](https://user-images.githubusercontent.com/17990895/48221255-9706be80-e35f-11e8-8283-1ca6d713e31c.png)



## Releasing a new version

This is for committers only.

### Prerequisite: Sonatype OSSRH

You need an account in Sonatype OSSRH, and configure it in your `~/.gradle/gradle.properties`.



ossrhUsername=(your Sonatype OSSRH username)

ossrhPassword=(your Sonatype OSSRH password)



### Prerequisite: PGP signatures

You need your PGP signatures to release artifacts into Maven Central, and configure Gradle to use your key to sign.

Configure it in your `~/.gradle/gradle.properties`.



```

signing.gnupg.executable=gpg

signing.gnupg.useLegacyGpg=false

signing.gnupg.keyName=(the last 8 symbols of your keyId)

signing.gnupg.passphrase=(the passphrase used to protect your private key)

```



### Release procedure



As mentioned in the prerequirements, we need to build with JDK 8 in this procedure.



1. run `git pull upstream master --tags`.

1. run `./gradlew setVersion -Pto=` command.

1. write release notes to `releases/release-.rst` file. It must include at least version (the first line) and release date (the last line).

1. run `./gradlew clean cli site check releaseCheck`.

1. make a release branch. `git checkout -b release_v` and commit.

1. push the release branch to origin and create a PR.

1. after the PR is merged to master, checkout master and pull latest upstream/master.

1. run `./gradlew clean cli site check releaseCheck` again.

1. if it succeeded, run `./gradlew release`.

1. create a tag `git tag -a v` and push `git push upstream v`

1. create a release in [GitHub releases](https://github.com/treasure-data/digdag/releases).

1. upload `pkg/digdag-.jar` to the release

1. a few minutes later, run `digdag selfupdate` and confirm the version.



If major version is incremented, also update `version =` and `release =` at [digdag-docs/src/conf.py](digdag-docs/src/conf.py).



If you are expert, skip 5. to 7. and directly update master branch.



### Post-process of new release



You also need following steps after new version has been released.



1. create next snapshot version, run `./gradlew setVersion -Pto=-SNAPSHOT`.

1. push to master.



### Releasing a SNAPSHOT version



```

./gradlew releaseSnapshot

```

**Note**

Snapshot release is not supported currently.