https://github.com/axiom-data-science/docker-luigi
🐍 🐳 Luigi in Docker - alpine and ubuntu images available
https://github.com/axiom-data-science/docker-luigi
docker luigi python scheduling
Last synced: 7 months ago
JSON representation
🐍 🐳 Luigi in Docker - alpine and ubuntu images available
- Host: GitHub
- URL: https://github.com/axiom-data-science/docker-luigi
- Owner: axiom-data-science
- Created: 2015-11-23T17:51:44.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2021-05-07T03:47:52.000Z (over 4 years ago)
- Last Synced: 2024-10-26T23:55:25.527Z (12 months ago)
- Topics: docker, luigi, python, scheduling
- Language: Dockerfile
- Homepage:
- Size: 79.1 KB
- Stars: 50
- Watchers: 6
- Forks: 20
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Luigi on Docker
In addition to supporting the `sqlite` task history backend, this container includes all of the dependencies to use the `mysql` and `postgresql` backends.
## Versions
[All available images](https://hub.docker.com/r/axiom/docker-luigi/tags)
### alpine
Uses `pip` to install `luigi` and dependencies
* `axiom/docker-luigi:latest-alpine` (2.8.13)
* `axiom/docker-luigi:{version}-alpine`
### baseimage (ubuntu xenial)
Uses `conda` to install `luigi` and dependencies
* `axiom/docker-luigi:latest` (2.8.13)
* `axiom/docker-luigi:{version}`
## Exposed Volumes
#### `/etc/luigi/`
Both `luigi` and `luigid` load their configuration from `/etc/luigi/`.
* Configuration: `/etc/luigi/luigi.conf`
* Logging: `/etc/luigi/logging.conf`
Mount a directory containing a `luigi.conf` and `logging.conf` file(s) to
`/etc/luigi` to provide your own configuration(s).
```
docker run \
-v /your/directory:/etc/luigi \
axiom/docker-luigi
```
Or mount a single configuration file:
```
docker run \
-v /your/directory/logging.conf:/etc/luigi/logging.conf \
axiom/docker-luigi
```
The default can be found in the `luigi.conf` and `logging.conf` files in this
repository. Be aware that these specify the paths to the logging configration
and the state persistence database. If you change these values in `luigi.conf`
the examples in this document will not work!
#### `/luigi/state`
Mount a volume at `/luigi/state` for the `luigid` **state to be persisted**
between restarts. Example below uses a named docker volume to persist the state:
```
docker run \
-v luigistate:/luigi/state \
axiom/docker-luigi
```
### Testing
```bash
docker-compose -f tests/docker-compose.yml up -d maria pgsql
# Wait ~10s for databases to start
docker-compose -f tests/docker-compose.yml build
docker-compose -f tests/docker-compose.yml up
docker-compose down
```