https://github.com/hudl/luigi-monitor
Send summary messages of your Luigi jobs to Slack
https://github.com/hudl/luigi-monitor
decision-science
Last synced: 18 days ago
JSON representation
Send summary messages of your Luigi jobs to Slack
- Host: GitHub
- URL: https://github.com/hudl/luigi-monitor
- Owner: hudl
- Created: 2015-09-28T19:30:35.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2019-04-25T16:04:32.000Z (about 6 years ago)
- Last Synced: 2025-05-06T18:14:58.101Z (18 days ago)
- Topics: decision-science
- Language: Python
- Homepage:
- Size: 53.7 KB
- Stars: 46
- Watchers: 137
- Forks: 16
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
## Luigi Monitor
[](https://img.shields.io/pypi/v/luigi-monitor.svg)
[](https://img.shields.io/pypi/l/luigi-monitor.svg)
[](https://img.shields.io/pypi/pyversions/luigi-monitor.svg)
[](https://img.shields.io/pypi/format/luigi-monitor.svg)
[](https://travis-ci.com/hudl/luigi-monitor)
Send summary messages of your Luigi jobs to Slack.
### Overview
[Luigi](https://github.com/spotify/luigi) is a great tool for making
job pipelines, but it's hard to know the status of a run. Luigi has
support for error emails, but this requires configuring your machine
to send email, which is a hassle for short-lived EMR clusters. Further,
it sends an email for every failure, which can quickly swamp your inbox.
By contrast, this tool gathers all your failures and missing dependencies
and sends a summary Slack message when the job is finished.
### Usage
With default app username:
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=, max_print=10):
luigi.run(main_task_cls=MainClass)
```
With dynamic app username:
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=, max_print=10, username="FooBar Monitor"):
luigi.run(main_task_cls=MainClass)
```
Monitoring and notifying on various events:
Currently supports: `SUCCESS`, `DEPENDENCY_MISSING`, and `FAILURE`
By default, all three of the above are monitored and notified on. If, `SUCCESS` event is monitored and
all tasks succeed then the notification text is "Job ran successfully" instead of listing _all_
successful tasks.
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=, events=['DEPENDENCY_MISSING', 'FAILURE']):
luigi.run(main_task_cls=MainClass)
```
Alternatively:
`luigi-monitor --module path.to.module TaskName`
NB: if you plan to use luigi-monitor from the command line, set options using `luigi.cfg`:
```
[luigi-monitor]
slack_url=
max_print=
username=
```
This is a work in progress. Particularly, note that:
* It only sends notifications via Slack