https://github.com/snimmagadda1/luigi-etl-example
🔍 Example of an ETL pipeline using Spotify's Luigi
https://github.com/snimmagadda1/luigi-etl-example
data luigi luigi-pipeline python spotify
Last synced: 7 months ago
JSON representation
🔍 Example of an ETL pipeline using Spotify's Luigi
- Host: GitHub
- URL: https://github.com/snimmagadda1/luigi-etl-example
- Owner: snimmagadda1
- Created: 2018-01-09T12:01:56.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2019-01-16T14:44:13.000Z (over 6 years ago)
- Last Synced: 2025-02-05T15:42:42.647Z (8 months ago)
- Topics: data, luigi, luigi-pipeline, python, spotify
- Language: Python
- Homepage:
- Size: 34.2 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Companion repo for post: http://www.s11a.com/2018/01/09/first-impressions-luigi/
## Running Tasks
The preferred way to run Luigi tasks is through the `luigi` command line tool that is installed with the pip package. Ex:
```
# my_module.py, available in your sys.path
import luigi
class MyTask(luigi.Task):
x = luigi.IntParameter()
y = luigi.IntParameter(default=45)
def run(self):
print(self.x + self.y)
```
Should be run like this
```
$ luigi --module my_module MyTask --x 123 --y 456 --local-scheduler
```
Or alternatively like this
```
$ python -m luigi --module my_module MyTask --x 100 --local-scheduler
```
Note that if a parameter name contains '_', it should be replaced by '-'. For example, if MyTask had a parameter called 'my_parameter':
```
$ luigi --module my_module MyTask --my-parameter 100 --local-scheduler
```