https://github.com/pyladiesams/etl-intermediate-dec2020
Talks co-organized with PyAmsterdam about ETL/ELT in action in the cloud
https://github.com/pyladiesams/etl-intermediate-dec2020
aws aws-glue docker etl gcp
Last synced: 8 months ago
JSON representation
Talks co-organized with PyAmsterdam about ETL/ELT in action in the cloud
- Host: GitHub
- URL: https://github.com/pyladiesams/etl-intermediate-dec2020
- Owner: pyladiesams
- License: mit
- Created: 2020-12-09T21:41:46.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2020-12-30T09:53:19.000Z (almost 5 years ago)
- Last Synced: 2025-01-03T22:14:08.825Z (9 months ago)
- Topics: aws, aws-glue, docker, etl, gcp
- Homepage:
- Size: 530 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# [ONLINE] aggregate(PyLadies, PyAmsterdam, ETL).byebye(2020)
### Level: Intermediate - Advanced
## Meetup description
This meetup is co-organized with [PyAmsterdam](https://py.amsterdam/), and it's all about ETL/ELT in action in the cloud.
### Talk 1 - Data Processing with Python: a Containerized, Scheduled, and Monitored Pipeline on the Google Cloud Platform
During the talk, you will learn how to containerize a Python pipeline for data processing and execute it as a scheduled job that sends email messages for critical errors. The presentation can be find [here](https://github.com/pyladiesams/ETL-intermediate-dec2020/blob/master/presentations/pyladies-amsterdam-etl-slides.pdf). The repo with code [here](https://github.com/nancyirisarri/pyladies-amsterdam-etl).
### Talk 2 - AWS Glue: the good, the bad, the ugly
AWS Glue offers Serverless ETL pipeline and workflows, while it can be rather simple to start using it via console it quickly become non trivial for deployment via CI/CD pipeline. Management of multiple environments, working with the pipeline state and keeping your code DRY is a challenge. During the talk I will walk you through guts of AWS Glue and show my approach to more streamlined deployment. The presentation can be find [here](http://slides.com/oglopzelgo/aws-glue-pyamsterdam). The repo with code [here](https://github.com/1oglop1/aws-glue-monorepo-style).
## Video record
[Video record](https://bit.ly/3rza70O)
## Credits
This event was set up by @pyladiesams, @PyAmsterdam, @nancyirisarri and @1oglop1