https://github.com/umitkaanusta/smol-elt

a smol elt (not etl) pipeline for smol tasks
https://github.com/umitkaanusta/smol-elt

analytics automation aws aws-sns data data-engineering data-pipeline elt etl google-sheets pandas pipeline python spreadsheet web-scraping

Last synced: 5 days ago
JSON representation

a smol elt (not etl) pipeline for smol tasks

• Host: GitHub
• URL: https://github.com/umitkaanusta/smol-elt
• Owner: umitkaanusta
• License: mit
• Created: 2020-09-26T22:50:11.000Z (about 4 years ago)
• Default Branch: master
• Last Pushed: 2021-12-13T21:04:30.000Z (about 3 years ago)
• Last Synced: 2024-10-30T11:24:07.559Z (about 2 months ago)
• Topics: analytics, automation, aws, aws-sns, data, data-engineering, data-pipeline, elt, etl, google-sheets, pandas, pipeline, python, spreadsheet, web-scraping
• Language: Python
• Homepage:
• Size: 95.7 KB
• Stars: 3
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# smol-elt (not ETL)

just a smol elt pipeline

u can use it as a template or a sample to look at

see how it works down there 👇

![Diagram](images/diagram.PNG)

## Sample Output

**Demo video** 

- https://www.youtube.com/watch?v=vEtckC8HS04

**Spreadsheet**

- https://docs.google.com/spreadsheets/d/1ByKu9j9e-R-N9O83Zszbrlb_3KBwyR5p2V743Vau3F8/edit#gid=0

### Alert Messages

#### Success

![Success](images/success.PNG)

#### Failure

![Failure](images/failure.PNG)

# How can I adjust smol-elt to my use case?

- Clone the repository

- Install the needed packages

    - `pip install -r requirements.txt`

- Create credentials for Google Sheets API

    - https://gspread.readthedocs.io/en/latest/oauth2.html

    - Follow the instructions below the `For Bots` part

    - Save the output as `credentials/gsheet_credentials.json`

- Create credentials for AWS

    - Open an AWS account if you don't have it

    - Create an IAM user with `AmazonSNSFullAccess` policy

    - Get your security credentials

    - Save your credentials with the given format in `credentials/aws_credentials.json`

- Configure SNS

    - Create an SNS topic and subscribe your e-mail address to that topic

- Edit files for your use case

    - Create your own pipeline by using ours as a sample

    - Change the schedule in tasks.py according to your needs

- Run `tests/run_tests.py` to automatically test the ELT part

- Run `smol/run.py` to use the pipeline