https://github.com/hong539/lctetl
LCTETL is a Linux crontab task to execute ETL(extract, transform, load) with python-scripts
https://github.com/hong539/lctetl
automation beautifulsoup crontab crontab-task linux pandas python selenium sqlalchemy webdriver
Last synced: 12 months ago
JSON representation
LCTETL is a Linux crontab task to execute ETL(extract, transform, load) with python-scripts
- Host: GitHub
- URL: https://github.com/hong539/lctetl
- Owner: hong539
- License: mit
- Created: 2023-10-24T17:27:45.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-25T06:46:30.000Z (over 2 years ago)
- Last Synced: 2023-10-25T18:41:48.098Z (over 2 years ago)
- Topics: automation, beautifulsoup, crontab, crontab-task, linux, pandas, python, selenium, sqlalchemy, webdriver
- Homepage:
- Size: 8.79 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# lctetl
LCTETL is a Linux crontab task to execute ETL(extract, transform, load) with python-scripts
## Prerequisites
* Python >= 3.8
* [pyenv + poetry](https://setup_dev_environment.ootori.dev/computer%20languages/programming_languages/python/python/#usage-with-pyenvpoetry)
* [Selenium](https://www.selenium.dev/)
* [WebDriver](https://www.selenium.dev/documentation/webdriver/)
* [Grid](https://www.selenium.dev/documentation/grid/)
* [pandas](https://pandas.pydata.org/)
* [SQLAlchemy](https://www.sqlalchemy.org/)
## Quick start
```shell
poetry add
```