https://github.com/jeffbrennan/covid_redux
Modernized version of my TexasPandemics repo
https://github.com/jeffbrennan/covid_redux
Last synced: 3 months ago
JSON representation
Modernized version of my TexasPandemics repo
- Host: GitHub
- URL: https://github.com/jeffbrennan/covid_redux
- Owner: jeffbrennan
- Created: 2022-10-28T02:44:24.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2023-05-14T04:54:07.000Z (about 2 years ago)
- Last Synced: 2025-01-17T06:45:20.672Z (5 months ago)
- Language: Python
- Homepage:
- Size: 42 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# COVID Redux
Modernizing the ETL of my [largest project]("https://github.com/jeffbrennan/TexasPandemics")
## Goals
- Transition all csv/xlsx output to database tables
- Transition manual diagnostic checking & etl to dbt+dagster
- Only clean and write updates for newest data
## Tools
- ### Local
- Python
- Docker
- Postgres
- Dagster
- ### Cloud (soon)
- AWS
- PowerBI
## Next steps
- [x] Vitals data loaded and cleaned
- [x] Vaccination data loaded and cleaned
- [x] Dbt models reorganized
- [ ] Tests added to all scripts
- [ ] Weekly run set up
- [ ] Migrate local parquet storage to aws s3
- [ ] Migrate local postgres to aws rds
- [ ] Set up config to run on ECS cluster