Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mbta/dmap-import
Import DMAP data into a PostgreSQL database
https://github.com/mbta/dmap-import
Last synced: about 1 month ago
JSON representation
Import DMAP data into a PostgreSQL database
- Host: GitHub
- URL: https://github.com/mbta/dmap-import
- Owner: mbta
- License: mit
- Created: 2023-10-27T13:07:43.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-19T21:49:00.000Z (about 1 month ago)
- Last Synced: 2024-11-19T22:38:05.848Z (about 1 month ago)
- Language: Python
- Size: 208 KB
- Stars: 0
- Watchers: 9
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# dmap-import
Import DMAP data into a PostgreSQL database
## Dev Setup (Mac)
* Install ASDF, Python3, Postgres
```sh
brew install asdf
brew install python3
brew install postgresql
```
* run `asdf install` to install tools via asdf
* run `poetry install` to install python dependencies
* run `cp .env.template .env` and fill out any missing environment variables
* run `docker-compose build` to build the docker images for local testing
* run `docker-compose up dmap_local_rds` to stand up a local postgres db
* this imnage could be used when running pytest
* run `docker-compose up dmap_import` to run the importer application
### Outside of Docker
1. Navigate to repository directory.
2. Update `.env` variable. Source it `source .env`.
3. Run `poetry run start` to run the ingestion process.
4. Run `psql postgresql://postgres:[email protected]:5432/dmap_importer` to get into the database. Alternatively, after `docker-compose up`, you can:
1. `docker exec -it dmap_local_rds bash`
2. `psql -U postgres -d dmap_importer`
5. Run format, type and lint checker:
* `poetry run black .`
* `poetry run mypy .`
* `poetry run pylint src tests`
6. Run tests, `poetry run pytest`.
### Alembic
To create new migration:
```sh
alembic revision -m "adding a new column"
# [optional] rename generated file so as to sort migrations by name by prepending '0xx_'
```