https://github.com/jcserv/manaql-ingest

manaql-ingest is a Django application that retrieves Scryfall bulk data and processes it into Postgres, to enable the manaql-api
https://github.com/jcserv/manaql-ingest

django etl postgres

Last synced: about 2 months ago
JSON representation

manaql-ingest is a Django application that retrieves Scryfall bulk data and processes it into Postgres, to enable the manaql-api

• Host: GitHub
• URL: https://github.com/jcserv/manaql-ingest
• Owner: jcserv
• License: gpl-3.0
• Created: 2024-11-23T20:12:55.000Z (6 months ago)
• Default Branch: main
• Last Pushed: 2024-12-27T04:27:58.000Z (5 months ago)
• Last Synced: 2025-02-02T03:44:37.488Z (4 months ago)
• Topics: django, etl, postgres
• Language: Python
• Homepage:
• Size: 200 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# manaql-ingest

![visitors](https://img.shields.io/endpoint?url=https://vu-mi.com/api/v1/views?id=jcserv/manaql-ingest)

manaql-ingest is a Django application that ingests Scryfall card data into a PostgreSQL database, to be used by [manaql-api](https://github.com/jcserv/manaql) project.

## installation

### prerequisites

- python3

- pipx

- poetry

### running locally

1. Create a virtual environment

`python3 -m venv .venv`

2. Activate the virtual environment

`source .venv/bin/activate`

3. Run `make setup`

4. Run `poetry install`

5. Run `make dev-db` to start a local postgres database, then `make migrate`

Then, to run the application:

Run `make run` to run the entire etl process OR:

- `make download` to download the Scryfall data

- `make ingest` to ingest the Scryfall data into the database

- `make process` to process the Scryfall data into the card/printings tables

TODO:

- async.io instead of tqdm?