https://github.com/abeusher/cast-iron
The platform-agnostic data storage and ETL system leveraging MinIO and other tools to enable mission endstates.
https://github.com/abeusher/cast-iron
Last synced: 3 months ago
JSON representation
The platform-agnostic data storage and ETL system leveraging MinIO and other tools to enable mission endstates.
- Host: GitHub
- URL: https://github.com/abeusher/cast-iron
- Owner: abeusher
- License: mit
- Created: 2020-08-31T18:12:20.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2020-08-18T23:57:08.000Z (almost 5 years ago)
- Last Synced: 2025-01-14T12:13:39.967Z (4 months ago)
- Homepage:
- Size: 56.6 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Cast Iron
The platform-agnostic data storage and ETL system leveraging MinIO and other tools to enable mission endstates.
## Basics
This project provides an example environment using [MinIO] to create a basic, cloud-agnostic ETL system.
Main Parts of the system include:
* Object Storage ([AWS S3], [MinIO], etc.)
* Message Queue ([Kafka], [Nats], etc.)
* Worker ([Celery], [Faust], etc.)
* ETL (Bash Scripts, Python)
* Database ([MS SQL], [MySQL], [PostgreSQL], [SQLite], etc.)
## Getting Started
* Install [Docker]
* Run the docker compose
```bash
$ docker-compose up
```
Once started, the following areas are accessible:
* MinIO at localhost:9000
* PostgreSQL
* User: castiron
* Password: castiron
* Host: localhost
* Port: 5432
* Database: castiron
[AWS S3]: https://aws.amazon.com/s3/
[Celery]: https://docs.celeryproject.org/en/stable/index.html
[Docker]: https://www.docker.com/
[Faust]: https://faust.readthedocs.io/en/latest/index.html
[Kafka]: https://kafka.apache.org/
[MinIO]: https://min.io/
[MySQL]: https://www.mysql.com/
[Nats]: https://nats.io/
[PostgreSQL]: https://www.postgresql.org/
[SQLite]: https://www.sqlite.org/index.html
[MS SQL]: https://www.microsoft.com/en-us/sql-server