Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/leobitto/cloudforge
Data foundry
https://github.com/leobitto/cloudforge
airflow data data-engineering django docker docker-compose grafana postgresql prometheus
Last synced: 13 days ago
JSON representation
Data foundry
- Host: GitHub
- URL: https://github.com/leobitto/cloudforge
- Owner: leoBitto
- License: gpl-3.0
- Created: 2024-12-03T22:27:43.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2024-12-26T17:00:17.000Z (15 days ago)
- Last Synced: 2024-12-26T18:17:02.483Z (15 days ago)
- Topics: airflow, data, data-engineering, django, docker, docker-compose, grafana, postgresql, prometheus
- Language: Python
- Homepage: https://leobitto.github.io/DataFoundry/
- Size: 592 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 22
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# DataFoundry
[](LICENSE)
[](https://github.com/leoBitto/DataFoundry/issues)
[](https://github.com/leoBitto/DataFoundry)
Welcome to **DataFoundry** 🚀, a flexible and scalable platform designed to orchestrate **data engineering workflows** for small and medium-sized enterprises (SMEs).
DataFoundry seamlessly integrates tools for **data automation**, **monitoring**, and **business analytics**:
- **Django**: Build business-driven web applications with ease.
- **Streamlit**: Deliver interactive **visual insights** and data visualizations.
- **Airflow**: Automate and manage your data pipelines.
- **PostgreSQL**: Separate data storage layers (**Gold** & **Silver**) for different quality of the data.
- **Grafana & Prometheus**: Monitor and alert on system health and performance.
Together, these tools provide an **end-to-end solution** for automating workflows, analyzing data, and supporting business decision-making.
---
## Features ✨
- **Django**: Core framework for building dynamic and robust business applications.
- **Streamlit**: Visualize your data interactively with custom dashboards.
- **Airflow**: Automate workflows and orchestrate complex data pipelines.
- **PostgreSQL (Gold & Silver)**: Two instances to manage refined and aggregated data.
- **Grafana & Prometheus**: Monitor metrics and get real-time system insights.
- **Nginx**: Proxy and route requests efficiently.
---
## Getting Started 🛠️
Follow these steps to get DataFoundry up and running locally:
1. **Clone the repository**:
```bash
git clone https://github.com/leoBitto/DataFoundry.git
cd DataFoundry
```
2. **Build and run the project** using the provided script:
```bash
source manager.sh build
```
3. **Access the services**:
- **Airflow**: [http://localhost/airflow](http://localhost/airflow)
Username: `admin` | Password: `admin`
- **Django Admin**: [http://localhost/admin](http://localhost/admin)
Username: `admin` | Password: `admin`
- **Streamlit**: [http://localhost:8501](http://localhost:8501)
- **Grafana**: [http://localhost:3000](http://localhost:3000)
Default Login: `admin` / `admin`
---
## Project Structure 📂
```plaintext
DataFoundry/
├── config
│ ├── airflow.conf
│ ├── airflow-db.conf
│ ├── django.conf
│ ├── gold.conf
│ └── silver.conf
├── docker
│ ├── django
│ │ └── Dockerfile
│ ├── docker-compose.airflow.yml
│ ├── docker-compose.dev.yml
│ └── docker-compose.django.yml
├── docs
│ ├── assets
│ │ └── img
│ │ └── DataFoundry.png
│ ├── base.md
│ ├── _config.yml
│ └── index.md
├── LICENSE
├── manager.sh
├── nginx
│ ├── nginx.conf
│ └── nginx.dev.conf
├── README.md
└── src
├── airflow
│ ├── dags
│ │ └── test.py
│ ├── logs
│ │ └── scheduler
│ └── plugins
├── django
│ ├── base
│ │ ├── asgi.py
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ ├── create_superuser.py
│ ├── manage.py
│ └── requirements.txt
└── streamlit
```
---
## Contributing 🤝
Contributions are welcome! If you'd like to contribute to **DataFoundry**, please follow these steps:
1. Fork the repository.
2. Create a new branch (`git checkout -b feature-xyz`).
3. Commit your changes (`git commit -m "Add feature xyz"`).
4. Push to the branch (`git push origin feature-xyz`).
5. Open a Pull Request.
---
## License 📜
This project is licensed under the **GNU General Public License (GPL)**. See the [LICENSE](LICENSE) file for details.
---
## Contact 📬
For any questions, suggestions, or feedback, feel free to open an issue on [GitHub](https://github.com/leoBitto/DataFoundry/issues).
---