https://github.com/leobitto/cloudforge
CloudForge
https://github.com/leobitto/cloudforge
airflow data data-engineering django docker docker-compose postgresql
Last synced: 3 months ago
JSON representation
CloudForge
- Host: GitHub
- URL: https://github.com/leobitto/cloudforge
- Owner: leoBitto
- License: gpl-3.0
- Created: 2024-12-03T22:27:43.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2025-01-03T21:04:25.000Z (5 months ago)
- Last Synced: 2025-01-03T21:32:27.450Z (5 months ago)
- Topics: airflow, data, data-engineering, django, docker, docker-compose, postgresql
- Language: Python
- Homepage: https://leobitto.github.io/CloudForge/
- Size: 585 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# CloudForge
[](LICENSE)
[](https://github.com/leoBitto/CloudForge/issues)
[](https://github.com/leoBitto/CloudForge)
[](https://github.com/leoBitto/CloudForge/actions/workflows/CI.yml)
Welcome to **CloudForge** 🚀, a flexible and scalable platform designed to orchestrate **data engineering workflows** for small and medium-sized enterprises (SMEs).
CloudForge seamlessly integrates tools for **data automation**, **monitoring**, and **business analytics**:
- **Django**: Build business-driven web applications with ease.
- **Streamlit**: Deliver interactive **visual insights** and data visualizations.
- **Airflow**: Automate and manage your data pipelines.
- **PostgreSQL**: Separate data storage layers (**Gold** & **Silver** & **Bronze**) for different quality of the data.
Together, these tools provide an **end-to-end solution** for automating workflows, analyzing data, and supporting business decision-making.
---
## Features ✨
- **Django**: Core framework for building dynamic and robust business applications.
- **Streamlit**: Visualize your data interactively with custom dashboards.
- **Airflow**: Automate workflows and orchestrate complex data pipelines.
- **PostgreSQL (Gold & Silver & Bronze)**: Three instances to manage raw, refined and aggregated data.
- **Nginx**: Proxy and route requests efficiently.
---
## Getting Started 🛠️
Follow these steps to get CloudForge up and running locally:
1. **Clone the repository**:
```bash
git clone https://github.com/leoBitto/CloudForge.git
cd CloudForge
```
2. **Build and run the project** using the provided script:
```bash
source manager.sh build
```
3. **Access the services**:
- **Airflow**: [http://localhost/airflow](http://localhost/airflow)
Username: `admin` | Password: `admin`
- **Django Admin**: [http://localhost/admin](http://localhost/admin)
Username: `admin` | Password: `admin`
- **Streamlit**: [http://localhost/streamlit](http://localhost/streamlit)
---
## Project Structure 📂
```plaintext
CloudForge/
├── docker
│ ├── dev
│ │ ├── compose.airflow.yml
│ │ ├── compose.base.yml
│ │ ├── compose.databases.yml
│ │ ├── compose.django.yml
│ │ ├── compose.nginx.yml
│ │ └── compose.streamlit.yml
│ └── prod
├── LICENSE
├── manager.sh
├── README.md
└── src
├── airflow
│ ├── config
│ │ ├── airflow.conf
│ │ └── airflow-db.conf
│ └── plugins
├── django
│ ├── app
│ │ ├── authentication
│ │ │ ├── admin.py
│ │ │ ├── apps.py
│ │ │ ├── __init__.py
│ │ │ ├── models.py
│ │ │ ├── signals.py
│ │ │ ├── templates
│ │ │ │ └── authentication
│ │ │ │ ├── login.html
│ │ │ │ └── logout.html
│ │ │ ├── tests.py
│ │ │ ├── urls.py
│ │ │ └── views.py
│ │ ├── backoffice
│ │ │ ├── admin.py
│ │ │ ├── apps.py
│ │ │ ├── __init__.py
│ │ │ ├── models.py
│ │ │ ├── templates
│ │ │ │ └── backoffice
│ │ │ │ └── backoffice.html
│ │ │ ├── tests.py
│ │ │ ├── urls.py
│ │ │ └── views.py
│ │ ├── base
│ │ │ ├── asgi.py
│ │ │ ├── __init__.py
│ │ │ ├── settings.py
│ │ │ ├── urls.py
│ │ │ └── wsgi.py
│ │ ├── create_superuser.py
│ │ ├── manage.py
│ │ └── website
│ │ ├── admin.py
│ │ ├── apps.py
│ │ ├── __init__.py
│ │ ├── migrations
│ │ │ └── __init__.py
│ │ ├── models.py
│ │ ├── templates
│ │ │ └── website
│ │ │ └── home.html
│ │ ├── tests.py
│ │ ├── urls.py
│ │ └── views.py
│ ├── config
│ │ ├── databases
│ │ │ ├── bronze.conf
│ │ │ ├── gold.conf
│ │ │ └── silver.conf
│ │ └── django.conf
│ ├── Dockerfile
│ └── requirements.txt
├── nginx
│ └── config
│ └── nginx.dev.conf
└── streamlit
├── app
│ ├── app.py
│ ├── authentication.py
├── config
│ ├── authorized_groups.yml
│ └── streamlit.conf
├── Dockerfile
└── requirements.txt
33 directories, 68 files
```
---
## Contributing 🤝
Contributions are welcome! If you'd like to contribute to **CloudForge**, please follow these steps:
1. Fork the repository.
2. Create a new branch (`git checkout -b feature-xyz`).
3. Commit your changes (`git commit -m "Add feature xyz"`).
4. Push to the branch (`git push origin feature-xyz`).
5. Open a Pull Request.
---
## License 📜
This project is licensed under the **GNU General Public License (GPL)**. See the [LICENSE](LICENSE) file for details.
---
## Contact 📬
For any questions, suggestions, or feedback, feel free to open an issue on [GitHub](https://github.com/leoBitto/CloudForge/issues).
---