Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/brenofariasdasilva/dagster-education-model
Dagster Education Model using Dagster 1.3.11 and Python 3.7.17.
https://github.com/brenofariasdasilva/dagster-education-model
dagster makefile matplotlib pandas pyenv python3 scikit-learn seaborn shellscript
Last synced: 21 days ago
JSON representation
Dagster Education Model using Dagster 1.3.11 and Python 3.7.17.
- Host: GitHub
- URL: https://github.com/brenofariasdasilva/dagster-education-model
- Owner: BrenoFariasdaSilva
- License: apache-2.0
- Created: 2023-06-22T14:08:28.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-26T00:53:04.000Z (8 months ago)
- Last Synced: 2024-04-20T14:46:20.495Z (7 months ago)
- Topics: dagster, makefile, matplotlib, pandas, pyenv, python3, scikit-learn, seaborn, shellscript
- Language: Python
- Homepage:
- Size: 21.9 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# [Dagster - Education Model.](https://github.com/BrenoFariasdaSilva/Dagster-EducationModel) 
---
#### This repository is a translation of a Jupyter notebook education model to Dagster, in which uses Dagster 1.3.11 and Python 3.7.17.
#### Dagster is a data orchestrator for machine learning, analytics, and ETL. It lets you define pipelines in terms of the data flow between reusable, logical components, then test locally and run anywhere. With a unified view of pipelines and the assets they produce, Dagster can schedule and orchestrate Pandas, Spark, SQL, or anything else that Python can invoke. It abstracts away the infrastructure details and job execution, so that you can focus on what matters: building data applications. For more information, please visit the [Dagster website](https://dagster.io/).
---
## Table of Contents
- [Project Files Structure:](#project-files-structure)
- [Install Dependencies:](#install-dependencies)
- [Pre-requisites:](#pre-requisites)
- [Backend:](#backend)
- [Model Dependencies:](#model-dependencies)
- [How to Run:](#how-to-run)
## Project Files Structure:
There are two levels of folders in this project. The first level is the project folder (root), and the second level is the Dagster folder (translated model). The project folder contains the files that are related setting up the python to a specific version (3.7.17) and dagster depencies. The Dagster folder contains the files that are related to Dagster project. Each folder level have a README.md file and a Makefile file.
## Install Dependencies:
### Pre-requisites:
First you need to run the `prerequisites.sh` file to install the project dependencies, such as `python`, `homebrew` and `node`. This file will install the `pyenv` and `pyenv-virtualenv` to manage the Python versions and virtual environments. The `homebrew` will be installed as package manager for the dagster packages. Lastly, the `node` is installed for the use of the website, in which we will have the dagster UI.
```shell
chmod +x ./prerequisites.sh # Gives execution permission to the file
./prerequisites.sh # Installs project dependencies
```
### Backend:
Now you need to run the `backend.sh` file (also located in the root directory of the project), which will create the python virtual environment and install the dagster packages, install `yarn`, the python modules and `dagit` (dagster UI).
```shell
chmod +x ./backend.sh # Gives execution permission to the file
./backend.sh # Installs project dependencies
```
Great, the dagster dependencies are installed. Now we need to install the project/model dependencies, located in `Dagster-Model/` folder.
### Model Dependencies:
Now that you have Python 3.7.17 installed, the Dagster dependencies and the pyenv virtual environment created, open a new terminal in the `Dagster-Model/` folder and execute the following makefile rule:
```shell
cd Dagster-Model # Enters the project folder
make dependecies # Installs the project dependencies, such as:
# Install Python dependencies of the model -> pandas, matplotlib, seaborn and scikit-learn.
# Install the database module dependencies -> psycopg2-binary and sqlalchemy, etc.
# Install Docker and Docker-Compose.
# Build the docker-compose.
```
Now you have the project dependencies installed. Let's run the model.
## How to Run:
At this point, you can simply run the following command to run the model:
```shell
make dagster # Run the dagster model
```
This will run the dagster dev command, which will be executed on port 3000 and receive requests from any host. You can access the dagster UI by accessing the following link: http://localhost:3000.