https://github.com/ocneve/4bigd
https://github.com/ocneve/4bigd
Last synced: 8 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/ocneve/4bigd
- Owner: OCNeve
- Created: 2024-06-07T09:54:46.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2024-06-11T21:47:45.000Z (about 2 years ago)
- Last Synced: 2025-01-14T21:46:39.065Z (over 1 year ago)
- Language: Python
- Size: 657 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.MD
Awesome Lists containing this project
README
# Startup
_this project depends on docker, make sure the engine is running before starting the project._
## Setup
First of all, you must pull remote images and build local ones, for this, position yourself at the project root and run the following command.
```bash
docker compose build
```
## Running the project
To startup the whole infrastructure and run the pipeline, run the following command.
```bash
docker compose up -d
```
## Reading the pipeline logs
So as to see what the pipeline is up to, and read the results of the data validation checks, you can run the following command.
```bash
docker logs --follow pyspark
```