https://github.com/sreedevk/csv-ingestion-pipeline
A Demo Project Implementing CSV Ingestion Pipelines using the Broadway Framework (Elixir)
https://github.com/sreedevk/csv-ingestion-pipeline
broadway csv data-ingestion demo elixir etl-pipeline
Last synced: 10 months ago
JSON representation
A Demo Project Implementing CSV Ingestion Pipelines using the Broadway Framework (Elixir)
- Host: GitHub
- URL: https://github.com/sreedevk/csv-ingestion-pipeline
- Owner: sreedevk
- Created: 2021-09-28T17:14:51.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2024-12-17T14:02:33.000Z (over 1 year ago)
- Last Synced: 2025-06-21T03:50:37.460Z (about 1 year ago)
- Topics: broadway, csv, data-ingestion, demo, elixir, etl-pipeline
- Language: Elixir
- Homepage:
- Size: 186 KB
- Stars: 6
- Watchers: 3
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# GenreMatcher
GenreMatcher is an example implementation of Broadway Pipelines for CSV data ingestion & matching.
The sample CSV used here is a movie dataset that was downloaded from kaggle - https://www.kaggle.com/chaitanyahivlekar/large-movie-dataset
The application uses redis streams to communicate between pipelines and processes.
Authors: Sreedev Kodichath, Joseph Giralt
## Setup
* Install asdf
[ASDF Documentation](http://asdf-vm.com/guide/getting-started.html#_1-install-dependencies)
* Install Language Servers + Compilers + Interpreters
``` sh
asdf install
```
Please note that the above command will install the following software on your system:
``` markdown
- elixir 1.12.2-otp-24
- erlang 24.0.5
- postgres 13.2
- nodejs 8.5.0
- docker 20.10.8
```
* Install Elixir + Erlang Dependencies
``` sh
mix deps.get
```
* Install Phoenix
``` sh
mix local.hex # will install hex package manager
mix archive.install hex phx_new # will install phoenix
```
* Edit `.envrc` with your local database creds
* Create Database
``` sh
mix ecto.create
```
* Run Migrations
``` sh
mix ecto.migrate
```
## Screenshot
