https://github.com/neo-ciber94/etl-data-analytics

An extract-transform-load pipeline for stock transactions
https://github.com/neo-ciber94/etl-data-analytics

clean-architecture docker etl-pipeline mongodb mongoose mysql nodejs prisma rabbitmq repository-pattern typescript

Last synced: 8 days ago
JSON representation

An extract-transform-load pipeline for stock transactions

• Host: GitHub
• URL: https://github.com/neo-ciber94/etl-data-analytics
• Owner: Neo-Ciber94
• Created: 2022-12-03T00:45:58.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2022-12-05T01:59:00.000Z (almost 2 years ago)
• Last Synced: 2024-10-12T07:15:52.983Z (about 1 month ago)
• Topics: clean-architecture, docker, etl-pipeline, mongodb, mongoose, mysql, nodejs, prisma, rabbitmq, repository-pattern, typescript
• Language: TypeScript
• Homepage:
• Size: 5.13 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# ETL Data Analytics

[![CI][ci_badge]][ci]

[ci]: https://github.com/Neo-Ciber94/ETL-Data-Analytics/actions/workflows/ci.yml

[ci_badge]: https://github.com/Neo-Ciber94/ETL-Data-Analytics/actions/workflows/ci.yml/badge.svg

An Extract-Transform-Load (ETL) pipeline for stocks from 3 sources.

## Diagram

![Flow graph](./assets/process_transactions_graph.png)

## Resources

1. Environment files

   - `.env.local` used for local development

   - `.env.local.container` used when running in a container

## Getting started

1. To start the project use `docker compose up -d` and wait for all the services to initialize

2. After that hit the endpoint `http://localhost:18080/etl/process` to start processing the transactions

   - **(Windows)**

     ```bash

     Invoke-WebRequest http://localhost:18080/etl/process -Method POST -UseBasicParsing

     ```

   - **(Linux)**

     ```bash

     curl -X POST http://localhost:18080/etl/process

     ```

   This may take some seconds

3. Check the reports in the `mongodb` database, in the `reports` collection: 

   - user: root

   - password: EtLTest2022

4. Check the rabbit message queue, in the `transactions.error` and `transactions.insight` queues: , to prevent consuming messages twice I added an expiration, so the messages could disappear.

   - user: root

   - password: EtLTest2022

5. To remove the containers run `docker compose down && npm run clean:etl`

   - The last command `npm run clean:etl` removes the `etl` docker image.