https://github.com/hemicharly/on-demand-archive-process-nodejs
This project demonstrates an example Node.js application, with the goal of applying the power of Node.js stream and pipeline in data processing, aiming to efficiently process large data sets in batches, minimizing memory consumption.
https://github.com/hemicharly/on-demand-archive-process-nodejs
batch-file-insert data-processing memory mysql nodejs performance pipeline scalability streams
Last synced: 6 months ago
JSON representation
This project demonstrates an example Node.js application, with the goal of applying the power of Node.js stream and pipeline in data processing, aiming to efficiently process large data sets in batches, minimizing memory consumption.
- Host: GitHub
- URL: https://github.com/hemicharly/on-demand-archive-process-nodejs
- Owner: hemicharly
- Created: 2024-04-18T21:35:59.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-09T10:03:59.000Z (10 months ago)
- Last Synced: 2025-01-05T03:13:18.817Z (9 months ago)
- Topics: batch-file-insert, data-processing, memory, mysql, nodejs, performance, pipeline, scalability, streams
- Language: JavaScript
- Homepage:
- Size: 146 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## ON-DEMAND ARCHIVE PROCESS
This project demonstrates an example Node.js application, with the goal of applying the power of Node.js stream and
pipeline in data processing, aiming to efficiently process large data sets in batches, minimizing memory consumption. It
reads a CSV file into chunks using streams, parses the csv data and batch processes it before inserting in the MySQL
database. The use of streams and pipelines allows for more efficient and scalable execution, especially when
deal with large volumes of data.
### Keywords
*Keywords*: Node.js, Stream, Pipeline, Batch File Insert, Performance, Scalability, Memory, Data processing, MySQL.
**Overall, the code performs the following steps**:
* Establishes the connection to the MySQL database.
* Reads a CSV file.
* Processes CSV data in batches.
* Inserts batches of data into the database.
* Finalizes execution, dealing with errors and ending the process.
#### SYSTEM DESIGN DIAGRAM
#### SEQUENCE DIAGRAM
#### 1. Requirements Installation
##### 1.1. Installed on your host
* Install docker *26.0.2*
* Install docker-compose *1.29.2**
* Install Makefile *4.3*
##### 1.1. Installed with docker image
* NodeJs *20.9*
* MySql *8.0.28*
#### 2. Steps to run this project
##### 2.1. Start MySQL docker container
```bash
make mysql
```
##### 2.2. Create table in MySQL
```bash
create table if not exists PERSON (
ID bigint auto_increment primary key,
NAME varchar(255) not null,
CITY varchar(255) not null,
STATE varchar(100) not null,
CREATED_AT datetime default CURRENT_TIMESTAMP not null
);
```
##### 2.3. Installing project dependencies
```bash
make install
```
##### 2.4. Run application
```bash
make start
```