https://github.com/mfloriach/file_processor
batch processor example to test concurrency in golang
https://github.com/mfloriach/file_processor
concurrency go mongoengine s3
Last synced: about 1 year ago
JSON representation
batch processor example to test concurrency in golang
- Host: GitHub
- URL: https://github.com/mfloriach/file_processor
- Owner: mfloriach
- Created: 2024-07-04T23:44:20.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-07-19T11:18:34.000Z (over 1 year ago)
- Last Synced: 2025-01-11T10:16:08.227Z (about 1 year ago)
- Topics: concurrency, go, mongoengine, s3
- Language: Go
- Homepage:
- Size: 10.3 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Batch processor
[](https://github.com/mfloriach/file_processor/actions/workflows/test.yml)
Batch processor example to test concurrency patterns in golang.
Modules
- Metadata: get year, name, start, authors,....
- Compress: compress and move to S3 bucket
- Store: save into Mongo DB
## Visual tools
| Tool | Description |
|:-----|:--------:|
| [mongo](http://localhost:8082/) |Database management |
| [minio](http://127.0.0.1:9001) |S3 alternative manager |
| [stats](http://localhost:18066/debug/statsview) |Golang related statuses|
| [cadvisor](http://localhost:8080/docker) |Hardware related statuses|
## How to use
### Start
```bash
$ docker compose up -d # start services
$ make run # run in parallel mode
$ make run-sequencial # run sequential
$ make run-concurrent # run concurrent
$ make run-parallel # run parallel
```
### Benchmark
```bash
$ make bench
$ make benchErrs
$ make benchConc
```
## Architecture
### Sequencial
```mermaid
flowchart LR
FA[read file] --> M[Metada]
M[Metada] --> |job1|C[Compress]
C[Compress] --> S[Store]
```
### Concurrent
```mermaid
flowchart LR
FA[read file] -->|job3| Metadata
subgraph Metadata
direction RL
M1[Worker]
end
Metadata --> |job2|Compress
subgraph Compress
direction RL
C[Worker]
end
Compress --> |job1|Store
subgraph Store
direction RL
S1[Worker]
end
```
### Parallel
```mermaid
flowchart LR
FA[read file] --> Metadata
subgraph Metadata
direction RL
M[Worker job 6]
M1[Worker job 5]
end
Metadata --> Compress
subgraph Compress
direction RL
C[Worker job 3]
C2[Worker job 4]
end
Compress --> Store
subgraph Store
direction RL
S[Worker job 1]
S1[Worker job 2]
end
```