{"id":19831238,"url":"https://github.com/mfloriach/file_processor","last_synced_at":"2025-02-28T17:42:19.914Z","repository":{"id":247897167,"uuid":"824346160","full_name":"mfloriach/file_processor","owner":"mfloriach","description":"batch processor example to test concurrency in golang","archived":false,"fork":false,"pushed_at":"2024-07-19T11:18:34.000Z","size":10775,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-11T10:16:08.227Z","etag":null,"topics":["concurrency","go","mongoengine","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mfloriach.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-04T23:44:20.000Z","updated_at":"2024-07-19T11:18:37.000Z","dependencies_parsed_at":"2025-01-11T10:15:51.713Z","dependency_job_id":"09927095-ab70-4f06-98de-6d05a87d1821","html_url":"https://github.com/mfloriach/file_processor","commit_stats":null,"previous_names":["mfloriach/file_processor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mfloriach%2Ffile_processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mfloriach%2Ffile_processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mfloriach%2Ffile_processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mfloriach%2Ffile_processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mfloriach","download_url":"https://codeload.github.com/mfloriach/file_processor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241191405,"owners_count":19925243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","go","mongoengine","s3"],"created_at":"2024-11-12T11:29:28.466Z","updated_at":"2025-02-28T17:42:19.894Z","avatar_url":"https://github.com/mfloriach.png","language":"Go","readme":"# Batch processor\n[![test](https://github.com/mfloriach/file_processor/actions/workflows/test.yml/badge.svg)](https://github.com/mfloriach/file_processor/actions/workflows/test.yml)\n![go](https://img.shields.io/badge/go-1.22-blue)\n\nBatch processor example to test concurrency patterns in golang.\n\nModules\n- Metadata: get year, name, start, authors,....\n- Compress: compress and move to S3 bucket\n- Store: save into Mongo DB\n\n## Visual tools\n\n| Tool |  Description  |\n|:-----|:--------:|\n| [mongo](http://localhost:8082/)                   |Database management |\n| [minio](http://127.0.0.1:9001)                    |S3 alternative manager  |\n| [stats](http://localhost:18066/debug/statsview)   |Golang related statuses|\n| [cadvisor](http://localhost:8080/docker)          |Hardware related statuses|\n\n## How to use\n\n### Start\n```bash\n$ docker compose up -d # start services\n$ make run             # run in parallel mode\n$ make run-sequencial  # run sequential\n$ make run-concurrent  # run concurrent\n$ make run-parallel    # run parallel\n```\n\n### Benchmark\n```bash\n$ make bench \n$ make benchErrs \n$ make benchConc \n```\n\n## Architecture\n\n### Sequencial\n```mermaid\nflowchart LR\n    FA[read file] --\u003e M[Metada]\n    M[Metada] --\u003e |job1|C[Compress]\n    C[Compress] --\u003e S[Store]\n```\n\n### Concurrent\n```mermaid\nflowchart LR\n    FA[read file] --\u003e|job3| Metadata\n    subgraph Metadata\n    direction RL\n    M1[Worker]\n    end\n    Metadata --\u003e |job2|Compress\n    subgraph Compress\n    direction RL\n    C[Worker]\n    end\n    Compress --\u003e |job1|Store\n    subgraph Store\n    direction RL\n    S1[Worker]\n    end\n```\n\n### Parallel\n```mermaid\nflowchart LR\n    FA[read file] --\u003e Metadata\n    subgraph Metadata\n    direction RL\n    M[Worker job 6]\n    M1[Worker job 5]\n    end\n    Metadata --\u003e Compress\n    subgraph Compress\n    direction RL\n    C[Worker job 3]\n    C2[Worker job 4]\n    end\n    Compress --\u003e Store\n    subgraph Store\n    direction RL\n    S[Worker job 1]\n    S1[Worker job 2]\n    end\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmfloriach%2Ffile_processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmfloriach%2Ffile_processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmfloriach%2Ffile_processor/lists"}