{"id":19149267,"url":"https://github.com/restuwahyu13/csv-streaming","last_synced_at":"2026-06-13T01:33:48.103Z","repository":{"id":241598340,"uuid":"806737794","full_name":"restuwahyu13/csv-streaming","owner":"restuwahyu13","description":"Example implementation CSV streaming data as worker service","archived":false,"fork":false,"pushed_at":"2024-06-01T19:18:13.000Z","size":37,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-22T20:32:04.271Z","etag":null,"topics":["api-rest","csv-import","kafka-streams","message-queue","streaming-data","worker-service"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/restuwahyu13.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-27T19:48:59.000Z","updated_at":"2024-06-03T07:31:33.000Z","dependencies_parsed_at":"2025-01-03T23:45:27.193Z","dependency_job_id":null,"html_url":"https://github.com/restuwahyu13/csv-streaming","commit_stats":null,"previous_names":["restuwahyu13/csv-streaming"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/restuwahyu13/csv-streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/restuwahyu13%2Fcsv-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/restuwahyu13%2Fcsv-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/restuwahyu13%2Fcsv-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/restuwahyu13%2Fcsv-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/restuwahyu13","download_url":"https://codeload.github.com/restuwahyu13/csv-streaming/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/restuwahyu13%2Fcsv-streaming/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34269364,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-rest","csv-import","kafka-streams","message-queue","streaming-data","worker-service"],"created_at":"2024-11-09T08:07:26.423Z","updated_at":"2026-06-13T01:33:48.078Z","avatar_url":"https://github.com/restuwahyu13.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSV Streaming\n\nBerikut adalah tutorial bagaimana cara melakukan processing data tanpa membebani proses dari kinerja api dan jika anda sedang mencari tutorial terkait kafka bisa cek [disini](https://github.com/restuwahyu13/node-kafka), jika anda mebutuhkan sesuatu yang lebih dari yang saya berikan, anda bisa mengcustomisasi nya sesuai dengan kebutuhan anda, untuk data dummy csv filenya anda bisa menggunakannya [disini](https://www.rndgen.com/data-generator).\n\n\n## User Table\n```sql\nCREATE EXTENSION \"uuid-ossp\";\n\nCREATE TABLE public.user (\n  id uuid primary key default uuid_generate_v4(),\n  first_name varchar(255) not null,\n  last_name varchar(255) not null,\n  email varchar(255) unique not null,\n  gender varchar (255) null\n)\n\nCREATE INDEX user_email ON public.user (email)\n```\n\n## Factors Affecting Performance\n\n- Banyaknya jumlah csv data\n- Koneksi jaringan internet\n- Hardware device\n- Database koneksi\n- Condingan \u0026 SQL Query\n- Rebalancing consumer (Kafka)\n- Etc\n\n## Tested By Me\n\nSeperti yang saya bilang jika anda upload **1JT** csv data nanti akan dibagi dengan `CHUNK_CSV_SIZE`, contoh jadi **1JT / 100RB** berarti nanti bakal ada **10 Process Batch** yang dimana nanti akan di publish ke `kafka`, anda bisa sesuaikan `CHUNK_CSV_SIZE` dan `GORUTINE_POOL_SIZE` sesuai kebutuhan anda masing - masing, jika dirasa data yang di upload semakin besar anda bisa increase `GORUTINE_POOL_SIZE`, perlu di ingat jangan terlalu besar karena bisa memperlambat dan terlalu banyak menggunakan `cpu`. saya testing `300RB` sampai `500RB` data itu masih terbilang lumayan agak cepat dan low consuming memory walau cpu agak sedikit naik.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frestuwahyu13%2Fcsv-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frestuwahyu13%2Fcsv-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frestuwahyu13%2Fcsv-streaming/lists"}