{"id":23074847,"url":"https://github.com/thenriquedb/reading-large-files-with-node-streams","last_synced_at":"2026-05-04T23:33:46.393Z","repository":{"id":267875889,"uuid":"902093568","full_name":"thenriquedb/reading-large-files-with-node-streams","owner":"thenriquedb","description":"Read and write large CSV files efficiently using Node.js streams.","archived":false,"fork":false,"pushed_at":"2024-12-14T16:00:13.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-24T08:43:52.556Z","etag":null,"topics":["csv-parse","node","node-streams","nodejs","sqlite"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thenriquedb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-11T22:28:18.000Z","updated_at":"2024-12-14T16:00:17.000Z","dependencies_parsed_at":"2024-12-13T01:19:31.610Z","dependency_job_id":"20faaf35-e518-4f7a-9deb-f4c916e572d4","html_url":"https://github.com/thenriquedb/reading-large-files-with-node-streams","commit_stats":null,"previous_names":["thenriquedb/reading-large-files-with-node-streams"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/thenriquedb/reading-large-files-with-node-streams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenriquedb%2Freading-large-files-with-node-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenriquedb%2Freading-large-files-with-node-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenriquedb%2Freading-large-files-with-node-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenriquedb%2Freading-large-files-with-node-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thenriquedb","download_url":"https://codeload.github.com/thenriquedb/reading-large-files-with-node-streams/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thenriquedb%2Freading-large-files-with-node-streams/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32628830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"ssl_error","status_checked_at":"2026-05-04T10:08:02.005Z","response_time":58,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv-parse","node","node-streams","nodejs","sqlite"],"created_at":"2024-12-16T08:29:43.429Z","updated_at":"2026-05-04T23:33:46.376Z","avatar_url":"https://github.com/thenriquedb.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSV Read/Write with Node.js Streams and SQLite\n\nThis is a proof of concept (POC) demonstrating how to read and write large CSV files efficiently using Node.js streams. The data read from the CSV is also inserted into an SQLite database concurrently, making it suitable for handling large datasets without consuming excessive memory.\n\n## Features\n\n- **Efficiently read large CSV files** using Node.js streams.\n- **Write data to new CSV files** with minimal memory usage.\n- **Insert CSV data into an SQLite database** as it is read, line by line.\n- Suitable for processing large datasets without loading the entire file into memory.\n\n## Install\n\n1. Install dependecies\n```bash\nyarn\n```\n\n2. Generate CSV file. \n```bash\n# If the number of records is not passeddd, the default value will be 50\nyarn seed \u003cnumber of records\u003e\n```\n\n3. Run aplication\n```bash\nyarn migrate\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthenriquedb%2Freading-large-files-with-node-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthenriquedb%2Freading-large-files-with-node-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthenriquedb%2Freading-large-files-with-node-streams/lists"}