{"id":24935809,"url":"https://github.com/palcarazm/batchjs-data","last_synced_at":"2026-03-05T02:03:20.072Z","repository":{"id":273074224,"uuid":"918628843","full_name":"palcarazm/batchjs-data","owner":"palcarazm","description":"Extension of Batch JS adding data storage support for databases.","archived":false,"fork":false,"pushed_at":"2026-02-10T01:14:31.000Z","size":133,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":1,"default_branch":"version/v1","last_synced_at":"2026-02-17T08:59:38.421Z","etag":null,"topics":["batch","database","database-batch","framework","node-stream"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/palcarazm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["palcarazm"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2025-01-18T12:48:39.000Z","updated_at":"2025-09-09T16:40:34.000Z","dependencies_parsed_at":"2025-01-18T14:29:31.139Z","dependency_job_id":"827ae376-1c8f-4c25-9d8e-9c414d997375","html_url":"https://github.com/palcarazm/batchjs-data","commit_stats":null,"previous_names":["palcarazm/batchjs-sql","palcarazm/batchjs-data"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/palcarazm/batchjs-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/palcarazm%2Fbatchjs-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/palcarazm%2Fbatchjs-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/palcarazm%2Fbatchjs-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/palcarazm%2Fbatchjs-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/palcarazm","download_url":"https://codeload.github.com/palcarazm/batchjs-data/tar.gz/refs/heads/version/v1","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/palcarazm%2Fbatchjs-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30106156,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T01:39:18.192Z","status":"online","status_checked_at":"2026-03-05T02:00:06.710Z","response_time":93,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","database","database-batch","framework","node-stream"],"created_at":"2025-02-02T15:57:01.819Z","updated_at":"2026-03-05T02:03:20.062Z","avatar_url":"https://github.com/palcarazm.png","language":"TypeScript","readme":"[![GitHub license](https://img.shields.io/github/license/palcarazm/batchjs-data.svg?color=informational)](https://github.com/palcarazm/batchjs-data/blob/version/v1/LICENSE)\n[![Latest release](https://img.shields.io/github/package-json/v/palcarazm/batchjs-data/version/v1?logo=github)](https://github.com/palcarazm/batchjs-data/releases)\n[![NPM Badge](https://img.shields.io/npm/dm/batchjs-data?logo=npm)](https://www.npmjs.com/package/batchjs-data)\n[![CI](https://img.shields.io/github/actions/workflow/status/palcarazm/batchjs-data/ci-workflow.yml?branch=version/v1\u0026label=CI\u0026logo=Node.js\u0026logoColor=white)](https://github.com/palcarazm/batchjs-data/actions/workflows/ci-workflow.yml)\n[![Coverage](https://coveralls.io/repos/github/palcarazm/batchjs-data/badge.svg?branch=version/v1)](https://coveralls.io/github/palcarazm/batchjs-data?branch=version/v1)\n[![Funding](https://img.shields.io/badge/sponsor-30363D?style=flat\u0026logo=GitHub-Sponsors\u0026logoColor=#white)](https://github.com/sponsors/palcarazm)\n\n# BatchJS-Data\n\nExtension of [Batch JS](https://github.com/palcarazm/batchjs) adding data storage support for databases.\n\n---\n\n- [BatchJS-Data](#batchjs-data)\n- [Download](#download)\n  - [NPM](#npm)\n  - [Yarn](#yarn)\n- [Usage](#usage)\n- [Documentation](#documentation)\n- [Collaborators welcome!](#collaborators-welcome)\n\n---\n\n# Download\n\n[![Latest release](https://img.shields.io/github/package-json/v/palcarazm/batchjs-data/version/v1?logo=github)](https://github.com/palcarazm/batchjs-data/releases)\n\n## NPM\n\n[![NPM Badge](https://img.shields.io/npm/dm/batchjs-data?logo=npm)](https://www.npmjs.com/package/batchjs-data)\n\n```sh\nnpm install batchjs-data --no-optional\nnpm install sqlite sqlite3  #For SQLite implementation\nnpm install mariadb         #For MariaDB implementation\nnpm install mysql2          #For MySQL implementation\nnpm install pg @types/pg    #For PostgreSQL implementation\n```\n\n## Yarn\n\n```sh\nyarn add batchjs-data --no-optional\nyarn add mariadb         #For MariaDB implementation\nyarn add mysql2          #For MySQL implementation\nyarn add pg @types/pg    #For PostgreSQL implementation\nyarn add sqlite sqlite3  #For SQLite implementation\n```\n\n\n# Usage\n\n1. Create your reader\n    ```typescript\n    import sqlite3 from \"sqlite3\";\n    import  {open} from \"sqlite\";\n    import { SqliteBatchEntityReader } from \"batchjs-data/sqlite\";\n    import { UserDTO } from \"./UserDTO\";\n\n    export class UserBatchReader extends SqliteBatchEntityReader\u003cUserDTO,UserDTO\u003e {\n        constructor(options:{batchSize:number,query?:string}) {\n            super({\n                batchSize: options.batchSize,\n                dbConnectionFactory: () =\u003e { return open({filename: './database.db', driver: sqlite3.Database});},\n                query: options.query || \"SELECT id, username FROM users\",\n                rowToEntity: (row: UserDTO) =\u003e row\n            });\n        }\n    }\n    ```\n2. Create your writer\n    ```typescript\n    import sqlite3 from \"sqlite3\";\n    import sqlite, {open} from \"sqlite\";\n    import { SqliteBatchEntityWriter } from \"batchjs-data/sqlite\";\n    import { UserDTO } from \"./UserDTO\";\n\n    export class UserBatchWriter extends SqliteBatchEntityWriter\u003cUserDTO\u003e {\n        constructor(options:{batchSize:number}){\n            super({\n                batchSize: options.batchSize,\n                dbConnectionFactory: () =\u003e { return open({filename: './database.db', driver: sqlite3.Database});},\n                prepareStatement: \"INSERT INTO users (id, username) VALUES (@id, @username)\",\n                saveEntity:(entity: UserDTO, stmt: sqlite.Statement)=\u003estmt.all\u003cvoid\u003e({'@id': entity.id, '@username': entity.username})\n            });\n        }\n    }\n    ```\n3. Use them in your BatchJS Job\n    ```typescript\n    import { Job, Step } from \"batchjs\";\n\n    // Implement a step\n    class StepImplementation extends Step {\n        // Set a name to the step\n        constructor(name: string = \"DemoStep\") {\n            super(name);\n        }\n\n        // Implement the reader to load step data source\n        protected _reader() {\n            return new UserBatchReader({batchSize:2});\n        }\n\n        // Implement the processors to transform data sequently using our streams or your own streams\n        protected _processors() {\n            const opts: TransformOptions = {\n                objectMode: true,\n                transform(\n                    chunk: unknown,\n                    encoding: BufferEncoding,\n                    callback: TransformCallback\n                ) {\n                    this.push(chunk);\n                    callback();\n                },\n            };\n            return [new Transform(opts), new Transform(opts)];\n        }\n\n        // Implement the write to stock final step data\n        protected _writer() {\n            return new UserBatchWriter({batchSize:2})\n        }\n    }\n\n    // Implement a Job\n    class JobImplementation extends Job {\n        // Implement to set the steps to be sequently executed.\n        protected _steps() {\n            return [new StepImplementation(), new StepImplementation()];\n        }\n    }\n\n    // Instance the Job\n    const job = new JobImplementation(\"My job\");\n\n    // Set events listener\n    job.on(\"stepStart\", (step: step) =\u003e {\n        console.log(`Starting step ${step.name}`);\n    });\n\n    // Launch the job\n    job.run()\n        .then(() =\u003e {\n            console.log(\"Job completed successfully\");\n        })\n        .catch((error) =\u003e {\n            console.log(\"Job completed with errors\");\n        });\n    ```\n\n# Documentation\n\n- [Core API](./docs/common-api.md)\n- [MariaDB API](./docs/mariadb-api.md)\n- [MySQL API](./docs/mysql-api.md)\n- [PostgreSQL API](./docs/postgresql-api.md)\n- [SQLite API](./docs/sqlite-api.md)\n\n# Collaborators welcome!\n\n- ¿Do you like the project? Give us a :star: in [GitHub](https://github.com/palcarazm/batchjs-data).\n- :sos: ¿Do you need some help? Open a discussion in [GitHub help wanted](https://github.com/palcarazm/batchjs/discussions/new?category=q-a)\n- :bug: ¿Do you find a bug? Open a issue in [GitHub bug report](https://github.com/palcarazm/batchjs-data/issues/new?assignees=\u0026labels=bug\u0026projects=\u0026template=01-BUG_REPORT.yml)\n- :bulb: ¿Do you have a great idea? Open a issue in [GitHub feature request](https://github.com/palcarazm/batchjs-data/issues/new?assignees=\u0026labels=feature\u0026projects=\u0026template=02-FEATURE_REQUEST.yml)\n- :computer: ¿Do you know how to fix a bug? Open a pull request in [GitHub pull request](https://github.com/palcarazm/batchjs-data/compare).\n- ¿Do you know a security issue? Take a read to our [security strategy](https://github.com/palcarazm/batchjs-data/blob/version/v1/SECURITY.md).\n\n[![GitHub Contributors](https://contrib.rocks/image?repo=palcarazm/batchjs-data)](https://github.com/palcarazm/batchjs-data/graphs/contributors)\n\n[Subscribe our code of conduct](https://github.com/palcarazm/batchjs-data/blob/version/v1/CODE_OF_CONDUCT.md) and follow the [Contribution Guidelines](https://github.com/palcarazm/batchjs-data/blob/version/v1/CONTRIBUTING.md).","funding_links":["https://github.com/sponsors/palcarazm"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpalcarazm%2Fbatchjs-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpalcarazm%2Fbatchjs-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpalcarazm%2Fbatchjs-data/lists"}