{"id":46296621,"url":"https://github.com/zx2486/dbconnectortoolkit","last_synced_at":"2026-03-04T10:02:23.943Z","repository":{"id":258765677,"uuid":"875639248","full_name":"zx2486/DBConnectorToolkit","owner":"zx2486","description":"This module provides an integrated tool to connect database under different settings, single db, master and slave, with cache, and use message queue to collect writes.","archived":false,"fork":false,"pushed_at":"2025-11-20T07:01:43.000Z","size":119,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-20T08:31:57.746Z","etag":null,"topics":["database","db","postgres","sql"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zx2486.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-20T14:11:20.000Z","updated_at":"2025-11-20T06:53:36.000Z","dependencies_parsed_at":"2025-01-29T17:24:36.330Z","dependency_job_id":"7651ed83-aa87-46dd-ad32-6aae8b8cc77e","html_url":"https://github.com/zx2486/DBConnectorToolkit","commit_stats":null,"previous_names":["zx2486/dbconnectortoolkit"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/zx2486/DBConnectorToolkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zx2486%2FDBConnectorToolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zx2486%2FDBConnectorToolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zx2486%2FDBConnectorToolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zx2486%2FDBConnectorToolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zx2486","download_url":"https://codeload.github.com/zx2486/DBConnectorToolkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zx2486%2FDBConnectorToolkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30078307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T08:01:56.766Z","status":"ssl_error","status_checked_at":"2026-03-04T08:00:42.919Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","db","postgres","sql"],"created_at":"2026-03-04T10:02:23.026Z","updated_at":"2026-03-04T10:02:23.934Z","avatar_url":"https://github.com/zx2486.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DBConnectorToolkit\nThis module provides an integrated tool to connect database under different settings, single db, master with read replica, with redis for caching, and allow using message queue to centralize db writes requests.\nThe main purpose is to reduce code changes on application with growing traffic and upgrading in infra.\n\nDocumentation is available at [https://dbconnectortoolkit.readthedocs.io](https://dbconnectortoolkit.readthedocs.io).\n\nWe have created a demo project ([DBConnectorSampleWeb](https://github.com/zx2486/DBConnectorSampleWeb)) and a demo site ([dbconnectorapi.authpaper.com](https://dbconnectorapi.authpaper.com)) to illustrate how this library works.\n\n## Installation\n```bash\n# Include installation commands here\nnpm install dbconnectortoolkit\n```\n\n## Sample use case\nSuppose you have a postgres database (web) and would like to select data from it.\n```typescript\nconst dbConnector = require('dbconnectortoolkit').default;\n\nconst masterDBConfig = {\n  client: 'pg',\n  endpoint: 'localhost',\n  port: 5432,\n  username: 'your_username',\n  password: 'your_password',\n  database: 'web',\n}\nconst dbConnector = dbConnector(masterDBConfig)\ntry {\n    await dbConnector.connect()\n    // Select all data from the users table\n    const result = await dbConnector.select(\n        [{ table: 'users' }],\n        ['*']\n    )\n    console.log('Data from users table:', result)\n    // also have insert, update, upsert, delete methods and query method for raw queries\n} catch (err) {\n    // throw when database cannot login or something wrong with the query\n    console.error('Error:', err)\n} finally {\n    await dbConnector.disconnect()\n}\n```\n\nNow suppose your application should only have 1-10 concurrent connections to database (so it will not blow up) and they are managed by a pool\n```typescript\nconst masterDBConfig2 = {\n    ...masterDBConfig,\n    minConnection: 1,\n    maxConnection: 10\n}\n```\nDone.\n\nNow your application is getting great, your lead asks you to use replica database for read queries. Instead of rewriting lots of SQL query code to use slave DB client, all you need is\n```typescript\nconst replicaDBConfig = [\n    {\n    ...\n    },...\n]\ndb = dbConnector(masterDBConfig, replicaDBConfig);\n```\nAll select queries will go to replica and write queries will go to master. No other code change is needed.\n\nNow the traffic is even higher, redis is introduced as cache layer. How many code changes?\n```typescript\nconst cacheConfig = {\n    client: 'ioredis',\n    url: 'localhost@6379',\n    dbIndex: 1, // database index, default is 0\n    cacheHeader: 'dbCache:' // optional, default is dbCache:\n}\ndb = dbConnector(masterDBConfig, replicaDBConfig, cacheConfig);\n```\nDone, no other code change is needed. All read query results will be cached in redis.\nThe key in redis will be dbCache:${sha256 hash of the raw query}\n\nIf data should come from database instead of cache, set _getLatest to true.\n\n## Supporting databases, caches and message queues\nDatabase clients supported: pg (PostgreSQL, using mainly pg pool), mariadb (MariaDB or MySQL, using mariadb pool), sqlite (SQLite, using sqlite3)\nCache clients supported: ioredis (Redis), redis (Redis), nodecache (node-cache), memcached (memcached)\nMessage queue clients supported: kafka (Kafka, using kafkajs)\n\n## TODO\nUnit test cases\nSupport of db transaction when message queue is used to centralize db writes.\n\nDatabases pending to support: MSSQL, Aurora DSQL, Oracle\n\nCache pending to support: Valkey\n\nMessage queue pending to support: Amazon MSK, rabbitMQ, Amazon Simple Queue Service SQS\n\n## Contributing\nIf you want to contribute to this project, please submit a pull request or create an issue for discussion. \n\nWorking on README / documentations are also welcomed.\n\n## FAQ / Troubleshooting / Support\nPlease open an issue on github.\n\n## Funding\nYou may buy me a coffee.\n\nBTC: bc1qgl8g2xu3f60lkxgzg80jvykkmf3gywaky3c2tt\n\nETH / BNB / POL (pologon): 0xA5BC03ddc951966B0Df385653fA5b7CAdF1fc3DA\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzx2486%2Fdbconnectortoolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzx2486%2Fdbconnectortoolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzx2486%2Fdbconnectortoolkit/lists"}