{"id":23773539,"url":"https://github.com/beenotung/better-sqlite3-schema","last_synced_at":"2025-08-28T19:28:55.141Z","repository":{"id":57189556,"uuid":"300890817","full_name":"beenotung/better-sqlite3-schema","owner":"beenotung","description":"helper utilities on top of better-sqlite3-helper","archived":false,"fork":false,"pushed_at":"2025-02-19T07:42:23.000Z","size":92,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-10T05:46:04.684Z","etag":null,"topics":["better-sqlite3","codegeneration","schema","sqlite","typescript-library","utility-library"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/better-sqlite3-schema","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/beenotung.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-03T13:48:26.000Z","updated_at":"2025-02-19T07:42:26.000Z","dependencies_parsed_at":"2024-04-09T19:52:35.992Z","dependency_job_id":"121b6543-bfc0-4251-aad9-2330bc9fe10e","html_url":"https://github.com/beenotung/better-sqlite3-schema","commit_stats":{"total_commits":97,"total_committers":1,"mean_commits":97.0,"dds":0.0,"last_synced_commit":"fdcadef8d787ec21e6b38b108fb65ad410a3a608"},"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"purl":"pkg:github/beenotung/better-sqlite3-schema","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fbetter-sqlite3-schema","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fbetter-sqlite3-schema/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fbetter-sqlite3-schema/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fbetter-sqlite3-schema/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/beenotung","download_url":"https://codeload.github.com/beenotung/better-sqlite3-schema/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fbetter-sqlite3-schema/sbom","scorecard":{"id":230234,"data":{"date":"2025-08-11","repo":{"name":"github.com/beenotung/better-sqlite3-schema","commit":"b0d59d1d23730bf988de39f8d855b0fc2d58b590"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: BSD 2-Clause \"Simplified\" License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-17T04:43:01.902Z","repository_id":57189556,"created_at":"2025-08-17T04:43:01.902Z","updated_at":"2025-08-17T04:43:01.902Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272542682,"owners_count":24952570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["better-sqlite3","codegeneration","schema","sqlite","typescript-library","utility-library"],"created_at":"2025-01-01T05:41:06.896Z","updated_at":"2025-08-28T19:28:55.080Z","avatar_url":"https://github.com/beenotung.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# better-sqlite3-schema\n\nMigrate (nested and multi-dimensional) json data to/from sqlite database with better-sqlite3-helper\n\n[![npm Package Version](https://img.shields.io/npm/v/better-sqlite3-schema.svg)](https://www.npmjs.com/package/better-sqlite3-schema)\n\n## Usage Example\n\nSample json data type:\n\n```typescript\ninterface Thread {\n  tid: number\n  subject: string\n  uid: string\n  author: string\n  posts: Post[]\n  tags: string[]\n}\n\ninterface Post {\n  pid: number\n  uid: string\n  author: string\n  content: string\n  imgs: string[]\n}\n```\n\nSample table schema:\n\n```typescript\nimport { TableSchema } from 'better-sqlite3-schema'\n\nconst threadSchema: TableSchema = {\n  table: 'thread',\n  fields: {\n    tid: 'integer',\n    subject: 'text',\n    uid: 'integer',\n  },\n  refFields: ['type'],\n}\n\nconst threadTagSchema: TableSchema = {\n  table: 'thread_tag',\n  fields: {\n    tid: 'integer',\n  },\n  refFields: ['tag'],\n}\n\nconst postSchema: TableSchema = {\n  table: 'post',\n  fields: {\n    pid: 'integer',\n    tid: 'integer',\n    uid: 'integer',\n    content: 'text',\n  },\n}\n\nconst postImgSchema: TableSchema = {\n  table: 'post_img',\n  fields: {\n    pid: 'integer',\n  },\n  refFields: ['img'],\n}\n```\n\n### Functional Approach (compose at runtime)\n\nThe functional approach allows one to compose customizable helper functions at runtime.\n\nExplore the dataset and auto built schema with\n\n- `makeSchemaScanner()`\n\nCompose insert functions with\n\n- `makeInsertRowFnFromSchema()`\n- `makeDeduplicatedInsertRowFnFromSchema()`\n\nCompose select functions with\n\n- `makeSelectRowFnFromSchema()`\n- `makeSelectRefFieldArray()`\n- `makeGetRefValueFnFromSchema()`\n\nDetail example see `makePredefinedInsertRowFn()` and `makeGeneralInsertRowFn()` in [functional-test.ts](./example/functional-test.ts)\n\n### Code Generation Approach (compose at build-time)\n\nThe code generation approach allows one to compose customizable helper functions at build-time. Which can archive ~50% speed up compared to the runtime composing.\n\n## Benchmark\n\n### Sample 1: HTTP Proxy Log\n\n8GiB of HTTP proxy server log.\nEach line is a compact json text.\n\nSample text:\n\n```json\n{\n  \"timestamp\": 1600713130016,\n  \"type\": \"request\",\n  \"userAgent\": \"Mozilla/5.0 (Linux; Android 10; LIO-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Mobile Safari/537.36\",\n  \"referer\": \"https://www.example.net/sw.js\",\n  \"protocol\": \"https\",\n  \"host\": \"www.example.net\",\n  \"method\": \"GET\",\n  \"url\": \"/build/p-7794655c.js\"\n}\n```\n\nWhen stored into sqlite3, the data are normalized into multiple tables to avoid duplication, e.g. only storing the full text of each type of user agent and url once.\n\nFile size in varies format:\n\n| storage                        | size  | size compared with plain text | Remark                        |\n| ------------------------------ | ----- | ----------------------------- | ----------------------------- |\n| plain text                     | 8256M | -                             |                               |\n| sqlite without index           | 920M  | 11.1%                         |                               |\n| zip of non-indexed sqlite file | 220M  | 2.7%                          | 23.9% of sqlite3 file         |\n| sqlite with indices            | 1147M | 13.9%                         | +24% of sqlite file           |\n| zip of indexed sqlite file     | 268M  | 3.2%                          | 23.4% of indexed sqlite3 file |\n\nTime used to import:\n\n- 6 minutes 10 seconds: with inlined helper functions with code generation\n- 14 minutes: with runtime-composed helper functions\n\nOptimization used:\n\n- code generation from schema\n- bulk insert (batch each 8K items with a transaction)\n- cache id of normalized, repeatable values (with js object)\n- create unique index on normalized values\n- `PRAGMA synchronous = OFF`\n- `PRAGMA journal_mode = MEMORY`\n- `PRAGMA cache_size = ${(200 * 1000 ** 2) / 4}`\n  (default page size is 4K, we largely increase the cache_size to avoid massive tedious disk write)\n\n**Remark**:\n\nUsing index increases the file size by 1/4, but hugely speeds up the import process.\n\nTo archive the best of both aspects, create indices during import;\nand remove indices (then VACUUM) for archive file.\n\nIt takes 4.9s to build the indices;\nand 16.3s to vacuum the database after removal of indices.\n\n### Sample 2: Online Forum Data\n\n291119 sample json data crawled from online forum (threads and posts)\n\nTotal size: 843M\n\nThe objects have consistent shape.\n\nSome data are duplicated, e.g. user name, and some common comments.\n\nSame as the dataset used in [binary-object](https://github.com/beenotung/binary-object)\n\nFile size in varies format:\n\n| storage                      | size |\n| ---------------------------- | ---- |\n| json text                    | 843M |\n| sqlite3 with index           | 669M |\n| sqlite3 without index        | 628M |\n| zip of sqlite3 without index | 171M |\n\n**Remark**:\nThe data in sqlite3 are normalized to avoid duplication\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeenotung%2Fbetter-sqlite3-schema","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbeenotung%2Fbetter-sqlite3-schema","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeenotung%2Fbetter-sqlite3-schema/lists"}