{"id":15013019,"url":"https://github.com/pubkey/jsonschema-key-compression","last_synced_at":"2025-05-16T14:05:49.692Z","repository":{"id":34979580,"uuid":"191626893","full_name":"pubkey/jsonschema-key-compression","owner":"pubkey","description":"Compress json-data based on its json-schema while still having valid json","archived":false,"fork":false,"pushed_at":"2025-05-14T11:05:29.000Z","size":340,"stargazers_count":99,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-14T12:28:56.075Z","etag":null,"topics":["compression","json","json-schema","jsonschema","nosql"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pubkey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"pubkey"}},"created_at":"2019-06-12T18:46:27.000Z","updated_at":"2025-05-14T11:05:32.000Z","dependencies_parsed_at":"2023-09-27T13:47:08.710Z","dependency_job_id":"cfc4c17a-c1f1-4a70-8e5f-484773ff94d3","html_url":"https://github.com/pubkey/jsonschema-key-compression","commit_stats":{"total_commits":618,"total_committers":5,"mean_commits":123.6,"dds":0.5307443365695793,"last_synced_commit":"2c1011c42183657196d31141c9d442e2e7f18ea1"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pubkey%2Fjsonschema-key-compression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pubkey%2Fjsonschema-key-compression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pubkey%2Fjsonschema-key-compression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pubkey%2Fjsonschema-key-compression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pubkey","download_url":"https://codeload.github.com/pubkey/jsonschema-key-compression/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254544146,"owners_count":22088807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","json","json-schema","jsonschema","nosql"],"created_at":"2024-09-24T19:43:36.751Z","updated_at":"2025-05-16T14:05:49.673Z","avatar_url":"https://github.com/pubkey.png","language":"TypeScript","funding_links":["https://github.com/sponsors/pubkey"],"categories":[],"sub_categories":[],"readme":"# jsonschema-key-compression\n\nCompress json-data based on its [json-schema](https://json-schema.org/) while still having valid json.\nIt works by compressing long attribute-names into smaller ones and backwards.\n\nFor example this:\n\n```json\n\n{\n    \"firstName\": \"Corrine\",\n    \"lastName\": \"Ziemann\",\n    \"title\": \"Ms.\",\n    \"gender\": \"f\",\n    \"zipCode\": 75963,\n    \"countryCode\": \"en\",\n    \"birthYear\": 1960,\n    \"active\": false,\n    \"shoppingCartItems\": [\n        {\n            \"productNumber\": 29857,\n            \"amount\": 1\n        },\n        {\n            \"productNumber\": 53409,\n            \"amount\": 6\n        }\n    ]\n}\n```\n\nbecomes this:\n\n```json\n{\n    \"|e\": \"Corrine\",\n    \"|g\": \"Ziemann\",\n    \"|j\": \"Ms.\",\n    \"|f\": \"f\",\n    \"|k\": 75963,\n    \"|d\": \"en\",\n    \"|c\": 1960,\n    \"|a\": false,\n    \"|i\": [\n        {\n            \"|h\": 29857,\n            \"|b\": 1\n        },\n        {\n            \"|h\": 53409,\n            \"|b\": 6\n        }\n    ]\n}\n```\n\n## Efficiency\n\nThe efficiency depends on the amount and length of the attribute names. \n* The uncompressed json-object from above has about **230 chars** as string\n* With the key-compression, this can be reduced to **140 chars** which saves about **40%**\n* Just using gzip on the json would result in **180 chars**\n* Using gzip+key-compression ends in a string with only **127 chars**\n\nYou can reproduce these results by running `npm run test:efficiency`.\n\n## Performance\n\nThe compression works pretty fast. Here are some time measurements on a single intel i7 CPU.\n\n* Creating a compression-table from the schema of the object above takes about `0.02ms`\n* Compressing the example-object from above takes about `0.021ms`\n* Decompressing takes about `0.027ms` per object\n\nYou can reproduce these results by running `npm run test:performance`.\n\n## You should use this when\n- you want to save storage space in an NoSQL-database but still want to have valid queryable json-data\n- you transmit many objects in many small requests over the network so that gzip cannot be efficient\n- you want to store json-data inside of the browser-storage (indexedDB or localstorage) and you reach the storage limit\n\n## You should NOT use this when\n- you send many objects in a single request, you should rely on gzip instead\n- you do not want to still have valid json-data, you should use [protobuf](https://developers.google.com/protocol-buffers/) instead\n- you have schema-less data\n\n\n## Comparison with gzip\n\nGzip generates its compression-flags [from the input](https://en.wikipedia.org/wiki/Gzip). This makes it more efficient, the more data is compressed at once. But gzip is less efficient the smaller the dataset is.\nThe key-compression creates the compression-table from the jsonschema up front with has advantages when small pieces of data are compressed.\n\n## Usage\n\n### Install\n\n```bash\nnpm install jsonschema-key-compression --save\n```\n\n### createCompressionTable\nCreates a compression-table from the [json-schema](https://json-schema.org/).\n\n```js\nimport {\n    createCompressionTable\n} from 'jsonschema-key-compression';\nconst compressionTable = createCompressionTable(jsonSchema);\n```\n\n### compressObject\nCompress a json-object based on its schema.\n\n```js\nimport {\n    compressObject\n} from 'jsonschema-key-compression';\nconst compressedObject = compressObject(\n    compressionTable,\n    jsonObject\n);\n```\n\n### decompressObject\nDecompress a compressed object.\n\n```js\nimport {\n    decompressObject\n} from 'jsonschema-key-compression';\nconst jsonObject = decompressObject(\n    compressionTable,\n    compressedObject\n);\n```\n\n### compressedPath\nTransform a chain of json-attributes into its compressed format.\n\n```js\nimport {\n    compressedPath\n} from 'jsonschema-key-compression';\nconst compressed = compressedPath(\n    compressionTable,\n    'whateverNested.firstName'\n); // \u003e '|a.|b'\n```\n\n### decompressedPath\nDecompress a compressed path.\n\n```js\nimport {\n    decompressedPath\n} from 'jsonschema-key-compression';\nconst decompressed = decompressedPath(\n    compressionTable,\n    '|a.|b' // from compressedPath\n); // \u003e 'whateverNested.firstName'\n```\n\n\n### compressQuery\nCompress a [mango-query](https://docs.mongodb.com/manual/tutorial/query-documents/) so that it can run over a NoSQL-database that has stored compressed documents.\n\n```js\nimport {\n    compressQuery\n} from 'jsonschema-key-compression';\nconst compressed = compressQuery(\n    compressionTable,\n    {\n        selector: {\n            active: {\n                $eq: true\n            }\n        },\n        skip: 1,\n        limit: 1,\n        fields: [\n            'id',\n            'name'\n        ],\n        sort: [\n            'name'\n        ]\n    }\n);\n```\n### createCompressedJsonSchema\n\nTransforms a json-schema into a compressed form, so that it can be used to validate compressed objects.\n\n```js\nimport {\n    createCompressedJsonSchema\n} from 'jsonschema-key-compression';\n\n\nconst schema = {\n    type: 'object',\n    properties: {\n        firstName: {\n            type: 'string'\n        }\n    },\n    required: [\n        'firstName'\n    ]\n}\n\nconst compressedSchema = createCompressedJsonSchema(\n    compressionTable,\n    schema\n);\n\nconsole.dir(compressedSchema);\n\n/**\n{\n    type: 'object',\n    properties: {\n        |a: {\n            type: 'string'\n        }\n    },\n    required: [\n        '|a'\n    ]\n}\n */\n\n```\n\n## About\n\nThis module was originally created for the [RxDB compression plugin](https://rxdb.info/key-compression.html) but in theory it can be used in any json based state management system or database.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpubkey%2Fjsonschema-key-compression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpubkey%2Fjsonschema-key-compression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpubkey%2Fjsonschema-key-compression/lists"}