{"id":25972825,"url":"https://github.com/kzolti/sqlite3-pb-ext-gen","last_synced_at":"2025-03-05T01:17:33.794Z","repository":{"id":237066335,"uuid":"793698336","full_name":"kzolti/sqlite3-pb-ext-gen","owner":"kzolti","description":"SQLite3 Protobuf extension generator","archived":false,"fork":false,"pushed_at":"2025-02-02T17:29:58.000Z","size":406,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-04T04:15:50.325Z","etag":null,"topics":["developer-tools","protobuf","protocol-buffers","sqlite","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kzolti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"kzolti"}},"created_at":"2024-04-29T17:45:54.000Z","updated_at":"2025-02-28T17:18:45.000Z","dependencies_parsed_at":"2024-05-07T16:39:46.597Z","dependency_job_id":"44a122aa-de63-4b0c-9a5a-fabbb88432bf","html_url":"https://github.com/kzolti/sqlite3-pb-ext-gen","commit_stats":null,"previous_names":["kzolti/sqlite3-pb-ext-gen"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kzolti%2Fsqlite3-pb-ext-gen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kzolti%2Fsqlite3-pb-ext-gen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kzolti%2Fsqlite3-pb-ext-gen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kzolti%2Fsqlite3-pb-ext-gen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kzolti","download_url":"https://codeload.github.com/kzolti/sqlite3-pb-ext-gen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241945530,"owners_count":20046870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["developer-tools","protobuf","protocol-buffers","sqlite","sqlite3"],"created_at":"2025-03-05T01:17:33.203Z","updated_at":"2025-03-05T01:17:33.781Z","avatar_url":"https://github.com/kzolti.png","language":"TypeScript","funding_links":["https://github.com/sponsors/kzolti"],"categories":[],"sub_categories":[],"readme":"# sqlite3-pb-ext-gen \n**SQLite3 Protobuf extension generator**\n\nThe Node.js program generates the source code in C/C++ from the *.proto files to be used in the database. After the build, an extension that can be loaded into the database manager is created. Then, the protobuf data stored in binary can be read using the **pb_extract** function and the **pb_field** and **pb_each** virtual tables. It is designed to be efficient and fast, as it avoids using the descriptor and reflection APIs . This extension builds directly on the C++ code generated with 'protoc'.\n\n### Table of Contents\n0. [Platform Support](#0-platform-support)\n1. [Installation](#1-installation)\n2. [Extension source code generation](#2-extension-source-code-generation)\n3. [Making an extension](#3-making-an-extension)\n3. [SQL Function and Virtual Tables](#3-sql-function-and-virtual-tables)\n4. [SQL example](#4-sql-example)\n5. [Performance](#5-performance)\n7. [Conclusion](#7-conclusion)\n\n### 0. Platform Support\nLinux x86_64  \nThis project has been exclusively tested on Linux operating systems. We do not guarantee its functionality on other operating systems.\n\n### 1. Installation\n``` \nnpm i -g sqlite3-pb-ext-gen\n```\n##### Check protobuf-compiler dependency:\n``` protoc --version```\n\u003c!---\n##### Check sqlite3-pb-ext-gen:\n``` sqlite3-pb-ext-gen --version```\n##### Update for sqlite3-pb-ext-gen:\n``` npm update -g sqlite3-pb-ext-gen```\n##### View sqlite3-pb-ext-gen:\n``` npm view sqlite3-pb-ext-gen```\n---\u003e\n### 2. Extension source code generation\n\nexample command: \n```\nsqlite3-pb-ext-gen --proto_path=/tmp/proto --out_path=/tmp/out addressbook.proto\n```\nusable arguments:\n```\nUsage: sqlite3-pb-ext-gen [OPTION] PROTO_FILES\ndependencies: protoc\nParse PROTO_FILES and generate output based on the options given:\n  -v, --version               Show sqlite3-pb-ext-gen version and exit.\n  -h, --help                  Show this text and exit.\n  -oPATH, --out_path=PATH     Generated SQLite3 Protobuf extension path.\n  -IPATH, --proto_path=PATH   Specify the directory in which to search for\n                              imports.  May be specified multiple times;\n                              directories will be searched in order.  If not\n                              given, the current working directory is used.\n                              If not found in any of the these directories,\n                              the --descriptor_set_in descriptors will be\n                              checked for required proto file.\n  @\u003cfilename\u003e                 Read options and filenames from file. If a\n                              relative file path is specified, the file will\n                              be searched in the working directory.\n                              The --proto_path option will not affect how this\n                              argument file is searched. Content of the file\n                              will be expanded in the position of @\u003cfilename\u003e\n                              as in the argument list. Note that shell \n                              expansion is not applied to the content of the \n                              file (i.e., you cannot use quotes, wildcards,\n                              escapes, commands, etc.). Each line corresponds\n                              to a single argument, even if it contains spaces. \n\n```\n\n###  3. Making an extension \n####  System SQLite3 \nThis command creates an extension for the sqlite3 version installed on the system\n```\nmkdir build \u0026\u0026 cd build  \u0026\u0026 \\\ncmake .. \u0026\u0026 cmake --build . \n```\n#### Official SQLite3 versions\nBy using the CUSTOM_SQLITE3=ON switch, it is possible to create a plugin for other versions of sqlite3.  If you want to use the official sqlite version, use the variable SQLITE_VERSION.\nIt is important to check that the SQLite Compile-time Options are correct for your project. You can change the compile time options in \u003c--out_path\u003e/sqlite3/build_options.txt. \n\n```\nmkdir build \u0026\u0026 cd build  \u0026\u0026 \\\ncmake -DCUSTOM_SQLITE3=ON -DSQLITE_VERSION=3.44.0 .. \u0026\u0026 cmake --build .\n```\n#### SQLite3 based database\nFor other SQLite3-based databases, you'll need to manually place the files required to create the extension in the \u003cout_path\u003e/sqlite3 directory.\n  - \u003cout_path\u003e/sqlite3/include/ - sqlite3ext.h, sqlite3.h\n  - \u003cout_path\u003e/sqlite3/target/ - libsqlite3.a\n```\nmkdir build \u0026\u0026 cd build  \u0026\u0026 \\\ncmake -DCUSTOM_SQLITE3=ON .. \u0026\u0026 cmake --build .\n```\n### 3. SQL Function and Virtual Tables\n\nThe extension introduces `pb_extract` function and two virtual tables: `pb_field` and `pb_each`.\n\n#### 3.1.  pb_extract\n The `pb_extarct` function extracts a specific value from a protobuf message based on a given path. It does not return a pointer of the Message type, for this use the `pb_each` or `pb_field` virtual table module. \n \n ##### Parameters of the pb_extract:\n  - `message`: A `sqlite3_value_blob` or `sqlite3_value_pointer` representing the protobuf message.\n  - `message_type`: The fully qualified name of the protobuf message type name.\n  - `path`: The path to the desired value within the protobuf message.\n\nValues returned with the pb_extract function:\n| protobuf type | sqlite3_value |\n|:----------------:|:---------------:|\n| `Message`       | `sqlite3_value_error`    |\n| `bool, enum, fixed32, int32, sfixed32, sint32, uint32` | `sqlite3_value_int` |\n| `fixed64, int64, sfixed64, sint64, uint64` | `sqlite3_value_int64` |\n| `double, float` | `sqlite3_value_double` |\n| `string` | `sqlite3_value_text` |\n| `bytes` | `sqlite3_value_blob` |\n| `oneof, map` | not supported |\n\n #### 3.2. pb_field\n The `pb_filed` virtual table returns the details of a single field within a protobuf message. \n\n #### 3.3. pb_each\n The `pb_each` virtual table returns information about repeated fields in a protobuf message.\n \n##### Parameters of virtual tables\n- `message`: Either `sqlite3_value_blob` or `sqlite3_value_pointer` representing the protobuf message.\n- `message_type`: String specifying the protobuf message type.\n- `field`: String containing the field name. If empty string, the initialized message is returned.\n\n##### Fields of virtual tables\npb_each virtual table provides information about repeated fields in a protobuf message. The columns provide the following details:\n\n - `value`: The value of the field. Its types can be: \n\n| protobuf type | sqlite3_value |\n|:----------------:|:---------------:|\n| `Message` | `sqlite3_value_pointer` |\n| `bool, enum, fixed32, int32, sfixed32, sint32, uint32` | `sqlite3_value_int` |\n| `fixed64, int64, sfixed64, sint64, uint64` | `sqlite3_value_int64` |\n| `double, float` | `sqlite3_value_double` |\n| `string` | `sqlite3_value_text` |\n| `bytes` | `sqlite3_value_blob` |\n| `oneof, map` | not supported |\n - `repeated_id`: This column represents the index of the repeating field.(pb_each only)\n\n - `name`: This column displays the name of the field.\n - `number`: The field number as specified in the protobuf definition file.\n - `label`: The field label (optional or repeated). Note that maps are not supported.\n - `type`: This column indicates the type of the field, as defined in the protobuf definition (message, enum, or scalar type).\n - `type_name`: In the case of enum or message types, this column contains the type name.\n\nThese columns provide a comprehensive overview of the fields\n\n### 4. SQL example\n\nSuppose you have a table named `person` with the following structure:\n\n```sql\nCREATE TABLE person (id INTEGER PRIMARY KEY, proto BLOB); /* proto type: [tutorial.Person] https://github.com/protocolbuffers/protobuf/blob/main/examples/addressbook.proto */\n```\n\nYou can use the extension to retrieve phone numbers of a person with the name \"John Doe 2005\" and the phone type \"HOME\" using the following query:\n\n```sql\nSELECT\n  pb_extract(phone.value, phone.type_name, 'number') phone_number\nFROM person AS person_t\n, pb_field(proto, 'tutorial.Person','') person\n, pb_each(person.value, person.type_name, 'phones') phone\n, pb_field(phone.value, phone.type_name, 'type') phone_type\nWHERE\n  pb_extract(proto,'tutorial.Person','name')='John Doe 2005'\n  AND phone_type.type_name='tutorial.Person.PhoneType.HOME';\n```\nYou can find the [complete cpp example](https://github.com/kzolti/sqlite3-pb-ext-example) in the examples directory.\n```\ngit submodule update --init \\\n\u0026\u0026 cd example\n```\n\n### 5. Performance\n\nThe extension's reliance on the protobuf compiler's generated C++ code ensures superior performance compared to using descriptor and reflection APIs.\n\n### 7. Conclusion\n\nThe extension created with sqlite3-pb-ext-gen allows you to work with Protocol Buffers data within SQLite databases, providing a seamless and efficient solution for querying binary data. Make sure to follow the recommended practices for Protobuf (https://protobuf.dev/programming-guides/dos-donts/).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkzolti%2Fsqlite3-pb-ext-gen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkzolti%2Fsqlite3-pb-ext-gen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkzolti%2Fsqlite3-pb-ext-gen/lists"}