{"id":13691021,"url":"https://github.com/KxSystems/protobufkdb","last_synced_at":"2025-05-02T11:32:00.497Z","repository":{"id":47029178,"uuid":"272713863","full_name":"KxSystems/protobufkdb","owner":"KxSystems","description":"Kdb+ integration with Google Protocol Buffers","archived":false,"fork":false,"pushed_at":"2023-11-08T14:43:01.000Z","size":178,"stargazers_count":14,"open_issues_count":1,"forks_count":9,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-07T16:09:54.043Z","etag":null,"topics":["kdb","kdb-q","protobuf","protocol-buffers"],"latest_commit_sha":null,"homepage":"https://code.kx.com/q/interfaces","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KxSystems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-06-16T13:21:35.000Z","updated_at":"2024-11-02T11:23:09.000Z","dependencies_parsed_at":"2023-01-23T09:15:47.113Z","dependency_job_id":"1cc88225-74b4-4ed2-9f11-a89d9d79219a","html_url":"https://github.com/KxSystems/protobufkdb","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Fprotobufkdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Fprotobufkdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Fprotobufkdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Fprotobufkdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KxSystems","download_url":"https://codeload.github.com/KxSystems/protobufkdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224308829,"owners_count":17290071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kdb","kdb-q","protobuf","protocol-buffers"],"created_at":"2024-08-02T17:00:38.745Z","updated_at":"2024-11-12T16:32:06.717Z","avatar_url":"https://github.com/KxSystems.png","language":"C++","funding_links":[],"categories":[":fireworks: Fusion interfaces"],"sub_categories":[],"readme":"# protobufkdb\r\n\r\n[![GitHub release (latest by date)](https://img.shields.io/github/v/release/kxsystems/protobufkdb?include_prereleases)](https://github.com/kxsystems/protobufkdb/releases) [![Travis (.org) branch](https://img.shields.io/travis/kxsystems/protobufkdb/master?label=travis%20build)](https://travis-ci.org/kxsystems/protobufkdb/branches)\r\n\r\n\r\n\r\nThis interface allows kdb+ users to parse data which has been encoded using Google Protocol Buffers [(protobuf)](https://github.com/protocolbuffers/protobuf) into kdb+ according to the proto schema and serialise it back to the encoded wire format. The interface utilises the `libprotobuf` descriptor and reflection C++ [APIs](https://developers.google.com/protocol-buffers/docs/reference/cpp#google.protobuf).\r\n\r\nThis is part of the [*Fusion for kdb+*](http://code.kx.com/q/interfaces/#fusion-interfaces) interface collection.\r\n\r\n\r\n\r\n## New to kdb+ ?\r\n\r\nKdb+ is the world’s fastest timeseries database, optimized for ingesting, analyzing and storing massive amounts of structured data. To  get started with kdb+, visit https://code.kx.com/q/learn/ for downloads and developer information. For general information, visit https://kx.com/\r\n\r\n\r\n## New to Protocol Buffers ?\r\n\r\nProtocol Buffers (Protobuf) is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. It is used both in the development of programs which are required to communicate over the wire or for data storage. Developed originally for internal use by Google, it is released under an open source BSD license. The core principle behind this data format is to allow a user to define the expected structure of their data and incorporate this within specially generated source code. This allows a user to easily read and write structured data to and from a variety of languages. \r\n\r\n:globe_with_meridians:\r\n[Protobuf documentation](https://developers.google.com/protocol-buffers/docs/overview)\r\n\r\n\r\n\r\n## Importing protobuf schema files\r\n\r\nProtobuf messages are defined in a `.proto` schema file and these message definitions must be imported into the interface in order for it to be able to create messages of those types.  The interface supports two ways to do this (or a combination of both) but the method used will impact how protobufkdb should be installed.\r\n\r\n### 1.  Compiling in the generated message definitions\r\n\r\nNormally the Protocol Buffers compiler is used to generate source code from the `.proto` schema files which is then compiled in to the binary:\r\n\r\n`protoc` compiles your message definitions based on a file defined as:\r\n\r\n```bash\r\n\u003cschema\u003e.proto\r\n```\r\n\r\nproducing both a C++ source and header file defined as:\r\n\r\n```bash\r\n\u003cschema\u003e.pb.cc\r\n\u003cschema\u003e.pb.h\r\n```\r\n\r\nThese files contain the classes and metadata which describe the schema and the functionality required to serialize to and parse from this schema.\r\n\r\nThis mechanism is more performant but does require that protobufkdb be built from source since the binary needs to be rebuilt to change the statically available messages.\r\n\r\n### 2.  Dynamically importing the message definitions at runtime\r\n\r\nTo provide greater flexibility and usability it is also possible to dynamically import a `.proto` schema file at runtime from within the q session.  Imported message definitions can then be used subsequently by the interface and behave similarly to compiled in ones (the import procedure leverages the same functionality as used by the protobuf compiler).\r\n\r\nIf only dynamically imported message definitions are required then the packaged installation of protobufkdb can be used.  However, importing message definitions is less performant - in addition to the one-off import cost, there is also an overhead from the subsequent use of these dynamically created messages (approx. 10% for parsing, 20% for serializing).  Alternatively a hybrid approach can be employed where dynamic messages are used during development until the schemas are finalized, at which point they are compiled into the interface.\r\n\r\n\r\n\r\n## Installation\r\n\r\n### Requirements\r\n\r\n- kdb+ ≥ 3.5 64-bit (Linux/macOS/Windows)\r\n- protobuf ≥ 3.0 (recommended [^1])\r\n- C++11 or later [^2] \r\n- CMake ≥ 3.1 [^2]\r\n\r\n[^1]: Protocol Buffers language version 3 ([proto3](https://developers.google.com/protocol-buffers/docs/proto3)) simplifies the protocol buffer language, both for ease of use and to make it available in a  wider range of programming languages.  However, schemas defined in [proto2](https://developers.google.com/protocol-buffers/docs/cpptutorial) should also be supported.\r\n\r\n[^2]: Required when building from source\r\n\r\n### Installing a release\r\n\r\nThe protobufkdb releases are linked statically against libprotobuf to avoid potential C++ ABI [compatibility issues](https://github.com/protocolbuffers/protobuf/tree/master/src#binary-compatibility-warning) with different versions of libprotobuf.  Therefore it is unnecessary to install protobuf separately when used a packaged release.\r\n\r\n1.  Download a release from [here](https://github.com/KxSystems/protobufkdb/releases)\r\n\r\n2.  Install required q executable script `q/protobufkdb.q` and binary file `lib/protobufkdb.(so|dll)` to `$QHOME` and `$QHOME/[mlw](64)`, by executing the following from the Release directory\r\n\r\n    ```bash\r\n    ## Linux/MacOS\r\n    chmod +x install.sh \u0026\u0026 ./install.sh\r\n   \r\n    ## Windows\r\n    install.bat\r\n    ```\r\n\r\n3.  To use the KdbTypeSpecifier field option (described below) with dynamic messages then the directory containing `kdb_type_specifier.proto` must be specified to the interface as an import search location.  In the release package `kdb_type_specifier.proto` (and its dependencies) are found in the `proto` subdirectory.  Import paths can be relative or absolute.  For example, if the q session is started from the root of the release package run:\r\n\r\n    ```\r\n    .protobufkdb.addProtoImportPath[\"proto\"]\r\n    ```\r\n\r\n### Building and installing from source\r\n\r\n#### Third-party library installation\r\n\r\nProtobufkdb requires the full Protocol Buffers runtime (protoc compiler, libprotobuf and its header files) to be installed on your system.  Many packaged installations only contain a subset of the required functionality or use an incompatible build.  Furthermore, version mismatches can occur between protoc and libprotobuf if a new installation is applied on top of an existing one.  \r\n\r\nIt is therefore recommend that the protocol buffer runtime is built from source and installed to a non-system directory.  This directory can then be specified to the protobufkdb build so it will use that Protocol Buffers installation in preference to any existing system installs.\r\n\r\n#### Building Protocol Buffers - Linux/macOS\r\n\r\nThe tools required to build Protocol Buffers from source on Linux/macOS are described [here](https://github.com/protocolbuffers/protobuf/blob/master/src/README.md).\r\n\r\nHowever, do not build Protocol Buffers using Google's `configure` script, since that will create a debug version of `libprotobuf.a` which protobufkdb links against.  Rather, follow the instructions below to build Protocol Buffers using CMake with the correct compiler flags and install it to a non-system directory.\r\n\r\nClone the Protocol Buffers source from GitHub:\r\n\r\n```bash\r\ngit clone https://github.com/protocolbuffers/protobuf.git\r\ncd protobuf\r\n```\r\n\r\nCreate an install directory and set an environment variable to this directory (this is used again later when building protobufkdb):\r\n\r\n```bash\r\nmkdir install\r\nexport PROTOBUF_INSTALL=$(pwd)/install\r\n```\r\n\r\nCreate the CMake build directory and generate the build files, specifying position independent code (otherwise symbol relocation errors will occur during linking of protobufkdb):\r\n\r\n```bash\r\nmkdir cmake/build\r\ncd cmake/build\r\ncmake -DCMAKE_BUILD_TYPE=Release -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_PREFIX=$PROTOBUF_INSTALL ..\r\n```\r\n\r\nFinally build and install Protocol Buffers:\r\n\r\n```bash\r\ncmake --build . --config Release\r\ncmake --build . --config Release --target install\r\n```\r\n\r\n#### Building Protocol Buffers - Windows\r\n\r\nThe tools required to build Protocol Buffers from source on Windows are described [here](https://github.com/protocolbuffers/protobuf/blob/master/cmake/README.md) and details on how to setup your environment to build with VS2019 are [here](https://github.com/protocolbuffers/protobuf/blob/master/cmake/README.md#environment-setup). Then follow the below instructions to build a Release version Protocol Buffers and install it to a non-system directory.\r\n\r\nFrom a Visual Studio command prompt, clone the Protocol Buffers source from github:\r\n\r\n```shell\r\nC:\\Git\u003e git clone https://github.com/protocolbuffers/protobuf.git\r\nC:\\Git\u003e cd protobuf\r\n```\r\n\r\nCreate an install directory and set an environment variable to this directory (substituting the correct absolute path as appropriate).  This environment variable is used again later when building protobufkdb:\r\n\r\n```shell\r\nC:\\Git\\protobuf\u003e mkdir install\r\nC:\\Git\\protobuf\u003e set PROTOBUF_INSTALL=C:\\Git\\protobuf\\install\r\n```\r\n\r\nCreate the CMake build directory (note that if you also wish to build a Debug version of Protocol Buffers then a second CMake build directory is required):\r\n\r\n```shell\r\nC:\\Git\\protobuf\u003e mkdir cmake\\release_build\r\nC:\\Git\\protobuf\u003e cd cmake\\release_build\r\n```\r\n\r\nGenerate the build files (this will default to using the Visual Studio CMake generator when run from a VS command prompt):\r\n\r\n```shell\r\nC:\\Git\\protobuf\\cmake\\release_build\u003e cmake -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=%PROTOBUF_INSTALL% ..\r\n```\r\n\r\nFinally build and install Protocol Buffers:\r\n\r\n```shell\r\nC:\\Git\\protobuf\\cmake\\release_build\u003e cmake --build . --config Release\r\nC:\\Git\\protobuf\\cmake\\release_build\u003e cmake --build . --config Release --target install\r\n```\r\n\r\n#### Add the protobuf schema files to the build procedure\r\n\r\nProtobufkdb uses a factory to create a message class object of the correct type from the message type string passed from kdb.  The lookup requires that the message type string passed from kdb is the same as the message name in its .proto definition.\r\n\r\nIn order to populate the factory, the .proto files for all messages to be serialised/parsed must be incorporated into the build as follows:\r\n\r\n1.  Place the new `\u003cschema\u003e.proto` file into the `src/` subdirectory\r\n2.  Edit `src/CMakeLists.txt` file, adding the new .proto file to the line below the following comment:\r\n\r\n    ```cmake\r\n    # ### GENERATE PROTO FILES ###\r\n    ```\r\n\r\n    For example, to add `examples.proto` (which is already present in the `src/` subdirectory), in addition to the existing `tests.proto`, change:\r\n\r\n    ```cmake\r\n    set(MY_PROTO_FILES tests.proto)\r\n    ```\r\n\r\n    to:\r\n\r\n    ```cmake\r\n    set(MY_PROTO_FILES tests.proto examples.proto)\r\n    ```\r\n\r\n**Note:** `MY_PROTO_FILES` is a CMake-space separated list; do not wrap the list of `.proto` files in a string.\r\n\r\n#### Building protobufkdb\r\n\r\nA CMake script is provided to build protobufkdb. This uses the CMake functionality to locate the protobuf installation on your system.  By setting the CMake environment variable `CMAKE_PREFIX_PATH` to the Protocol Buffers installation directory created above when building protobuf from source, CMake will use this installation in preference to any existing system installs.  This avoids issues with existing incompatible or mismatched protobuf installs.\r\n\r\nFrom the root of this repository create and move into a directory in which to perform the build:\r\n\r\n```bash\r\nmkdir build \u0026\u0026 cd build\r\n```\r\n\r\nGenerate the build scripts,  specifying the protobuf buffers installation created above when building protobuf from source (referenced by the environment variable `$PROTOBUF_INSTALL` which should have been set during that procedure):\r\n\r\n```bash\r\n## Linux/MacOS\r\ncmake -DCMAKE_PREFIX_PATH=$PROTOBUF_INSTALL ..\r\n\r\n## Windows\r\ncmake -DCMAKE_PREFIX_PATH=%PROTOBUF_INSTALL% ..\r\n```\r\n\r\nStart the build:\r\n\r\n```bash\r\ncmake --build . --config Release\r\n```\r\n\r\nCreate the install package and deploy:\r\n\r\n```bash\r\ncmake --build . --config Release --target install\r\n```\r\n\r\n**Note:**  By default `src/CMakeLists.txt` is configured to link statically against libprotobuf to avoid potential C++ ABI [compatibility issues](https://github.com/protocolbuffers/protobuf/tree/master/src#binary-compatibility-warning) with different versions of libprotobuf.  This is a particular issue on [Windows](https://github.com/protocolbuffers/protobuf/blob/master/cmake/README.md#dlls-vs-static-linking).\r\n\r\n#### Build issues\r\n\r\nBecause the protobufkdb interface uses both the protoc compiler and the Protocol Buffers’ runtime, the versions of protoc, libprotobuf and its header files must be consistent and installed from the same build.  Otherwise build errors can occur when compiling any of the proto-generated `.pb.h` or `.pb.cc` files.  To help identify these problems the protobufkdb CMake scripts log the locations of the Protocol Buffers installation it has found.  For example:\r\n\r\n```txt\r\n[build]$ cmake ..\r\n -- The CXX compiler identification is GNU 4.8.5\r\n -- Check for working CXX compiler: /usr/bin/c++\r\n -- Check for working CXX compiler: /usr/bin/c++ - works\r\n -- Detecting CXX compiler ABI info\r\n -- Detecting CXX compiler ABI info - done\r\n -- Detecting CXX compile features\r\n -- Detecting CXX compile features - done\r\n -- Generator : Unix Makefiles\r\n -- Build Tool : /usr/bin/gmake\r\n -- Proto files: tests.proto;examples.proto\r\n -- [ /usr/share/cmake3/Modules/FindProtobuf.cmake:321 ] Protobuf_USE_STATIC_LIBS = ON\r\n -- [ /usr/share/cmake3/Modules/FindProtobuf.cmake:455 ] requested version of Google Protobuf is\r\n -- [ /usr/share/cmake3/Modules/FindProtobuf.cmake:463 ] location of common.h: /usr/local/include/google/protobuf/stubs/common.h\r\n -- [ /usr/share/cmake3/Modules/FindProtobuf.cmake:481 ] /usr/local/include/google/protobuf/stubs/common.h reveals protobuf 3.7.1\r\n -- [ /usr/share/cmake3/Modules/FindProtobuf.cmake:495 ] /home/protobuf/install/bin/protoc reveals version 3.11.4\r\n -- Found Protobuf: /usr/local/lib/libprotobuf.a;-lpthread (found version \"3.7.1\")\r\n -- Configuring done\r\n -- Generating done\r\n -- Build files have been written to: /home/protobufkdb/build\r\n```\r\n\r\nindicates it found protoc version 3.11.4 at `/home/protobuf/install/bin/protoc` but version 3.7.1 of `libprotobuf.a` (and the headers) installed on the system under `/usr/local/`.  This can occur if there was a conflicting packaged version of protobuf already on the system and will likely cause the protobufkdb build to fail.  \r\n\r\nThe solution, as described above, is to build the Protocol Buffers runtime from source, install it to non-system directory then specify that directory when building protobufkdb.\r\n\r\n#### Docker – Linux\r\n\r\nA sample Docker file is provided in the `docker_linux` directory to create a Ubuntu 18.04 LTS environment (including downloading and building the Protocol Buffers runtime from source) before building and installing the kdb+ `protobufkdb` interface.\r\n\r\nFor Docker Windows, the `PROTOBUFKDB_SOURCE` and `QHOME_LINUX` directories are specified at the top of `protobufkdb_build.bat`, which sets up the environment specified in `Dockerfile.build` and invokes `protobufkdb_build.sh` to build the interface.\r\n\r\n\r\n\r\n\r\n## Status\r\n\r\nThe protobufkdb interface is provided here as a beta release under an Apache 2.0 license.\r\n\r\nIf you find issues with the interface or have feature requests, please [raise an issue](../../issues).\r\n\r\nTo contribute to this project, please follow the [contribution guide](CONTRIBUTING.md).\r\n\r\nProtocol Buffers is used under the terms of [Google’s license](https://github.com/protocolbuffers/protobuf/blob/master/LICENSE):\r\n\r\n```txt\r\nCopyright 2008 Google Inc.  All rights reserved.\r\n\r\nRedistribution and use in source and binary forms, with or without\r\nmodification, are permitted provided that the following conditions are\r\nmet:\r\n\r\n    * Redistributions of source code must retain the above copyright\r\nnotice, this list of conditions and the following disclaimer.\r\n    * Redistributions in binary form must reproduce the above\r\ncopyright notice, this list of conditions and the following disclaimer\r\nin the documentation and/or other materials provided with the\r\ndistribution.\r\n    * Neither the name of Google Inc. nor the names of its\r\ncontributors may be used to endorse or promote products derived from\r\nthis software without specific prior written permission.\r\n\r\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\r\n\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\r\nLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\r\nA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\r\nOWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\r\nSPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\r\nLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\r\nDATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\r\nTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\r\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n\r\nCode generated by the Protocol Buffer compiler is owned by the owner\r\nof the input file used when generating it.  This code is not\r\nstandalone and requires a support library to be linked with it.  This\r\nsupport library is itself covered by the above license.\r\n```\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKxSystems%2Fprotobufkdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKxSystems%2Fprotobufkdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKxSystems%2Fprotobufkdb/lists"}