{"id":18743085,"url":"https://github.com/kxsystems/avrokdb","last_synced_at":"2025-08-24T22:21:27.317Z","repository":{"id":196557694,"uuid":"691008527","full_name":"KxSystems/avrokdb","owner":"KxSystems","description":"kdb+ integration with Apache Avro","archived":false,"fork":false,"pushed_at":"2023-10-12T17:32:45.000Z","size":5723,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-18T13:45:58.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KxSystems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-09-13T10:18:16.000Z","updated_at":"2023-10-16T12:21:06.000Z","dependencies_parsed_at":"2025-05-22T06:31:28.869Z","dependency_job_id":null,"html_url":"https://github.com/KxSystems/avrokdb","commit_stats":null,"previous_names":["kxsystems/avrokdb"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/KxSystems/avrokdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Favrokdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Favrokdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Favrokdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Favrokdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KxSystems","download_url":"https://codeload.github.com/KxSystems/avrokdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KxSystems%2Favrokdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271963000,"owners_count":24850597,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-24T02:00:11.135Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T16:09:58.385Z","updated_at":"2025-08-24T22:21:27.295Z","avatar_url":"https://github.com/KxSystems.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# avrokdb\n\n![Avro](Apache_Avro_Logo.svg)\n\n[![GitHub release (latest by date)](https://img.shields.io/github/v/release/kxsystems/avrokdb?include_prereleases)](https://github.com/kxsystems/avrokdb/releases) [![Travis (.com) branch](https://travis-ci.com/KxSystems/avrokdb.svg?branch=main)](https://travis-ci.com/KxSystems/avrokdb)\n\n\n## Introduction\n\nThis interface allows kdb+ to users encode and decode Apache Avro serialized data.\n\nThis is part of the [*Fusion for kdb+*](http://code.kx.com/q/interfaces/fusion/) interface collection.\n\n\n\n## New to kdb+ ?\n\nKdb+ is the world's fastest time-series database, optimized for  ingesting, analyzing and storing massive amounts of structured data. To  get started with kdb+, please visit https://code.kx.com/q/ for downloads and developer information. For general information, visit https://kx.com/\n\n\n\n## New to Apache Avro?\n\nApache Avro is a data serialization system.\n\nAvro provides:\n\n- Rich data structures.\n- A compact, fast, binary data format.\n\nAvro relies on schemas which are defined with JSON. When Avro data is read, the schema used when writing has to be provided. This permits each datum to be written with no per-value overheads, making serialization both fast and small.\n\n\n\n## Installation\n\n### Requirements\n\n- kdb+ ≥ 3.5 64-bit (Linux/macOS/Windows)\n\n### Installing a release\n\nIt is recommended that a user install this interface using a release package. This is completed in a number of steps:\n\n1. If running on Windows, ensure you have downloaded/installed the Avro C++ libraries following the instructions [here](#windows).  This step is not necessary on Linux or macOS because their `avrokdb` release packages are statically linked with `libavrocpp`.\n2. [Download a release](https://github.com/KxSystems/avrokdb/releases) for your system architecture.\n3. Install script `avrokdb.q` to `$QHOME`, and binary file `lib/arrowkdb.(so|dll)` to `$QHOME/[mlw](64)`, by executing the following from the unzipped release package directory:\n\n```bash\n## Linux/macOS\nchmod +x install.sh \u0026\u0026 ./install.sh\n\n## Windows\ninstall.bat\n```\n\n\n\n## Building and installing from source\n\n### Requirements\n\n- kdb+ ≥ 3.5 64-bit (Linux/macOS/Windows)\n- Apache Avro C++ libraries\n- C++11 or later and \n- CMake ≥ 3.1.3\n\n### Third-party library installation\n\n#### Linux\n\nOn linux `avrocpp` should be built from source.\n\n1. Install the `avrocpp` dependencies (boost and compression libraries).  \n\n   With an `apt` package manager:\n\n   ```bash\n   sudo apt -y update\n   sudo apt -y install libboost-dev libboost-filesystem-dev libboost-iostreams-dev libboost-program-options-dev libsnappy-dev\n   ```\n\n   With a `yum` package manager:\n\n   ```bash\n   sudo yum -y update\n   sudo yum -y install boost-devel boost-filesystem boost-iostreams boost-program-options snappy-devel\n   ```\n\n2. Clone the avro repo and switch to the `release-1.11.2` tag (which is the one used to build the `avrokdb` linux package)\n\n   ```bash\n   git clone https://github.com/apache/avro.git\n   cd avro\n   git checkout refs/tags/release-1.11.2 --\n   ```\n\n3. Create cmake build and install directories:\n\n   ```bash\n   cd lang/c++\n   mkdir build\n   mkdir install\n   export AVRO_INSTALL=$(pwd)/install\n   cd build\n   ```\n\n4. Generate the cmake build scripts:\n\n   ```bash\n   cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11 -DCMAKE_INSTALL_PREFIX=$AVRO_INSTALL ..\n   ```\n\n5. Build and install to `$AVRO_INSTALL`:\n\n   ```bash\n     cmake --build . --config Release\n     cmake --build . --config Release --target install\n   ```\n\n6. Copy the shared object to `$QHOME/l64`:\n\n   ```bash\n   cp $AVRO_INSTALL/lib/libavrocpp.so $QHOME/l64\n   ```\n\n#### macOS\n\nOn macOS `avrocpp` (and its dependency on `boost`) can be installed using `brew`:\n\n```bash\nbrew install avro-cpp boost\n```\n\n#### Windows\n\nOn Windows `avrocpp` should be built using [vcpkg](https://vcpkg.io/en/):\n\n1. Clone the `vcpkg` repo and bootstrap it:\n\n   ```bash\n   git clone https://github.com/microsoft/vcpkg.git\n   cd vcpkg\n   git checkout refs/tags/2023.08.09 --\n   bootstrap-vcpkg.bat\n   ```\n\n2. Install `avrocpp`:\n\n   ```bash\n   vcpkg install avro-cpp:x64-windows\n   set AVRO_INSTALL=%cd%\\installed\\x64-windows\n   ```\n\n3. Copy the DLLs to `$QHOME\\w64`:\n\n   ```bash\n   cd installed\\x64-windows\\bin\n   copy avrocpp.dll %QHOME%\\w64\n   copy boost_iostreams-vc143-mt-x64-*.dll %QHOME%\\w64\n   copy liblzma.dll %QHOME%\\w64\n   copy zstd.dll %QHOME%\\w64\n   copy bz2.dll %QHOME%\\w64\n   copy zlib1.dll %QHOME%\\w64\n   ```\n\n\n### Building avrokdb\n\nIn order to successfully build and install this interface from source, the following environment variables must be set:\n\n1. `AVRO_INSTALL` = Location of the Avro C++ API release (only required if `avrocpp` is not installed globally on the system, e.g. on Linux or Windows where `avrocpp` was built from source)\n2. `BOOST_INSTALL` = Locaion of the Boost C++ library (only required if `boost` is not installed globally on the system)\n3. `QHOME` = Q installation directory (directory containing `q.k`)\n\nFrom a shell prompt (on Linux/macOS) or Visual Studio command prompt (on Windows), clone the `avrokdb` source from github:\n\n```bash\ngit clone https://github.com/KxSystems/avrokdb.git\ncd avrokdb\n```\n\nCreate the cmake build directory and generate the build files (this will use the system's default cmake generator):\n\n```bash\nmkdir build\ncd build\n\n## Linux (using the Arrow installation which was build from source as above)\ncmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11 -DAVRO_INSTALL=$AVRO_INSTALL\n\n## macOS\ncmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11\n\n## Windows (using the Arrow installation which was build from source as above)\ncmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11 -DAVRO_INSTALL=%AVRO_INSTALL%\n```\n\nStart the build:\n\n```bash\ncmake --build . --config Release\n```\n\nCreate the install package and deploy to `$QHOME`:\n\n```bash\ncmake --build . --config Release --target install\n```\n\n\n\n## Documentation\n\nDocumentation outlining the functionality available for this interface can be found in the [`docs`](docs/introduction.md) folder.\n\n\n\n## Status\n\nThe avrokdb interface is provided here under an Apache 2.0 license.\n\nIf you find issues with the interface or have feature requests, please consider [raising an issue](https://github.com/KxSystems/avrokdb/issues).\n\nIf you wish to contribute to this project, please follow the [contribution guide](CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkxsystems%2Favrokdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkxsystems%2Favrokdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkxsystems%2Favrokdb/lists"}