https://github.com/manticoresoftware/columnar
Manticore Columnar Library
https://github.com/manticoresoftware/columnar
Last synced: 3 months ago
JSON representation
Manticore Columnar Library
- Host: GitHub
- URL: https://github.com/manticoresoftware/columnar
- Owner: manticoresoftware
- License: apache-2.0
- Created: 2021-03-02T04:45:26.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2026-01-31T09:35:09.000Z (3 months ago)
- Last Synced: 2026-01-31T11:12:29.194Z (3 months ago)
- Language: C++
- Size: 6.59 MB
- Stars: 99
- Watchers: 13
- Forks: 18
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
- Changelog: Changelog.md
- License: LICENSE
- Notice: NOTICE
Awesome Lists containing this project
README
Manticore Columnar Library
Manticore Columnar Library is a column-oriented storage and secondary indexing library, aiming to provide **decent performance with low memory footprint at big data volume**.
When used in combination with [Manticore Search](https://github.com/manticoresoftware/manticoresearch) can be beneficial for those looking for:
1. log analytics including rich free text search capabities (which is missing in e.g. [Clickhouse](https://github.com/ClickHouse/ClickHouse) - great tool for metrics analytics)
2. faster / low resource consumption log/metrics analytics. Since the library and Manticore Search are both written in C++ with low level optimizations in mind, in many cases the performance / RAM consumption is better than in Lucene / SOLR / Elasticsearch
3. running log / metric analytics in docker / kubernetes. Manticore Search + the library can work with as little as 30 megabytes of RAM which Elasticsearch / Clickhouse can't. It also starts in less than a second or a few seconds in the worst case. Since the overhead is so little you can afford having more nodes of Manticore Search + the library than Elasticsearch. More nodes and quicker start means higher high availability and agility.
4. powerful SQL for logs/metrics analytics and everything else [Manticore Search](https://github.com/manticoresoftware/manticoresearch) can give you
## Getting started
### Requirements
1. CPU: with SSE >= 4.2
2. Architecture: arm64 or x86_64
3. OS: Debian-based OS (Debian, Ubuntu, Mint), RHEL-based (RHEL, Centos, Alma, Oracle Linux, Amazon Linux) OS, Windows, MacOS
### Installation from yum/apt repositories
#### Ubuntu, Debian:
```bash
wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt install manticore-columnar-lib
```
#### Centos:
```bash
sudo yum install https://repo.manticoresearch.com/manticore-repo.noarch.rpm
sudo yum install manticore-columnar-lib
```
`searchd -v` should include `columnar x.y.z`, e.g.:
```bash
root@srv# searchd -v
Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
```
### Basic usage:
1. Read https://manual.manticoresearch.com/Creating_a_table/Data_types#Row-wise-and-columnar-attribute-storages
2. Create plain or real-time table specifying that the columnar storage should be used
## Benchmarks
### Log analytics - 6x faster than Elasticsearch
https://db-benchmarks.com/test-logs10m/#elasticsearch-tuned-vs-manticore-search-columnar-storage
### Log analytics - 1.4x faster than Clickhouse
https://db-benchmarks.com/test-logs10m/#clickhouse-vs-manticore-search-columnar-storage
### Medium data - 110M Hackernews comments - 5x faster than Elasticsearch
https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-elasticsearch
### Medium data - 110M Hackernews comments - 11x faster than Clickhouse
https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-clickhouse
### Big data - 1.7B NYC taxi rides - 4x faster than Elasticsearch
https://db-benchmarks.com/test-taxi/#manticore-search-vs-elasticsearch
### Big data - 1.7B NYC taxi rides - 1.8x faster than Clickhouse
https://db-benchmarks.com/test-taxi/#manticore-search-vs-clickhouse
