{"id":22117458,"url":"https://github.com/terma/fast-select","last_synced_at":"2025-07-25T11:32:30.072Z","repository":{"id":57723567,"uuid":"45509509","full_name":"terma/fast-select","owner":"terma","description":"Extremely fast and compact in-memory embedded column oriented database","archived":false,"fork":false,"pushed_at":"2017-08-29T23:43:41.000Z","size":839,"stargazers_count":19,"open_issues_count":11,"forks_count":12,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-17T13:03:43.637Z","etag":null,"topics":["cache-storage","compact","java","lock-free","nosql-database","storage"],"latest_commit_sha":null,"homepage":"https://terma.github.io/fast-select/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/terma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-04T02:19:41.000Z","updated_at":"2024-11-06T20:17:11.000Z","dependencies_parsed_at":"2022-08-26T12:33:16.208Z","dependency_job_id":null,"html_url":"https://github.com/terma/fast-select","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terma%2Ffast-select","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terma%2Ffast-select/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terma%2Ffast-select/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terma%2Ffast-select/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/terma","download_url":"https://codeload.github.com/terma/fast-select/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227568794,"owners_count":17787656,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache-storage","compact","java","lock-free","nosql-database","storage"],"created_at":"2024-12-01T13:34:20.137Z","updated_at":"2025-07-25T11:32:30.060Z","avatar_url":"https://github.com/terma.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fast-select\n\n[![Build Status](https://travis-ci.org/terma/fast-select.svg?branch=start)](https://travis-ci.org/terma/fast-select)\n[![Coverage Status](https://coveralls.io/repos/github/terma/fast-select/badge.svg?branch=master)](https://coveralls.io/github/terma/fast-select?branch=master) [![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.terma/fast-select/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.github.terma/fast-select/)\n\nCompact in-memory read-only storage with lock free ultra-fast quering by any attributes under Apache 2.0 license.\n\n* [Key Properties](#key-properties)\n* [Use Cases](#use-cases)\n* [Architecture](docs/ARHI.md), [Performance](docs/PERF.md), [Javadoc](http://terma.github.io/fast-select/)\n* [How To Use](#how-to-use)\n  * [Aggregate/Group By/Pivot](#aggregate)\n  * [Filter, Sort and first 25](#select-first-25-items-from-sorted-dataset)\n  * [Filter, Sort and get page](#filter-dataset-get-total-and-render-only-one-page)\n  * [JMX](#jmx)\n  * [Low Cardinality Strings](#low-cardinality-strings)\n\n## Key Properties\n\n* Compact \n  * No java object overhead (in avg ```x10``` less than object representation)\n  * Compact string representation (```UTF-8``` instead of Java ```UTF-16```)\n  * String data compression\n  * Small metadata footprint (no indexes overhead)\n* Fast\n  * All dimension available for search\n  * Using data statistic to avoid full scan\n  * Column oriented\n  * Thread safe and lock free\n* Support fast save/load to/from disk [details](USECASES.md)\n* Small jar file\n* Apache 2.0\n\n## Use Cases\n\n* Speed up analytical quering by caching main data in compact and query optimized way instead of using expensive solution [details](USECASES.md#speed-up-analytic)\n* Separate ETL and analytic load by keeping main data optimized for processing and add compact model optimizing for quering [details](USECASES.md#separate-processing-and-analytic)\n* Sub second quering of historical data by loading portion of data on demand in a seconds [details](USECASES.md#speed-up-history-analytic)\n\n## How to use\n\n### Create Data Class\n\n```java\npublic class Data {\n    public byte a;\n    public byte b;\n}\n```\n\n### Build storage\n\n```java\nFastSelect\u003cData\u003e database = new FastSelectBuilder\u003c\u003e(Data.class).create();\n\n// add your data\ndatabase.addAll(new ArrayList\u003cData\u003e(...)); \n```\n\n### Aggregate\n\nIn case if you just want ```select F1, F2, count(X) ... group by F1, F2```\n```java\nMultiGroupCountCallback callback = new MultiGroupCountCallback(fastSelect.getColumnsByNames().get(\"a\"));\ndatabase.select(\n  new Request[] {new IntRequest(\"a\", new int[]{12, 3})}, \n  callback);\ncallback.getCounters(); // your result here grouped by field 'a'\n```\n\nStarting from version ```3.2.0``` For more sophisticated and flexible cases you can use ```AggregateCallback``` which support user defined aggregation in fast way, so you don't need to worry about aggregation key perfromance etc.\n```java\nFastSelect fastSelect = ...;\nfinal ByteData data = fastSelect.getData(\"columnWithData\");\nAggregateCallback\u003cMutableInt\u003e callback = new AggregateCallback\u003c\u003e(\n    new Aggregator\u003cMutableInt\u003e() {\n        @Override\n        public MutableInt create(int position) {\n           // will be called when this unique key happens first time\n           return new MutableInt(data.data[position]);\n        }\n        \n        @Override\n        public void aggregate(MutableInt agg, int position) {\n            // will be called all other times\n            agg.add(data.data[position]);\n        }\n    },\n    fastSelect.getColumnsByNames().get(\"aggregationColumn1\"),\n    fastSelect.getColumnsByNames().get(\"aggregationColumn2\")\n    // you can specify any amount of columns with any type\n);\nfastSelect.select(new Request[0], callback);\nMap\u003cAggregateKey, MutableInt\u003e result = callback.getResult();\n```\n\n### Select first 25 items from sorted dataset\n```java\nListLimitCallback\u003cDemoData\u003e callback = new ListLimitCallback\u003c\u003e(25);\nfastSelect.selectAndSort(where, callback, \"a\");\ncallback.getResult();\n```\n\n### Filter dataset get total and render only one page\n```java\n// get ref to real data\nIntData id = (IntData) fastSelect.getColumnsByNames().get(\"id\").data;\n\nList\u003cInteger\u003e positions = fastSelect.selectPositions(new Request[] {...});\nCollections.sort(positions, new Comparator\u003cInteger\u003e() {\n    public int compare(Integer p1, Integer p2) { \n        return id.data[p1] - id.data[p2];\n    }\n});\n\n// page render\nList\u003cMap\u003cString, String\u003e\u003e page = new ArrayList\u003c\u003e();\nfor (int i = 10; i \u003c 20; i++) {\n    int p = positions.get(i);\n    Map\u003cString, String\u003e row = new HashMap\u003c\u003e();\n    row.put(\"id\", id.data[p]);\n    page.add(row);\n}\n\nint total = positions.size();\n```\n\n### Combine filters by AND\n\nJust add more requests\n\n```java\nfastSelect.select(new Request[] {\n    new IntRequest(\"id\", 12),\n    new StringLikeRequest(\"name\", \"bim\"); // name like '%bim%'\n    ...\n});\n```\n\n### Combine filters by OR\n\nWrap requests which should be by OR to ```OrRequest```\n\n```java\nnew OrRequest(\n    new IntRequest(\"id\", 12),\n    new StringLikeRequest(\"name\", \"bim\"); // name like '%bim%'\n)\n```\n\n### JMX\n\nTo publish information by JMX about instance of FastSelect you can use embedded class ```FastSelectMXBeanImpl``` from package ```com.github.terma.fastselect.jmx``` It provide read-only info like:\n* size (count of records)\n* allocated size\n* used mem\n* columns (type, name, mem)\n\n#### To register FastSelect instance by JMX\n```java\nFastSelect\u003cObject\u003e fastSelect = ...;\nFastSelectMXBean fastSelectMXBean = new FastSelectMXBeanImpl(fastSelect);\nMBeanServer mbs = ManagementFactory.getPlatformMBeanServer();\nmbs.registerMBean(fastSelectMXBean, new ObjectName(\"fastselect:type=mbeanname\"));\n```\n\n#### Unregister \nUse standard way for MBeans:\n```java\nString mbeanName = ...;\nMBeanServer mbs = ManagementFactory.getPlatformMBeanServer();\nmbs.unregisterMBean(new ObjectName(\"fastselect:type=mbeanname\"));\n```\n\n\nMore use cases you can find in javadoc ```callbacks``` package\n\n### Low Cardinality Strings\n\n```fast-select``` provides very compact storage for ```small``` Java types and String, which is provide you\n mem benefits because of no Java object overhead and ```UTF-8``` compression. However you can\n get even better result for low cardinality columns. Take a look on:\n \n * ```com.github.terma.fastselect.data.StringCompressedByteData```\n * ```com.github.terma.fastselect.data.StringCompressedShortData```\n * ```com.github.terma.fastselect.data.StringCompressedIntData```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterma%2Ffast-select","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fterma%2Ffast-select","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterma%2Ffast-select/lists"}