{"id":15056718,"url":"https://github.com/stratio/cassandra-lucene-index","last_synced_at":"2025-10-04T16:31:51.707Z","repository":{"id":31641734,"uuid":"35206972","full_name":"Stratio/cassandra-lucene-index","owner":"Stratio","description":"Lucene based secondary indexes for Cassandra","archived":true,"fork":false,"pushed_at":"2020-05-27T11:21:14.000Z","size":11040,"stargazers_count":600,"open_issues_count":64,"forks_count":170,"subscribers_count":140,"default_branch":"branch-3.0.14","last_synced_at":"2024-09-30T04:03:19.321Z","etag":null,"topics":["bitemporal","cassandra","fulltext","fuzzy-search","geospatial","index","lucene","relevance","search","sorting","stratio","stratio-cassandra","wildcard"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Stratio.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-07T08:04:05.000Z","updated_at":"2024-08-24T02:03:00.000Z","dependencies_parsed_at":"2022-09-11T10:11:50.130Z","dependency_job_id":null,"html_url":"https://github.com/Stratio/cassandra-lucene-index","commit_stats":null,"previous_names":[],"tags_count":129,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stratio%2Fcassandra-lucene-index","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stratio%2Fcassandra-lucene-index/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stratio%2Fcassandra-lucene-index/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stratio%2Fcassandra-lucene-index/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Stratio","download_url":"https://codeload.github.com/Stratio/cassandra-lucene-index/tar.gz/refs/heads/branch-3.0.14","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235285424,"owners_count":18965324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitemporal","cassandra","fulltext","fuzzy-search","geospatial","index","lucene","relevance","search","sorting","stratio","stratio-cassandra","wildcard"],"created_at":"2024-09-24T21:55:38.808Z","updated_at":"2025-10-04T16:31:46.225Z","avatar_url":"https://github.com/Stratio.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"================================\nStratio’s Cassandra Lucene Index\n================================\n\nStratio’s Cassandra Lucene Index, derived from `Stratio Cassandra \u003chttps://github.com/Stratio/stratio-cassandra\u003e`__, is\na plugin for `Apache Cassandra \u003chttp://cassandra.apache.org/\u003e`__ that extends its index functionality to provide near\nreal time search such as ElasticSearch or Solr, including `full text search \u003chttp://en.wikipedia.org/wiki/Full_text_search\u003e`__\ncapabilities and free multivariable, geospatial and bitemporal search. It is achieved through an `Apache Lucene \u003chttp://lucene.apache.org/\u003e`__\nbased implementation of Cassandra secondary indexes, where each node of the cluster indexes its own data. Stratio’s\nCassandra indexes are one of the core modules on which `Stratio’s BigData platform \u003chttp://www.stratio.com/\u003e`__ is based.\n\n.. image:: /doc/resources/architecture.png\n   :width: 100%\n   :alt: architecture\n   :align: center\n\nIndex `relevance searches \u003chttp://en.wikipedia.org/wiki/Relevance_(information_retrieval)\u003e`__ allow you to retrieve the\n*n* more relevant results satisfying a search. The coordinator node sends the search to each node in the cluster, each node\nreturns its *n* best results and then the coordinator combines these partial results and gives you the *n* best of them,\navoiding full scan. You can also base the sorting in a combination of fields.\n\nAny cell in the tables can be indexed, including those in the primary key as well as collections. Wide rows are also\nsupported. You can scan token/key ranges, apply additional CQL3 clauses and page on the filtered results.\n\nIndex filtered searches are a powerful help when analyzing the data stored in Cassandra with `MapReduce \u003chttp://es.wikipedia.org/wiki/MapReduce\u003e`__\nframeworks as `Apache Hadoop \u003chttp://hadoop.apache.org/\u003e`__ or, even better, `Apache Spark \u003chttp://spark.apache.org/\u003e`__.\nAdding Lucene filters in the jobs input can dramatically reduce the amount of data to be processed, avoiding full scan.\n\n.. image:: /doc/resources/spark_architecture.png\n   :width: 100%\n   :alt: spark_architecture\n   :align: center\n\nThe following benchmark result can give you an idea about the expected performance when combining Lucene indexes with\nSpark. We do successive queries requesting from the 1% to 100% of the stored data. We can see a high performance for the\nindex for the queries requesting strongly filtered data. However, the performance decays in less restrictive queries.\nAs the number of records returned by the query increases, we reach a point where the index becomes slower than the full\nscan. So, the decision to use indexes in your Spark jobs depends on the query selectivity. The trade-off between both\napproaches depends on the particular use case. Generally, combining Lucene indexes with Spark is recommended for jobs\nretrieving no more than the 25% of the stored data.\n\n.. image:: /doc/resources/spark_performance.png\n   :width: 100%\n   :alt: spark_performance\n   :align: center\n\nThis project is not intended to replace Apache Cassandra denormalized tables, inverted indexes, and/or secondary\nindexes. It is just a tool to perform some kind of queries which are really hard to be addressed using Apache Cassandra\nout of the box features, filling the gap between real-time and analytics.\n\n.. image:: /doc/resources/oltp_olap.png\n   :width: 100%\n   :alt: oltp_olap\n   :align: center\n\nMore detailed information is available at `Stratio’s Cassandra Lucene Index documentation \u003cdoc/documentation.rst\u003e`__.\n\nFeatures\n--------\n\nLucene search technology integration into Cassandra provides:\n\nStratio’s Cassandra Lucene Index and its integration with Lucene search technology provides:\n\n-  Full text search (language-aware analysis, wildcard, fuzzy, regexp)\n-  Boolean search (and, or, not)\n-  Sorting by relevance, column value, and distance\n-  Geospatial indexing (points, lines, polygons and their multiparts)\n-  Geospatial transformations (bounding box, buffer, centroid, convex hull, union, difference, intersection)\n-  Geospatial operations (intersects, contains, is within)\n-  Bitemporal search (valid and transaction time durations)\n-  CQL complex types (list, set, map, tuple and UDT)\n-  CQL user defined functions (UDF)\n-  CQL paging, even with sorted searches\n-  Columns with TTL\n-  Third-party CQL-based drivers compatibility\n-  Spark and Hadoop compatibility\n\nNot yet supported:\n\n-  Thrift API\n-  Legacy compact storage option\n-  Indexing ``counter`` columns\n-  Indexing static columns\n-  Other partitioners than Murmur3\n\nRequirements\n------------\n\n-  Cassandra (identified by the three first numbers of the plugin version)\n-  Java \u003e= 1.8 (OpenJDK and Sun have been tested)\n-  Maven \u003e= 3.0\n\nBuild and install\n-----------------\n\nStratio’s Cassandra Lucene Index is distributed as a plugin for Apache Cassandra. Thus, you just need to build a JAR\ncontaining the plugin and add it to the Cassandra’s classpath:\n\n-  Clone the project: ``git clone http://github.com/Stratio/cassandra-lucene-index``\n-  Change to the downloaded directory: ``cd cassandra-lucene-index``\n-  Checkout a plugin version suitable for your Apache Cassandra version: ``git checkout A.B.C.X``\n-  Build the plugin with Maven: ``mvn clean package``\n-  Copy the generated JAR to the lib folder of your compatible Cassandra installation:\n   ``cp plugin/target/cassandra-lucene-index-plugin-*.jar \u003cCASSANDRA_HOME\u003e/lib/``\n-  Start/restart Cassandra as usual.\n\nSpecific Cassandra Lucene index versions are targeted to specific Apache Cassandra versions. So, cassandra-lucene-index\nA.B.C.X is aimed to be used with Apache Cassandra A.B.C, e.g.\n`cassandra-lucene-index:3.0.7.1 \u003chttp://www.github.com/Stratio/cassandra-lucene-index/tree/3.0.7.1\u003e`__ for\n`cassandra:3.0.7 \u003chttp://www.github.com/apache/cassandra/tree/cassandra-3.0.7\u003e`__. Please note that production-ready\nreleases are version tags (e.g. 3.0.6.3), don't use branch-X nor master branches in production.\n\nAlternatively, patching can also be done with this Maven profile, specifying the path of your Cassandra installation,\nthis task also deletes previous plugin's JAR versions in CASSANDRA_HOME/lib/ directory:\n\n.. code-block:: bash\n\n    mvn clean package -Ppatch -Dcassandra_home=\u003cCASSANDRA_HOME\u003e\n\nIf you don’t have an installed version of Cassandra, there is also an alternative profile to let Maven download and\npatch the proper version of Apache Cassandra:\n\n.. code-block:: bash\n\n    mvn clean package -Pdownload_and_patch -Dcassandra_home=\u003cCASSANDRA_HOME\u003e\n\nNow you can run Cassandra and do some tests using the Cassandra Query Language:\n\n.. code-block:: bash\n\n    \u003cCASSANDRA_HOME\u003e/bin/cassandra -f\n    \u003cCASSANDRA_HOME\u003e/bin/cqlsh\n\nThe Lucene’s index files will be stored in the same directories where the Cassandra’s will be. The default data\ndirectory is ``/var/lib/cassandra/data``, and each index is placed next to the SSTables of its indexed column family.\n\nRemember that if you use geo shape search you need to `include the JTS jar \u003cdoc/documentation.rst#geo-shape-mapper\u003e`__.\n\nFor more details about Apache Cassandra please see its `documentation \u003chttp://cassandra.apache.org/\u003e`__.\n\nExamples\n--------\n\nWe will create the following table to store tweets:\n\n.. code-block:: sql\n\n    CREATE KEYSPACE demo\n    WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1};\n    USE demo;\n    CREATE TABLE tweets (\n       id INT PRIMARY KEY,\n       user TEXT,\n       body TEXT,\n       time TIMESTAMP,\n       latitude FLOAT,\n       longitude FLOAT\n    );\n\nNow you can create a custom Lucene index on it with the following statement:\n\n.. code-block:: sql\n\n    CREATE CUSTOM INDEX tweets_index ON tweets ()\n    USING 'com.stratio.cassandra.lucene.Index'\n    WITH OPTIONS = {\n       'refresh_seconds': '1',\n       'schema': '{\n          fields: {\n             id: {type: \"integer\"},\n             user: {type: \"string\"},\n             body: {type: \"text\", analyzer: \"english\"},\n             time: {type: \"date\", pattern: \"yyyy/MM/dd\"},\n             place: {type: \"geo_point\", latitude: \"latitude\", longitude: \"longitude\"}\n          }\n       }'\n    };\n\nThis will index all the columns in the table with the specified types, and it will be refreshed once per second.\nAlternatively, you can explicitly refresh all the index shards with an empty search with consistency ``ALL``:\n\n.. code-block:: sql\n\n    CONSISTENCY ALL\n    SELECT * FROM tweets WHERE expr(tweets_index, '{refresh:true}');\n    CONSISTENCY QUORUM\n\nNow, to search for tweets within a certain date range:\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"}\n    }');\n\nThe same search can be performed forcing an explicit refresh of the involved index shards:\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"},\n       refresh: true\n    }') limit 100;\n\nNow, to search the top 100 more relevant tweets where *body* field contains the phrase “big data gives organizations”\nwithin the aforementioned date range:\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"},\n       query: {type: \"phrase\", field: \"body\", value: \"big data gives organizations\", slop: 1}\n    }') LIMIT 100;\n\nTo refine the search to get only the tweets written by users whose names start with \"a\":\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: [\n          {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"},\n          {type: \"prefix\", field: \"user\", value: \"a\"}\n       ],\n       query: {type: \"phrase\", field: \"body\", value: \"big data gives organizations\", slop: 1}\n    }') LIMIT 100;\n\nTo get the 100 more recent filtered results you can use the *sort* option:\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: [\n          {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"},\n          {type: \"prefix\", field: \"user\", value: \"a\"}\n       ],\n       query: {type: \"phrase\", field: \"body\", value: \"big data gives organizations\", slop: 1},\n       sort: {field: \"time\", reverse: true}\n    }') limit 100;\n\nThe previous search can be restricted to tweets created close to a geographical position:\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: [\n          {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"},\n          {type: \"prefix\", field: \"user\", value: \"a\"},\n          {type: \"geo_distance\", field: \"place\", latitude: 40.3930, longitude: -3.7328, max_distance: \"1km\"}\n       ],\n       query: {type: \"phrase\", field: \"body\", value: \"big data gives organizations\", slop: 1},\n       sort: {field: \"time\", reverse: true}\n    }') limit 100;\n\nIt is also possible to sort the results by distance to a geographical position:\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: [\n          {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"},\n          {type: \"prefix\", field: \"user\", value: \"a\"},\n          {type: \"geo_distance\", field: \"place\", latitude: 40.3930, longitude: -3.7328, max_distance: \"1km\"}\n       ],\n       query: {type: \"phrase\", field: \"body\", value: \"big data gives organizations\", slop: 1},\n       sort: [\n          {field: \"time\", reverse: true},\n          {field: \"place\", type: \"geo_distance\", latitude: 40.3930, longitude: -3.7328}\n       ]\n    }') limit 100;\n\nLast but not least, you can route any search to a certain token range or partition, in such a way that only a\nsubset of the cluster nodes will be hit, saving precious resources:\n\n.. code-block:: sql\n\n    SELECT * FROM tweets WHERE expr(tweets_index, '{\n       filter: [\n          {type: \"range\", field: \"time\", lower: \"2014/04/25\", upper: \"2014/05/01\"},\n          {type: \"prefix\", field: \"user\", value: \"a\"},\n          {type: \"geo_distance\", field: \"place\", latitude: 40.3930, longitude: -3.7328, max_distance: \"1km\"}\n       ],\n       query: {type: \"phrase\", field: \"body\", value: \"big data gives organizations\", slop: 1},\n       sort: [\n          {field: \"time\", reverse: true},\n          {field: \"place\", type: \"geo_distance\", latitude: 40.3930, longitude: -3.7328}\n       ]\n    }') AND TOKEN(id) \u003e= TOKEN(0) AND TOKEN(id) \u003c TOKEN(10000000) limit 100;\n\nThis last is the basis for `Hadoop, Spark and other MapReduce frameworks support \u003cdoc/documentation.rst#spark-and-hadoop\u003e`__.\n\nPlease, refer to the comprehensive `Stratio’s Cassandra Lucene Index documentation \u003cdoc/documentation.rst\u003e`__.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratio%2Fcassandra-lucene-index","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstratio%2Fcassandra-lucene-index","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratio%2Fcassandra-lucene-index/lists"}