{"id":23902800,"url":"https://github.com/mtumilowicz/java12-elasticsearch-inverted-index-workshop","last_synced_at":"2026-05-15T17:40:23.310Z","repository":{"id":110877623,"uuid":"253254634","full_name":"mtumilowicz/java12-elasticsearch-inverted-index-workshop","owner":"mtumilowicz","description":"Simple inverted index implementation with term analysis and scoring.","archived":false,"fork":false,"pushed_at":"2025-05-03T18:52:35.000Z","size":340,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-03T19:39:18.756Z","etag":null,"topics":["elasticsearch","inverted-index","inverted-index-implementation","inverted-index-model","term-analysis","token","tokenizer","workshop","workshop-materials"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mtumilowicz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-04-05T14:30:24.000Z","updated_at":"2025-05-03T18:52:38.000Z","dependencies_parsed_at":"2025-05-03T19:32:24.894Z","dependency_job_id":"1521ee5c-045d-4b87-bd6c-ccfff1c32335","html_url":"https://github.com/mtumilowicz/java12-elasticsearch-inverted-index-workshop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mtumilowicz/java12-elasticsearch-inverted-index-workshop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mtumilowicz","download_url":"https://codeload.github.com/mtumilowicz/java12-elasticsearch-inverted-index-workshop/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33073438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","inverted-index","inverted-index-implementation","inverted-index-model","term-analysis","token","tokenizer","workshop","workshop-materials"],"created_at":"2025-01-04T22:50:35.949Z","updated_at":"2026-05-15T17:40:23.303Z","avatar_url":"https://github.com/mtumilowicz.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://app.travis-ci.com/mtumilowicz/java12-elasticsearch-inverted-index-workshop.svg?branch=master)](https://travis-ci.com/mtumilowicz/java12-elasticsearch-inverted-index-workshop)\n[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n\n# java12-elasticsearch-inverted-index-workshop\n\n* references\n    * http://siddhumehta.blogspot.com/2014/06/elasticsearch-tutorial-inverted-index.html\n    * https://www.manning.com/books/elasticsearch-in-action\n    * https://medium.com/elasticsearch/introduction-to-analysis-and-analyzers-in-elasticsearch-4cf24d49ddab\n    * https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-mapping-charfilter.html\n    * https://www.elastic.co/guide/en/elasticsearch/reference/7.6/index.html\n    * https://www.elastic.co/guide/en/elasticsearch/reference/current/glossary.html#glossary-primary-shard\n    * https://www.nurkiewicz.com/2019/03/mapmerge-one-method-to-rule-them-all.html\n    * https://www.quora.com/What-is-inverted-index-It-is-a-well-known-fact-that-you-need-to-build-indexes-to-implement-efficient-searches-What-is-the-difference-between-index-and-inverted-index-and-how-does-one-build-inverted-index\n    * https://www.book-editing.com/why-book-indexing/\n    * [2018 - Philipp Krenn - Full-Text Search Internals](https://www.youtube.com/watch?v=TiLYEqfdVhs)\n    * https://codingexplained.com/coding/elasticsearch/understanding-sharding-in-elasticsearch\n    * https://codingexplained.com/coding/elasticsearch/introduction-elasticsearch-architecture\n    * https://codingexplained.com/coding/elasticsearch/understanding-replication-in-elasticsearch\n    * https://chatgpt.com/\n    * https://www.elastic.co/docs/manage-data/data-store/mapping/removal-of-mapping-types\n    * https://www.elastic.co/docs/manage-data/data-store/mapping\n    * https://www.elastic.co/blog/what-is-an-elasticsearch-index\n\n## preface\n* goals of this workshop\n    * understand foundations of elasticsearch\n        * index and inverted index\n        * term analysis\n    * understand how queries are analyzed (character filters, tokenizers, token filters)\n    * introduction to internals: Lucene's segments, scoring, shards and nodes\n\n* workshop are in `workshop` package, answers: `answers`\n\n## elasticsearch\n### document\n```\n{\n    \"name\": \"Michal\",\n    \"surname\": \"Tumilowicz\",\n    \"address\": {\n        \"city\": \"Warsaw\",\n        \"postalCode\": \"00-349\"\n    }\n    \"tasks\": [\"Task1\", \"Task2\"]\n}\n```\n* Elasticsearch is a distributed document store\n* document is the smallest unit of data you index or search for\n* properties\n    * self-contained: both fields and their values\n    * can be hierarchical: documents within documents\n    * flexible structure: don’t depend on a predefined schema\n* JSON representation\n    * can contain arrays of values\n \n### types\n* elasticsearch 8.0.0 no longer supports mapping types\n    * only one type per index is allowed, and it's always `_doc`\n    * mapping is now defined at the index level, not per type\n\n### mapping\n* is the schema definition for the documents in an index\n    * document = collection of fields + data types\n    * includes metadata fields\n        * example\n            * `_index` - index to which the document belongs\n            * `_id` - document’s ID\n            * `_source` - original JSON representing the body of the document\n            * others\n* example\n    ```\n    GET /your-index-name/_mapping\n  \n    {\n      \"people\": {\n        \"mappings\": {\n        \"_source\": { // by default ommitted\n          \"enabled\": true\n        },\n        \"_meta\": { // custom metadata (ex.: for documentation or tooling)\n          \"version\": \"1.0\",\n          \"description\": \"People index for sanction screening\"\n        },\n          \"properties\": {\n            \"name\": { \"type\": \"text\" },\n            \"birthdate\": { \"type\": \"date\" },\n            \"country\": { \"type\": \"keyword\" },\n            \"bio_vector\": {\n              \"type\": \"dense_vector\",\n              \"dims\": 384\n            }\n          }\n        }\n      }\n    }\n    ```\n* two types\n    * dynamic mapping\n        * automatically detects the data types of fields\n            * might yield suboptimal results for specific use cases\n            * default mappings\n                * defined using dynamic templates\n                * example: map `app_*.code` as keywords\n                    ```\n                    PUT /logs\n                    {\n                      \"mappings\": {\n                        \"dynamic_templates\": [\n                          {\n                            \"map_app_codes_as_keyword\": {\n                              \"path_match\": \"app_*.code\",\n                              \"mapping\": {\n                                \"type\": \"keyword\"\n                              }\n                            }\n                          }\n                        ]\n                      }\n                    }\n                    ```\n                    will produce types\n                    ```\n                    \"app_error.code\": { \"type\": \"keyword\" }\n                    \"app_warning.code\": { \"type\": \"keyword\" }\n                    ```\n        * add new fields automatically\n            * use case: some fields cannot be known in advance\n        * some data types cannot be automatically detected\n            * example: `geo_point`\n                * can be represented in multiple ways\n                    * string: `\"41.12,-71.34\"`\n                        * looks like text\n                        * what is first - latitude or longitude?\n                    * array: `[ -71.34, 41.12 ]`\n                        * looks like numeric array\n                    * object: `{ \"lat\": 41.12, \"lon\": -71.34 }`\n                        * looks like JSON\n                * so Elasticsearch requires to explicitly declare `geo_point` fields in mapping\n    * explicit mapping\n        * used to have greater control over fields\n        * recommended for production use cases\n* can’t change mappings for fields that are already mapped\n    * requires reindexing\n    * sometimes multi-fields are solution (index same field in different ways)\n        * drawback: old documents will not have them\n        * example\n            ```\n            \"city\": {\n              \"type\": \"text\",\n              \"fields\": {\n                \"raw\": {\n                  \"type\":  \"keyword\"\n                }\n              }\n            }\n            ```\n* mapping explosion\n    * too many fields in an index =\u003e risk of out of memory errors\n    * can be caused by lack of control over dynamic mapping\n        * example: every new document inserted introduces new fields\n    * solution: use the mapping limit settings to limit the number of field mappings\n\n### indices\n* logical namespace that holds a collection of documents\n    * can be considered as a table\n* logical abstraction over one or more Lucene indices (called shards)\n    * by default: all shards are queried\n        * solution: create logical groups of data in separate indices\n            * example\n                ```\n                customers-switzerland → 2 shards\n                customers-germany → 2 shards\n                customers-rest → 1 shard\n                ```\n* can be thought of as an optimized collection of documents\n    * each indexed field has optimized data structure\n        * example\n            * text fields -\u003e inverted indices\n            * numeric and geo fields -\u003e BKD trees\n* near-real time search\n    * searches not run on the latest indexed data\n        * indexing ≠ search visibility\n        * however, document can be retrieved by ID immediately\n            * but a search query won’t return it until a refresh happens\n    * point-in-time view of the index\n        * multiple searches hit the same files and reuse the same caches\n* processes\n    * indexing = storing\n        * document is put in two places\n            * in-memory buffer (Lucene memory buffer)\n            * transaction log (called translog) on disk\n                * crash recovery log\n                * translog is not searchable\n        * when\n            * document sent to Elasticsearch\n        * after indexing\n            * document is durable (even if node crashes)\n            * not yet searchable\n    * refresh\n        * makes newly indexed documents searchable\n        * writes in-memory buffer into a new Lucene segment\n            * usually reside in the OS page cache (memory)\n                * aren’t guaranteed to be persisted until `fsync` or `flush`\n                * in particular: files may never hit the actual disk\n                    * Lucene will ignore them if there's no updated `segments_N`\n                        * =\u003e update is done during commit\n        * opens a new searcher\n            * sees all committed segments\n            * sees any new segments created by a refresh\n            * does not see uncommitted in-memory data\n                * example: buffer\n            * every search request is handled by\n                * grabbing the current active searcher\n                    * each shard knows its current searcher\n                * executing the query against that consistent view\n                    * writes don’t interfere with ongoing searches\n        * when\n            * automatically every 1 second (default)\n            * manually: `POST /my-index/_refresh`\n        * after refresh\n            * documents are searchable\n    * commit\n        * it is not about search\n            * does not affect search =\u003e searchers see segments based on refresh, not commit\n        * uses `fsync`\n            * the only way to guarantee that the operating system has actually written data to disk\n        * pauses index writers briefly\n            * to ensure that commit reflects a consistent index state\n        * clears the translog (since changes are now safely in Lucene index)\n        * each commit creates a new `segments_N` file with an incremented generation number `(N)`\n            * represents current state of the index\n                * lists all the active segments\n                * older `segments_N` files are effectively obsolete after a new one is committed\n            * binary file\n                * textual example\n                    ```\n                    Segments:\n                    ----------\n                    Segment: _0\n                    - Uses compound file: true\n                    - Doc count: 1,000\n                    - Deleted docs: 0\n                    - Files:\n                        _0.cfs\n                        _0.cfe\n                        _0.si\n                    - Codec: Lucene90\n                    - Segment created with Lucene 9.9.0\n                    ```\n            * Lucene reads this file on startup\n                * tells which `.cfs` segment files to load and use\n                * reads `segments.gen` to find the latest `segments_N` file\n        * when\n            * the memory buffer is full\n            * time since last flush\n            * the transaction log hit a threshold\n            * in particular: refreshing and committing are independent\n    \n### inverted indexing\n* Lucene data structure where it keeps a list of where each word belongs\n    ![alt text](img/inverted-index.jpg)\n* example: index in the book with words and what pages they appear\n    ![alt text](img/book-index.jpg)\n\n### analysis\n![alt text](img/analysis_overview.png)\n![alt text](img/analysis_example.png)\n* steps\n    * character filtering — transform character sequences into character sequences\n        * example\n            * stripping HTML out of text\n            * '4' -\u003e 'for', '2' -\u003e 'too', 'U' -\u003e 'you'\n    * breaking text into one or more tokens\n        * token - smaller, meaningful string\n        * lucene itself doesn’t act on large strings but on tokens\n        * example: splitting text into tokens based on whitespaces\n    * token filtering — transforms each token using a token filter\n        * example\n            * lowercase token filter, 'Good' -\u003e 'good'\n            * removing the stopwords ('and', 'the', 'my')\n               * note that sometimes (very rarely) stopwords are important and can be helpful: \"to be, or not to be\"\n            * adding synonyms\n    * token indexing — stores those tokens into the index\n* the query text undergoes the same analysis before the terms are looked up in the index\n\n### node\n* node is an instance of Elasticsearch\n* multiple nodes can join the same cluster\n* cluster\n    * same data can be spread across multiple servers (replication)\n        * helps performance: adds resources to work with\n        * helps reliability: data is replicated\n    * all nodes need to be on the same network\n        * balancing shards across data centers simply takes too long\n            * example: master issues relocation commands if it detects unbalanced shard distribution\n        * cross-cluster replication (CCR)\n            * allows you to replicate data from one cluster (leader cluster) to another (follower cluster)\n            * example: across data centers, regions, or cloud availability zones\n* roles\n    * master\n        * election: Raft-inspired\n        * maintains the cluster state (node joins/leaves, index creation, shard allocation)\n        * assign shards to nodes\n            * example: when new index is created\n                * based on node capabilities and existing shard distribution\n    * data\n        * stores actual index data (primary and replica shards)\n    * coordinating\n        * maintains a local copy of the cluster state\n            * only the master node updates the cluster state, but all nodes subscribe to it\n        * routes client requests\n            * formula: hash of id % number_of_primary_shards =\u003e picks the target shard\n                * number of primary shards in an index is fixed at the time that an index is created\n            * in particular: Elasticsearch always maps a routing value to a single shard\n        * returns final result\n            * example: merges responses to aggregate results\n        * every node in Elasticsearch can act as a coordinating node\n\n### shard\n* is a Lucene shard: a directory of files containing an inverted index\n* cannot be split or merged easily\n* index is just a logical grouping of physical shards\n    * each shard is actually a self-contained index\n    * example\n        * suppose that an index = 1 terabyte of data\n        * there are two nodes: each with 512 gigabytes available for storing data\n            * the entire index will not fit on either of the nodes\n        * we need some way of splitting the index\n            * sharding comes to the rescue\n* contains\n    * segment files\n    * metadata files (how to read, decode, and interpret the raw data files in a segment)\n        * example: `.fnm` (field names and types)\n    * commit files (which segments to load after a crash or restart)\n        * `segments_N` (snapshot of all current segments)\n        * `segments.gen` (tracks the latest segments_N file)\n        * `write.lock` (prevent concurrent writers)\n* can be hosted on any node within the cluster\n    * not necessarily be distributed across multiple physical or virtual machines\n        * example\n            * 1 terabyte index into four shards (256 gb each)\n            * shards could be distributed across the two nodes (2 per node)\n    * as you add more nodes to the same cluster, existing shards get balanced between all nodes\n* two types of shards: primaries and replicas\n    * primary shard: all operations that affect the index\n        * example: adding, updating, or removing documents\n    * flow\n        1. operation completes on primary shard =\u003e it is forwarded to each of the replica shards\n        1. operation completes on every replica =\u003e responds to the primary shard\n        1. primary shard responds to the client\n    * each document is stored in a single primary shard\n    * replica shard is a copy of a primary shard\n        * are never allocated to the same nodes as the primary shards\n        * serves two purposes\n            * provide high availability in case nodes or shards fail\n            * increase performance for search queries (searches can be executed on all replicas in parallel)\n* documents are distributed evenly between shards\n    * the shard is determined by hashing document id\n    * each shard has an equal hash range\n* two main reasons why sharding is important\n    * allows you to split and thereby scale volumes of data\n    * operations can be distributed across multiple nodes and thereby parallelized\n        * multiple machines can potentially work on the same query\n    \n### segment\n* contains\n    * inverted index\n        * term dictionary\n            * maps term to offset in a posting list\n            * contains document frequency\n            * example\n                * term: \"shoes\" → df = 2, offset = 842\n                    * Lucene know that from offset X should read exactly 2 document entries\n        * postings lists\n            * stores all the information Lucene needs to retrieve and rank documents\n                * example: document where the term appears, term frequency\n    * stored fields (original document fields)\n    * doc values (columnar storage for sorting, aggregations, faceting)\n        * example\n            ```\n            DocID   price (doc value)\n            --------------------------\n            0       59.99\n            1       19.99\n            2       129.99\n            ```\n    * norms (field-level stats for scoring)\n        * example: length of field\n            * longer fields tend to be less precise\n* involves 10+ small files per segment\n    * in particular: 100 segments =\u003e 1000+ files\n        * problem: file handle exhaustion\n        * solution: .cfs (compound file format)\n            * Lucene can read them as if they were separate files (using random-access lookups inside .cfs)\n    * not compressed\n        * just a flat concatenation of multiple Lucene data files\n* is immutable\n    * new ones are created as you index new documents\n    * deleting only marks documents as deleted\n        * Lucene supports deletes via live docs bitmap, not by physically removing the data immediately\n        * cleaned up during segment merges\n    * updating documents implies re-indexing\n        * updating a document can’t change the actual document; it can only index a new one\n    * are easily cached, making searches fast\n* Lucene queries all its segments, merge the results, and send them back\n    * normal indexing operations create many such small segments\n    * the more segments you have to go though, the slower the search\n        * solution: merging\n            * creating new and bigger segments with combined content\n                * commit: writes a new `segments_N` listing new merged segment and not segments that were merged\n                * example: excluding the deleted documents\n            * tiered - default merge policy\n                * segments divided into tiers by size\n                    * example\n                        ```\n                        Tier 1: segments ≤ 5 MB\n                        Tier 2: segments ≤ 25 MB\n                        Tier 3: segments ≤ 150 MB\n                        ...\n                        ```\n                * each tier has a threshold number of segments\n                    * if threshold hit in a tier =\u003e merge in that tier\n                * prioritizes merging small segments first (cheap, fast)\n                    * avoids merging huge segments unless necessary\n\n### scoring\n* TF: how often a term occurs in the document\n* IDF: the token's importance is inversely proportional to the number of occurrences across all of the documents\n    * `IDF = log(N / df)`\n        * N = total number of documents\n        * df = number of documents containing the term\n* Lucene’s default scoring formula, known as TF-IDF\n    * apart from normalization \u0026 other factors, in general, it is simply: `TF * IDF`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtumilowicz%2Fjava12-elasticsearch-inverted-index-workshop/lists"}