{"id":25390312,"url":"https://github.com/lindera/lindera-tantivy","last_synced_at":"2025-04-04T20:08:42.572Z","repository":{"id":37987555,"uuid":"241506979","full_name":"lindera/lindera-tantivy","owner":"lindera","description":"Lindera tokenizer for Tantivy.","archived":false,"fork":false,"pushed_at":"2024-12-06T06:54:50.000Z","size":117,"stargazers_count":57,"open_issues_count":1,"forks_count":13,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T19:06:47.076Z","etag":null,"topics":["lindera","tantivy","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lindera.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"mosuka","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2020-02-19T01:37:00.000Z","updated_at":"2025-02-25T17:57:26.000Z","dependencies_parsed_at":"2024-08-07T16:07:34.644Z","dependency_job_id":"5bc26e5d-efda-4f56-9cee-0f9d35e475ab","html_url":"https://github.com/lindera/lindera-tantivy","commit_stats":null,"previous_names":["lindera/lindera-tantivy","lindera-morphology/lindera-tantivy"],"tags_count":42,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lindera%2Flindera-tantivy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lindera%2Flindera-tantivy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lindera%2Flindera-tantivy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lindera%2Flindera-tantivy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lindera","download_url":"https://codeload.github.com/lindera/lindera-tantivy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247242678,"owners_count":20907134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lindera","tantivy","tokenizer"],"created_at":"2025-02-15T14:39:29.207Z","updated_at":"2025-04-04T20:08:42.507Z","avatar_url":"https://github.com/lindera.png","language":"Rust","funding_links":["https://github.com/sponsors/mosuka"],"categories":[],"sub_categories":[],"readme":"# Lindera tokenizer for Tantivy\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Join the chat at https://gitter.im/lindera-morphology/lindera](https://badges.gitter.im/lindera-morphology/lindera.svg)](https://gitter.im/lindera-morphology/lindera?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n[Lindera](https://github.com/lindera-morphology/lindera) Tokenizer for [Tantivy](https://github.com/tantivy-search/tantivy).\n\n\n## Usage\n\nMake sure you have activated the required dictionaries for the 　Lindera in Cargo.toml.\nThe following example enables IPADIC.\n\n```\n[dependencies]\nlindera = \"0.38\"\nlindera-tantivy = { version = \"0.38.0\", features = [\"ipadic\"] }\n```\n\n### Basic example\n\n```rust\nfn main() -\u003e tantivy::Result\u003c()\u003e {\n    use tantivy::{\n        collector::TopDocs,\n        doc,\n        query::QueryParser,\n        schema::{IndexRecordOption, Schema, TextFieldIndexing, TextOptions},\n        Document, Index, TantivyDocument,\n    };\n\n    use lindera::dictionary::DictionaryKind;\n    use lindera::{dictionary::load_dictionary_from_kind, mode::Mode, segmenter::Segmenter};\n    use lindera_tantivy::tokenizer::LinderaTokenizer;\n\n    // create schema builder\n    let mut schema_builder = Schema::builder();\n\n    // add id field\n    let id = schema_builder.add_text_field(\n        \"id\",\n        TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_tokenizer(\"raw\")\n                    .set_index_option(IndexRecordOption::Basic),\n            )\n            .set_stored(),\n    );\n\n    // add title field\n    let title = schema_builder.add_text_field(\n        \"title\",\n        TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_tokenizer(\"lang_ja\")\n                    .set_index_option(IndexRecordOption::WithFreqsAndPositions),\n            )\n            .set_stored(),\n    );\n\n    // add body field\n    let body = schema_builder.add_text_field(\n        \"body\",\n        TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_tokenizer(\"lang_ja\")\n                    .set_index_option(IndexRecordOption::WithFreqsAndPositions),\n            )\n            .set_stored(),\n    );\n\n    // build schema\n    let schema = schema_builder.build();\n\n    // create index on memory\n    let index = Index::create_in_ram(schema.clone());\n\n    // Tokenizer with IPADIC\n    let mode = Mode::Normal;\n    let dictionary = load_dictionary_from_kind(DictionaryKind::IPADIC).unwrap();\n    let user_dictionary = None;\n    let segmenter = Segmenter::new(mode, dictionary, user_dictionary);\n    let tokenizer = LinderaTokenizer::from_segmenter(segmenter);\n\n    // register Lindera tokenizer\n    index.tokenizers().register(\"lang_ja\", tokenizer);\n\n    // create index writer\n    let mut index_writer = index.writer(50_000_000)?;\n\n    // add document\n    index_writer.add_document(doc!(\n    id =\u003e \"1\",\n    title =\u003e \"成田国際空港\",\n    body =\u003e \"成田国際空港（なりたこくさいくうこう、英: Narita International Airport）は、千葉県成田市南東部から芝山町北部にかけて建設された日本最大の国際拠点空港である。首都圏東部（東京の東60km）に位置している。空港コードはNRT。\"\n    )).unwrap();\n\n    // add document\n    index_writer.add_document(doc!(\n    id =\u003e \"2\",\n    title =\u003e \"東京国際空港\",\n    body =\u003e \"東京国際空港（とうきょうこくさいくうこう、英語: Tokyo International Airport）は、東京都大田区にある日本最大の空港。通称は羽田空港（はねだくうこう、英語: Haneda Airport）であり、単に「羽田」と呼ばれる場合もある。空港コードはHND。\"\n    )).unwrap();\n\n    // add document\n    index_writer.add_document(doc!(\n    id =\u003e \"3\",\n    title =\u003e \"関西国際空港\",\n    body =\u003e \"関西国際空港（かんさいこくさいくうこう、英: Kansai International Airport）は大阪市の南西35㎞に位置する西日本の国際的な玄関口であり、関西三空港の一つとして大阪国際空港（伊丹空港）、神戸空港とともに関西エアポート株式会社によって一体運営が行われている。\"\n    )).unwrap();\n\n    // commit\n    index_writer.commit()?;\n\n    // create reader\n    let reader = index.reader()?;\n\n    // create searcher\n    let searcher = reader.searcher();\n\n    // create querhy parser\n    let query_parser = QueryParser::for_index(\u0026index, vec![title, body]);\n\n    // parse query\n    let query_str = \"東京\";\n    let query = query_parser.parse_query(query_str)?;\n    println!(\"Query String: {}\", query_str);\n\n    // search\n    let top_docs = searcher.search(\u0026query, \u0026TopDocs::with_limit(10))?;\n    println!(\"Search Result:\");\n    for (_, doc_address) in top_docs {\n        let retrieved_doc: TantivyDocument = searcher.doc(doc_address)?;\n        println!(\"{}\", retrieved_doc.to_json(\u0026schema));\n    }\n\n    Ok(())\n}\n```\n\n### Config by YAML\n\n```rust\nuse std::path::PathBuf;\n\nfn main() -\u003e tantivy::Result\u003c()\u003e {\n    use tantivy::{\n        collector::TopDocs,\n        doc,\n        query::QueryParser,\n        schema::{IndexRecordOption, Schema, TextFieldIndexing, TextOptions},\n        Document, Index, TantivyDocument,\n    };\n\n    use lindera_tantivy::tokenizer::LinderaTokenizer;\n\n    // create schema builder\n    let mut schema_builder = Schema::builder();\n\n    // add id field\n    let id = schema_builder.add_text_field(\n        \"id\",\n        TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_tokenizer(\"raw\")\n                    .set_index_option(IndexRecordOption::Basic),\n            )\n            .set_stored(),\n    );\n\n    // add title field\n    let title = schema_builder.add_text_field(\n        \"title\",\n        TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_tokenizer(\"lang_ja\")\n                    .set_index_option(IndexRecordOption::WithFreqsAndPositions),\n            )\n            .set_stored(),\n    );\n\n    // add body field\n    let body = schema_builder.add_text_field(\n        \"body\",\n        TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_tokenizer(\"lang_ja\")\n                    .set_index_option(IndexRecordOption::WithFreqsAndPositions),\n            )\n            .set_stored(),\n    );\n\n    // build schema\n    let schema = schema_builder.build();\n\n    // create index on memory\n    let index = Index::create_in_ram(schema.clone());\n\n    // Build tokenizer with config file\n    let config_file = PathBuf::from(env!(\"CARGO_MANIFEST_DIR\"))\n        .join(\"./examples\")\n        .join(\"lindera.yml\");\n    let tokenizer = LinderaTokenizer::from_file(config_file.as_path())?;\n\n    // register Lindera tokenizer\n    index.tokenizers().register(\"lang_ja\", tokenizer);\n\n    // create index writer\n    let mut index_writer = index.writer(50_000_000)?;\n\n    // add document\n    index_writer.add_document(doc!(\n    id =\u003e \"1\",\n    title =\u003e \"成田国際空港\",\n    body =\u003e \"成田国際空港（なりたこくさいくうこう、英: Narita International Airport）は、千葉県成田市南東部から芝山町北部にかけて建設された日本最大の国際拠点空港である。首都圏東部（東京の東60km）に位置している。空港コードはNRT。\"\n    )).unwrap();\n\n    // add document\n    index_writer.add_document(doc!(\n    id =\u003e \"2\",\n    title =\u003e \"東京国際空港\",\n    body =\u003e \"東京国際空港（とうきょうこくさいくうこう、英語: Tokyo International Airport）は、東京都大田区にある日本最大の空港。通称は羽田空港（はねだくうこう、英語: Haneda Airport）であり、単に「羽田」と呼ばれる場合もある。空港コードはHND。\"\n    )).unwrap();\n\n    // add document\n    index_writer.add_document(doc!(\n    id =\u003e \"3\",\n    title =\u003e \"関西国際空港\",\n    body =\u003e \"関西国際空港（かんさいこくさいくうこう、英: Kansai International Airport）は大阪市の南西35㎞に位置する西日本の国際的な玄関口であり、関西三空港の一つとして大阪国際空港（伊丹空港）、神戸空港とともに関西エアポート株式会社によって一体運営が行われている。\"\n    )).unwrap();\n\n    // commit\n    index_writer.commit()?;\n\n    // create reader\n    let reader = index.reader()?;\n\n    // create searcher\n    let searcher = reader.searcher();\n\n    // create querhy parser\n    let query_parser = QueryParser::for_index(\u0026index, vec![title, body]);\n\n    // parse query\n    let query_str = \"ＴＯＫＹＯ\";\n    let query = query_parser.parse_query(query_str)?;\n    println!(\"Query String: {}\", query_str);\n\n    // search\n    println!(\"Parsed Query: {:?}\", query);\n    let top_docs = searcher.search(\u0026query, \u0026TopDocs::with_limit(10))?;\n    println!(\"Search Result:\");\n    for (_, doc_address) in top_docs {\n        let retrieved_doc: TantivyDocument = searcher.doc(doc_address)?;\n        println!(\"{}\", retrieved_doc.to_json(\u0026schema));\n    }\n\n    Ok(())\n}\n```\n\n## API reference\n\nThe API reference is available. Please see following URL:\n- \u003ca href=\"https://docs.rs/lindera-tantivy\" target=\"_blank\"\u003elindera-tantivy\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flindera%2Flindera-tantivy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flindera%2Flindera-tantivy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flindera%2Flindera-tantivy/lists"}