{"id":23789915,"url":"https://github.com/blackglory/ultra-nlp","last_synced_at":"2026-04-17T22:30:25.336Z","repository":{"id":41880174,"uuid":"508175782","full_name":"BlackGlory/ultra-nlp","owner":"BlackGlory","description":"🌿 A NLP library.","archived":false,"fork":false,"pushed_at":"2024-03-03T17:34:35.000Z","size":221,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-03T15:53:03.632Z","etag":null,"topics":["crate","library","rust"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/ultra-nlp","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BlackGlory.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-06-28T06:07:14.000Z","updated_at":"2022-10-11T18:50:37.000Z","dependencies_parsed_at":"2024-02-27T21:24:20.212Z","dependency_job_id":"de59b08e-f7ee-4cc1-9c12-95f3ba2a59ef","html_url":"https://github.com/BlackGlory/ultra-nlp","commit_stats":{"total_commits":70,"total_committers":1,"mean_commits":70.0,"dds":0.0,"last_synced_commit":"60aca51d8794ecfbaf711dffceed34874991c463"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackGlory%2Fultra-nlp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackGlory%2Fultra-nlp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackGlory%2Fultra-nlp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackGlory%2Fultra-nlp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BlackGlory","download_url":"https://codeload.github.com/BlackGlory/ultra-nlp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240018691,"owners_count":19734872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crate","library","rust"],"created_at":"2025-01-01T17:17:41.114Z","updated_at":"2026-04-17T22:30:25.243Z","avatar_url":"https://github.com/BlackGlory.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ultra-nlp\n## Install\n```sh\ncargo add ultra-nlp\n```\n\n## Usage\n### ngrams\n```rs\nlet text = \"你好世界\";\n\nlet result = ngrams(text, 2);\n\nassert_eq!(\n    result\n        .into_iter()\n        .collect::\u003cVec\u003c\u0026str\u003e\u003e(),\n    vec![\"你好\", \"好世\", \"世界\"]\n);\n```\n\n### extract_consecutive_chinese_chars\n```rs\nlet text = \"foo中文bar字符baz\";\n\nlet result = extract_consecutive_chinese_chars(text);\n\nassert_eq!(\n    result\n        .into_iter()\n        .collect::\u003cVec\u003c\u0026str\u003e\u003e(),\n    vec![\"中文\", \"字符\"]\n);\n```\n\n### extract_consecutive_letters\n```rs\nlet text = \"foo中文，bar,字符baz\";\n\nlet result = extract_consecutive_letters(text);\n\nassert_eq!(\n    result\n        .into_iter()\n        .collect::\u003cVec\u003c\u0026str\u003e\u003e(),\n    vec![\"foo中文\", \"bar\", \"字符baz\"]\n);\n```\n\n### cedarwood(slow, low memory usage)\n#### Ingore unmatched contents\n```rs\nuse ultra_nlp::BehaviorForUnmatched,\nuse ultra_nlp::cedarwood::{\n    segment_fully,\n    ForwardDictionary,\n};\n\nlet text = \" 南京市长江大桥, hello world \";\nlet dict = ForwardDictionary::new(\n    vec![\"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \"你好世界\"]\n).unwrap();\n\nlet result = segment_fully(\n    text,\n    \u0026dict,\n    BehaviorForUnmatched::Ignore\n);\n\nassert_eq!(\n    result\n        .iter()\n        .map(|x| x.range().extract(text))\n        .collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![\"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\"]\n);\n```\n\n#### Keep unmatched contents as chars\n```rs\nuse ultra_nlp::BehaviorForUnmatched,\nuse ultra_nlp::cedarwood::{\n    segment_fully,\n    ForwardDictionary,\n};\n\nlet text = \" 南京市长江大桥, hello world \";\nlet dict = ForwardDictionary::new(\n    vec![\"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \"你好世界\"]\n).unwrap();\n\nlet result = segment_fully(\n    text,\n    \u0026dict,\n    BehaviorForUnmatched::KeepAsChars\n);\n\nassert_eq!(\n    result\n        .iter()\n        .map(|x| x.range().extract(text))\n        .collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![\n        \" \", \"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \",\", \" \", \"h\", \"e\", \"l\", \"l\", \"o\", \" \", \"w\", \"o\", \"r\", \"l\", \"d\", \" \",\n    ]\n);\n```\n\n#### Keep unmatched ocntents as words\n```rs\nuse ultra_nlp::BehaviorForUnmatched,\nuse ultra_nlp::cedarwood::{\n    segment_fully,\n    ForwardDictionary,\n};\n\nlet text = \" 南京市长江大桥, hello world \";\nlet dict = ForwardDictionary::new(\n    vec![\"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \"你好世界\"]\n).unwrap();\n\nlet result = segment_fully(\n    text,\n    \u0026dict,\n    BehaviorForUnmatched::KeepAsWords\n);\n\nassert_eq!(\n    result\n        .iter()\n        .map(|x| x.range().extract(text))\n        .collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![\n        \" \", \"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \", hello world \",\n    ]\n);\n```\n\n### daachorse(fast, high memory usage)\n#### Ignore unmatched contents\n```rs\nuse ultra_nlp::BehaviorForUnmatched,\nuse ultra_nlp::daachorse::{\n    segment_fully,\n    StandardDictionary,\n};\n\nlet text = \" 南京市长江大桥, hello world \";\nlet dict = StandardDictionary::new(\n    vec![\"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \"你好世界\"]\n).unwrap();\n\nlet result = segment_fully(text, \u0026dict, BehaviorForUnmatched::Ignore);\n\nassert_eq!(\n    result\n        .iter()\n        .map(|x| x.range().extract(text))\n        .collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![\n      \"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\",\n    ]\n);\n```\n\n#### Keep unmatched contents as chars\n```rs\nuse ultra_nlp::BehaviorForUnmatched,\nuse ultra_nlp::daachorse::{\n    segment_fully,\n    StandardDictionary,\n};\n\nlet text = \" 南京市长江大桥, hello world \";\nlet dict = StandardDictionary::new(\n    vec![\"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \"你好世界\"]\n).unwrap();\n\nlet result = segment_fully(text, \u0026dict, BehaviorForUnmatched::KeepAsChars);\n\nassert_eq!(\n    result\n        .iter()\n        .map(|x| x.range().extract(text))\n        .collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![\n        \" \", \"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \",\", \" \", \"h\", \"e\", \"l\", \"l\", \"o\", \" \", \"w\", \"o\", \"r\", \"l\", \"d\", \" \",\n    ]\n);\n```\n\n#### Keep unmatched contents as words\n```rs\nuse ultra_nlp::BehaviorForUnmatched,\nuse ultra_nlp::daachorse::{\n    segment_fully,\n    StandardDictionary,\n};\n\nlet text = \" 南京市长江大桥, hello world \";\nlet dict = StandardDictionary::new(\n    vec![\"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \"你好世界\"]\n).unwrap();\n\nlet result = segment_fully(text, \u0026dict, BehaviorForUnmatched::KeepAsWords);\n\nassert_eq!(\n    result\n        .iter()\n        .map(|x| x.range().extract(text))\n        .collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![\n        \" \", \"南京\", \"南京市\", \"市长\", \"长江\", \"大桥\", \", hello world \",\n    ]\n);\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackglory%2Fultra-nlp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackglory%2Fultra-nlp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackglory%2Fultra-nlp/lists"}