{"id":13935751,"url":"https://github.com/deeppavlov/ru_sentence_tokenizer","last_synced_at":"2025-04-22T04:32:21.150Z","repository":{"id":57463257,"uuid":"139168665","full_name":"deeppavlov/ru_sentence_tokenizer","owner":"deeppavlov","description":"A simple and fast rule-based sentence segmentation. Tested on OpenCorpora and SynTagRus datasets.","archived":false,"fork":false,"pushed_at":"2018-07-04T14:27:53.000Z","size":40,"stargazers_count":53,"open_issues_count":5,"forks_count":5,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-11-08T07:03:29.273Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deeppavlov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-29T15:51:08.000Z","updated_at":"2024-11-01T18:49:52.000Z","dependencies_parsed_at":"2022-09-13T11:00:49.048Z","dependency_job_id":null,"html_url":"https://github.com/deeppavlov/ru_sentence_tokenizer","commit_stats":null,"previous_names":["deepmipt/ru_sentence_tokenizer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fru_sentence_tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fru_sentence_tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fru_sentence_tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fru_sentence_tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deeppavlov","download_url":"https://codeload.github.com/deeppavlov/ru_sentence_tokenizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223754675,"owners_count":17197070,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-07T23:02:03.980Z","updated_at":"2024-11-09T22:07:11.164Z","avatar_url":"https://github.com/deeppavlov.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# ru_sent_tokenize\nA simple and fast rule-based sentence segmentation. Tested on OpenCorpora and SynTagRus datasets.\n\n# Installation\n```\npip install rusenttokenize\n```\n\n# Running\n```ipython\n\u003e\u003e\u003e from rusenttokenize import ru_sent_tokenize\n\u003e\u003e\u003e ru_sent_tokenize('Эта шоколадка за 400р. ничего из себя не представляла. Артём решил больше не ходить в этот магазин')\n['Эта шоколадка за 400р. ничего из себя не представляла.', 'Артём решил больше не ходить в этот магазин']\n```\n\n# Metrics\n\nThe tokenizer has been tested on OpenCorpora and SynTagRus. There are two important metrics. \n\nPrecision. First one is we took single sentences from the datasets and measured how many times tokenizer didn't split them.  \n\nRecall. Second metric is we took two consecutive sentences from the datasets and joined each pair with a space characted. We measured how many times tokenizer correctly splitted a long sentence into two.\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=2\u003etokenizer\u003c/th\u003e\n    \u003cth colspan=3\u003eOpenCorpora\u003c/th\u003e\n    \u003cth colspan=3\u003eSynTagRus\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003cth\u003ePrecision\u003c/th\u003e\n    \u003cth\u003eRecall\u003c/th\u003e\n    \u003cth\u003eExecution Time (sec)\u003c/th\u003e\n    \u003cth\u003ePrecision\u003c/th\u003e\n    \u003cth\u003eRecall\u003c/th\u003e\n    \u003cth\u003eExecution Time (sec)\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd\u003enltk.sent_tokenize\u003c/td\u003e\n      \u003ctd\u003e94.30\u003c/td\u003e\n      \u003ctd\u003e86.06\u003c/td\u003e\n      \u003ctd\u003e8.67\u003c/td\u003e\n      \u003ctd\u003e98.15\u003c/td\u003e\n      \u003ctd\u003e94.95\u003c/td\u003e\n      \u003ctd\u003e5.07\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003enltk.sent_tokenize(x, language='russian')\u003c/td\u003e\n      \u003ctd\u003e95.53\u003c/td\u003e\n      \u003ctd\u003e88.37\u003c/td\u003e\n      \u003ctd\u003e8.54\u003c/td\u003e\n      \u003ctd\u003e98.44\u003c/td\u003e\n      \u003ctd\u003e95.45\u003c/td\u003e\n      \u003ctd\u003e5.68\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003ebureaucratic-labs.segmentator.split\u003c/td\u003e\n      \u003ctd\u003e97.16\u003c/td\u003e\n      \u003ctd\u003e88.62\u003c/td\u003e\n      \u003ctd\u003e359\u003c/td\u003e\n      \u003ctd\u003e96.79\u003c/td\u003e\n      \u003ctd\u003e92.55\u003c/td\u003e\n      \u003ctd\u003e210\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eru_sent_tokenize\u003c/td\u003e\n      \u003ctd\u003e98.73\u003c/td\u003e\n      \u003ctd\u003e93.45\u003c/td\u003e\n      \u003ctd\u003e4.92\u003c/td\u003e\n      \u003ctd\u003e99.81\u003c/td\u003e\n      \u003ctd\u003e98.59\u003c/td\u003e\n      \u003ctd\u003e2.87\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n[Notebook](https://github.com/deepmipt/ru_sentence_tokenizer/blob/master/metrics/calculate.ipynb) shows how the table above was calculated ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fru_sentence_tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeppavlov%2Fru_sentence_tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fru_sentence_tokenizer/lists"}