{"id":17989470,"url":"https://github.com/evilfreelancer/rumorpheme","last_synced_at":"2025-03-25T23:30:46.719Z","repository":{"id":259708992,"uuid":"862450784","full_name":"EvilFreelancer/ruMorpheme","owner":"EvilFreelancer","description":"Проект языковой модели для проведения морфемного анализа, сегментации и токенизации слов русского языка.","archived":false,"fork":false,"pushed_at":"2025-01-10T01:17:50.000Z","size":5676,"stargazers_count":14,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-19T00:38:32.955Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EvilFreelancer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-24T16:10:19.000Z","updated_at":"2025-02-13T11:59:55.000Z","dependencies_parsed_at":"2024-10-27T14:02:41.746Z","dependency_job_id":"a869335e-0d4a-4142-a091-18cf5fc0f694","html_url":"https://github.com/EvilFreelancer/ruMorpheme","commit_stats":null,"previous_names":["evilfreelancer/rumorpheme"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruMorpheme","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruMorpheme/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruMorpheme/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruMorpheme/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EvilFreelancer","download_url":"https://codeload.github.com/EvilFreelancer/ruMorpheme/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245561145,"owners_count":20635681,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T19:14:45.714Z","updated_at":"2025-03-25T23:30:46.661Z","avatar_url":"https://github.com/EvilFreelancer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ruMorpheme - Russian Morphemes Segmentation\n\nПроект языковой модели для проведения морфемного анализа и сегментации слов русского языка.\n\nОбученная модель способна сегментировать слова, выделяя в них:\n\n- приставки (PREF)\n- корни (ROOT)\n- соединительные гласные (LINK)\n- дефисы (HYPH)\n- суффиксы (SUFF)\n- постфиксы (POSTFIX)\n- окончания (END)\n\nВеса модели [evilfreelancer/ruMorpheme-v0.2](https://huggingface.co/evilfreelancer/ruMorpheme-v0.2) на HuggingFace.\n\nВдохновлён кодовой базой\nпроекта [AlexeySorokin/NeuralMorphemeSegmentation](https://github.com/AlexeySorokin/NeuralMorphemeSegmentation), который\nреализован в рамках\nпубликации \"[Deep Convolutional Networks for Supervised Morpheme Segmentation of Russian Language](https://github.com/AlexeySorokin/NeuralMorphemeSegmentation/blob/master/Articles/MorphemeSegmentation_final.pdf)\"\nза авторством Алексея Сорокина и Анастасии Кравцовой.\n\n## Установка\n\nПроект доступен через PyPi, и его можно установить с помощью pip:\n\n```shell\npip install rumorpheme\n```\n\n## Примеры использования\n\nПосле установки можно использовать модель для сегментации морфем с помощью следующего скрипта:\n\n```python\nimport sys\nimport json\nfrom rumorpheme.model import RuMorphemeModel\nfrom rumorpheme.utils import labels_to_morphemes\n\n# Чтение входных слов из аргументов командной строки\nwords = sys.argv[1:]  # Список слов, переданных через командную строку\n\n# Загрузка модели\nmodel = RuMorphemeModel.from_pretrained(\"evilfreelancer/ruMorpheme-v0.1\")\nmodel.to(\"cuda\")\nmodel.eval()\n\n# Инференс\nall_predictions, all_log_probs = model.predict(words)\n\n# Обработка и отображение результатов\nfor idx, word in enumerate(words):\n    morphs, morph_types, morph_probs = labels_to_morphemes(\n        word.lower(),\n        all_predictions[idx],\n        all_log_probs[idx]\n    )\n\n    results = []\n    for morpheme, morpheme_type, morpheme_prob in zip(morphs, morph_types, morph_probs):\n        results.append({\"text\": morpheme, \"type\": morpheme_type, \"prob\": str(morpheme_prob.round(2))})\n\n    output = {\"word\": word, \"morphemes\": results}\n    print(json.dumps(output, ensure_ascii=False))\n```\n\n## Пример работы модели\n\nВ случае если вы используете скрипт предикшена из примера выше, то результат будет выглядеть следующим образом:\n\n```json lines\n{\"word\": \"В\", \"morphemes\": [{\"text\": \"в\", \"type\": \"ROOT\", \"prob\": \"98.59\"}]}\n{\"word\": \"воскресенье\", \"morphemes\": [{\"text\": \"воскрес\", \"type\": \"ROOT\", \"prob\": \"99.3\"}, {\"text\": \"ень\", \"type\": \"SUFF\", \"prob\": \"96.58\"}, {\"text\": \"е\", \"type\": \"END\", \"prob\": \"100.0\"}]}\n{\"word\": \"мы\", \"morphemes\": [{\"text\": \"мы\", \"type\": \"ROOT\", \"prob\": \"99.77\"}]}\n{\"word\": \"решили\", \"morphemes\": [{\"text\": \"решил\", \"type\": \"ROOT\", \"prob\": \"85.8\"}, {\"text\": \"и\", \"type\": \"END\", \"prob\": \"100.0\"}]}\n{\"word\": \"перезапланировать\", \"morphemes\": [{\"text\": \"пере\", \"type\": \"PREF\", \"prob\": \"100.0\"}, {\"text\": \"за\", \"type\": \"PREF\", \"prob\": \"77.91\"}, {\"text\": \"план\", \"type\": \"ROOT\", \"prob\": \"98.43\"}, {\"text\": \"ир\", \"type\": \"SUFF\", \"prob\": \"100.0\"}, {\"text\": \"ова\", \"type\": \"SUFF\", \"prob\": \"99.98\"}, {\"text\": \"ть\", \"type\": \"SUFF\", \"prob\": \"98.37\"}]}\n```\n\nА вот так её можно будет заставить выводить результат:\n\n```shell\nВ\tв:ROOT\t98.59\nвоскресенье\tвоскрес:ROOT/ень:SUFF/е:END\t99.30 96.58 100.00\nмы\tмы:ROOT\t99.77\nрешили\tрешил:ROOT/и:END\t85.80 100.00\nперезапланировать\tпере:PREF/за:PREF/план:ROOT/ир:SUFF/ова:SUFF/ть:SUFF\t100.00 77.91 98.43 100.00 99.98 98.37\n```\n\nЕсли форматировать вывод:\n\n```python\n# Обработка и отображение результатов\nfor idx, word in enumerate(words):\n    morphs, morph_types, morph_probs = labels_to_morphemes(\n        word.lower(),\n        all_predictions[idx],\n        all_log_probs[idx]\n    )\n\n    # Комбинируем морфемы и их типы через косую черту\n    morpheme_with_types = [\n        f\"{morpheme}:{morpheme_type}\"\n        for morpheme, morpheme_type in zip(morphs, morph_types)\n    ]\n\n    # Добавляем вероятности к морфемам\n    morpheme_str = '/'.join(morpheme_with_types)\n    probs_str = \" \".join(f\"{prob:.2f}\" for prob in morph_probs)\n    output_line = f\"{word}\\t{morpheme_str}\\t{probs_str}\\n\"\n    print(output_line)\n```\n\n## Про ручное обучение\n\nСклонируем проект и подготовим окружение:\n\n```shell\ngit clone https://github.com/EvilFreelancer/ruMorpheme.git\ncd ruMorpheme\npython3 -m venv venv\npip install -r requirements.txt\n```\n\nАктивируем окружение:\n\n```shell\nsource venv/bin/activate\n```\n\n### Тренировка модели\n\n```shell\npython3 train.py config/ruMorpheme.json\n```\n\nПо завершению тренировки будут созданы:\n\n- `model/pytorch_model.bin` - веса модели\n- `model/config.json` - конфигурация модели\n- `model/vocab.json` - словарь необходимый для работы предикшена\n\n### Валидация обученной модели\n\n```shell\npython3 eval.py\n```\n\nОтчёт валидации будет в `models/evaluation_report.txt`.\n\n### Инференс обученной модели\n\nЗапуск тестового инференса из файла [input_text.txt](./input_text.txt):\n\n```shell\npython predict_file.py input_text.txt --model-path=evilfreelancer/ruMorpheme-v0.1\n```\n\nЕсли не указывать `--model-path` то модель и конфигурация будут прочитаны из директории `./model`.\n\n## Лицензия\n\nЭтот проект лицензирован под лицензией `MIT`. Подробности см. в файле [LICENSE](./LICENSE).\n\n## Цитирование\n\n```\n@misc{rumorpheme2024sources,\n    title={ruMorpheme - Russian Morphemes Segmentation},\n    author={Pavel Rykov},\n    year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Frumorpheme","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevilfreelancer%2Frumorpheme","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Frumorpheme/lists"}