{"id":19287774,"url":"https://github.com/deeppavlov/deeppavloveval","last_synced_at":"2025-04-22T04:32:16.250Z","repository":{"id":39740425,"uuid":"156695111","full_name":"deeppavlov/deepPavlovEval","owner":"deeppavlov","description":"Sentence embeddings evaluation for russian tasks","archived":false,"fork":false,"pushed_at":"2023-03-24T21:55:16.000Z","size":449,"stargazers_count":8,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-08T07:03:24.214Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deeppavlov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-11-08T11:14:26.000Z","updated_at":"2024-06-25T09:26:31.000Z","dependencies_parsed_at":"2024-04-19T04:30:56.883Z","dependency_job_id":null,"html_url":"https://github.com/deeppavlov/deepPavlovEval","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FdeepPavlovEval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FdeepPavlovEval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FdeepPavlovEval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FdeepPavlovEval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deeppavlov","download_url":"https://codeload.github.com/deeppavlov/deepPavlovEval/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223888420,"owners_count":17220083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:07:10.743Z","updated_at":"2024-11-09T22:07:11.220Z","avatar_url":"https://github.com/deeppavlov.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# deepPavlovEval\nSentence embeddings evaluation for russian tasks.\n\n## Datasets and tasks\nCurrently we support the following tasks:\n\n|  Task                         | Dataset           | Dataset URL                                                            |\n|-------------------------------|-------------------|------------------------------------------------------------------------|\n|  Paraphrase detection         | Paraphraser       | http://paraphraser.ru/                                                 |\n|  Semantic textual similarity  | translated MSRVid | http://files.deeppavlov.ai/datasets/STS2012_MSRvid_translated.tar.gz   |\n|  Natural language inference   | XNLI              | http://www.nyu.edu/projects/bowman/xnli/                               |\n|  Sentiment analysis           | Rusentiment       | http://text-machine.cs.uml.edu/projects/rusentiment/                   |\n\nMSRVid is part of SEMEVAL-2012 TASK 17 dataset. It was automatically translated and checked manually. Original dataset and licence can be found [here](https://www.cs.york.ac.uk/semeval-2012/task6/data/uploads/datasets/train-readme.txt)\n\nAll data can be downloaded via `data/download.sh` script.\n\n# Usage\n\n```python\nfrom deepPavlovEval import Evaluator\nfrom deeppavlov.models.embedders.fasttext_embedder import FasttextEmbedder\n\nevaluator = Evaluator()\nfasttext = FasttextEmbedder('/data/embeddings/wiki.ru.bin', mean=True)\nresults_fasttext = evaluator.evaluate(fasttext)\n```\n\nResults have the following format\n```python\n\u003e\u003e\u003e results_fasttext\n{'paraphraser': {'pearson correlation': 0.37964098780007866},\n 'msrvid': {'pearson correlation': 0.7330145176159141},\n 'xnli': {'knn_f1': 0.3528115762625721,\n  'knn_accuracy': 0.3586826347305389,\n  'svm_f1': 0.46821573045481973,\n  'svm_accuracy': 0.4694610778443114},\n 'rusentiment': {'knn_f1': 0.357914627528629,\n  'knn_accuracy': 0.40310077519379844,\n  'svm_f1': 0.4482865224076,\n  'svm_accuracy': 0.574654533198517}}\n```\n\nIn order to use deepPavlovEval, model should have `.__call__()` method which returns\nsentence embeddings given list of *tokenized* sentences. For example: `[['first', 'sentence'], ['second', 'sentence']]`\n\n```python\nimport numpy as np\nclass MyRandomEmbedder:\n    def __call__(self, batch):\n        return np.random.uniform(size=(len(batch), 300))\n\nmy_embedder = MyRandomEmbedder()\nresults_random = evaluator.evaluate(my_embedder, model_name='random_embedder')\n```\n\nEvaluator object accumulates different experiments. They can be accessed via `.all_results`\nand saved as .jsonl via `.save_results(save_path)`.\n\n```python\n\u003e\u003e\u003e all_results = evaluator.all_results\n\u003e\u003e\u003e all_results\n[{'task': 'paraphraser',\n  'model': deeppavlov.models.embedders.fasttext_embedder.FasttextEmbedder,\n  'metrics': {'pearson correlation': 0.37964098780007866}},\n {'task': 'msrvid',\n  'model': deeppavlov.models.embedders.fasttext_embedder.FasttextEmbedder,\n  'metrics': {'pearson correlation': 0.7330145176159141}},\n {'task': 'xnli',\n  'model': deeppavlov.models.embedders.fasttext_embedder.FasttextEmbedder,\n  'metrics': {'knn_f1': 0.3528115762625721,\n   'knn_accuracy': 0.3586826347305389,\n   'svm_f1': 0.46821573045481973,\n   'svm_accuracy': 0.4694610778443114}},\n {'task': 'rusentiment',\n  'model': deeppavlov.models.embedders.fasttext_embedder.FasttextEmbedder,\n  'metrics': {'knn_f1': 0.357914627528629,\n   'knn_accuracy': 0.40310077519379844,\n   'svm_f1': 0.4482865224076,\n   'svm_accuracy': 0.574654533198517}},\n {'task': 'paraphraser',\n  'model': 'random_embedder',\n  'metrics': {'pearson correlation': -0.004955005150548055}},\n {'task': 'msrvid',\n  'model': 'random_embedder',\n  'metrics': {'pearson correlation': 0.025535004548834218}},\n {'task': 'xnli',\n  'model': 'random_embedder',\n  'metrics': {'knn_f1': 0.32845812688562903,\n   'knn_accuracy': 0.3401197604790419,\n   'svm_f1': 0.33784182130058665,\n   'svm_accuracy': 0.33812375249501}},\n {'task': 'rusentiment',\n  'model': 'random_embedder',\n  'metrics': {'knn_f1': 0.17511439267964699,\n   'knn_accuracy': 0.32861476238624876,\n   'svm_f1': 0.15701808622986574,\n   'svm_accuracy': 0.4236602628918099}}]\n```\n\nRead `deepPavlovEval/evaluator.py` for full api description.\n\nMore examples in `deeppavlov_models.ipynb`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fdeeppavloveval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeppavlov%2Fdeeppavloveval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fdeeppavloveval/lists"}