{"id":13646098,"url":"https://github.com/arbml/taqyim","last_synced_at":"2025-04-21T17:32:06.361Z","repository":{"id":170149843,"uuid":"646117279","full_name":"ARBML/Taqyim","owner":"ARBML","description":"Python intefrace for evaluation on chatgpt models","archived":false,"fork":false,"pushed_at":"2024-02-13T12:56:11.000Z","size":5999,"stargazers_count":19,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-24T18:13:53.795Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ARBML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-27T10:43:49.000Z","updated_at":"2024-05-17T16:02:52.000Z","dependencies_parsed_at":"2024-02-13T15:02:27.908Z","dependency_job_id":null,"html_url":"https://github.com/ARBML/Taqyim","commit_stats":null,"previous_names":["arbml/araevals","arbml/tqeem","arbml/taqyim"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ARBML%2FTaqyim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ARBML%2FTaqyim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ARBML%2FTaqyim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ARBML%2FTaqyim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ARBML","download_url":"https://codeload.github.com/ARBML/Taqyim/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250100602,"owners_count":21374973,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:02:48.411Z","updated_at":"2025-04-21T17:32:01.351Z","avatar_url":"https://github.com/ARBML.png","language":"Jupyter Notebook","funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# Taqyim تقييم\n\n\u003cp align=\"center\"\u003e\n    \u003cimg width = \"150px\" src=\"https://github.com/ARBML/Taqyim/assets/15667714/6710535a-4d0b-4c1a-8c35-49b2e2110600\"\u003e\u003c/img\u003e\n\u003c/p\u003e\n\nA library for evaluting Arabic NLP datasets on chatgpt models. \n\n## Installation\n\n```\npip install -e .\n```\n\n## Example \n\n```python\nimport taqyim as tq\npipeline = tq.Pipeline(\n    eval_name=\"ajgt-test\",\n    dataset_name=\"arbml/ajgt_ubc_split\",\n    task_class=\"classification\",\n    task_description= \"Sentiment Analysis\",\n    input_column_name=\"content\",\n    target_column_name=\"label\",\n    prompt=\"Predict the sentiment\",\n    api_key=\"\u003copenai-key\u003e\",\n    train_split=\"train\",\n    test_split=\"test\",\n    model_name=\"gpt-3.5-turbo-0301\",\n    max_samples=1,)\n\n# run the evaluation\npipeline.run()\n\n# show the output data frame\npipeline.show_results()\n\n# show the eval metrics\npipeline.get_final_report()\n\n```\n\n## Run on custom dataset\n\n[custom_dataset.ipynb](notebooks/custom_dataset.ipynb) has a complete example on how to run evaluation on a custom dataset. \n\n\n## parameters\n\n-    `eval_name` choose an eval name\n-    `task_class` class name from supported class names\n-    `task_description` short description about the task\n-    `dataset_name` dataset name for evaluation\n-    `subset` If the dataset has subset\n-    `train_split` train split name in the dataset\n-    `test_split`test split name in the dataset\n-    `input_column_name` input column name in the dataset\n-    `target_column_name` target column name in the dataset\n-    `prompt` the prompt to be fed to the model\n-    `task_description` short string explaining the task\n-    `api_key` api key from [keys](https://platform.openai.com/account/api-keys)\n-    `preprocessing_fn` function used to process inputs and targets \n-    `threads` number of threads used to fetch the api\n-    `threads_timeout` thread timeout \n-    `max_samples` max samples used for evaluation from the dataset \n-    `model_name` choose either `gpt-3.5-turbo-0301` or `gpt-4-0314`\n-    `temperature` temperature passed to the model between 0 and 2, higher temperature means more random results\n-    `num_few_shot` number of fewshot samples to be used for evaluation\n-    `resume_from_record` if `True` it will continue the run from the sample that has no results. \n-    `seed` seed to redproduce the results\n\n## Supported Classes and Tasks\n\n* `Classification` classification tasks see [classification.py](examples/classification.py).\n* `Pos_Tagging` part of speech tagging tasks [pos_tagging.py](examples/pos_tagging.py).\n* `Translation` machine translation [translation.py](examples/translation.py).\n* `Summarization` machine translation [summarization.py](examples/summarization.py).\n* `MCQ` multiple choice question answering [mcq.py](examples/mcq.py).\n* `Rating` rating multiple LLMs outputs [rating.py](examples/rating.py).\n* `Diacritization` machine translation [diacritization.py](examples/diacritization.py).\n\n# Evaluation on Arabic Tasks \n\n|Tasks              |Dataset        |Size       |Metrics\t    |GPT-3.5 \t    |GPT-4      |SoTA|\n| :---              | :---:         | :---:     | :---:         | :---:         | :---:     |:---:|\n|Summarization      |[EASC](https://huggingface.co/datasets/arbml/EASC)\t        |153\t    |RougeL\t        |23.5\t\t    |18.25\t    |13.3|\n|PoS Tagging\t    |[PADT](https://huggingface.co/datasets/universal_dependencies/viewer/ar_padt/train)\t        |680        |Accuracy\t    |75.91\t\t    |86.29\t    |96.83|\n|classification\t    |[AJGT](https://huggingface.co/datasets/ajgt_twitter_ar)\t        |360        |Accuracy\t    |86.94\t\t    |90.30\t    |96.11|\t\n|transliteration\t|[BOLT Egyptian](https://catalog.ldc.upenn.edu/LDC2021T17)✢  |6,653      |BLEU           |13.76\t\t    |27.66\t    |65.88|\n|translation\t    |[UN v1](https://drive.google.com/file/d/13GI1F1hvwpMUGBSa0QC6ov4eE57GC_Zx/view)          |4,000\t    |BLEU\t        |35.05\t\t    |38.83\t    |53.29|\n|Paraphrasing\t    |[APB](https://github.com/marwah2001/Arabic-Paraphrasing-Benchmark)\t        |1,010      |BLEU\t        |4.295\t\t    |6.104\t    |17.52|\n|Diacritization\t    |[WikiNews](https://aclanthology.org/W17-1302/)✢✢      |393\t    |WER/DER\t    |32.74/10.29\t| 38.06/11.64\t\t   |4.49/1.21|\n\n✢ BOLT requires LDC subscription\n\n✢✢ WikiNews not public, contact [authors](https://aclanthology.org/W17-1302/) to access the dataset\n\n```\n@misc{alyafeai2023taqyim,\n      title={Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models}, \n      author={Zaid Alyafeai and Maged S. Alshaibani and Badr AlKhamissi and Hamzah Luqman and Ebrahim Alareqi and Ali Fadel},\n      year={2023},\n      eprint={2306.16322},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farbml%2Ftaqyim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farbml%2Ftaqyim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farbml%2Ftaqyim/lists"}