{"id":18832932,"url":"https://github.com/declare-lab/llm-reasoningtest","last_synced_at":"2025-07-18T16:38:31.842Z","repository":{"id":259930339,"uuid":"875498857","full_name":"declare-lab/LLM-ReasoningTest","owner":"declare-lab","description":"Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions","archived":false,"fork":false,"pushed_at":"2025-04-20T09:20:19.000Z","size":1868,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-20T10:29:29.558Z","etag":null,"topics":["gsm8k","humaneval","reasoning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/declare-lab.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-20T06:21:22.000Z","updated_at":"2025-02-02T18:27:23.000Z","dependencies_parsed_at":"2024-12-30T07:23:29.432Z","dependency_job_id":"eae3339e-220e-42f5-a929-435ea9e8702f","html_url":"https://github.com/declare-lab/LLM-ReasoningTest","commit_stats":null,"previous_names":["declare-lab/llm-reasoningtest"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/declare-lab/LLM-ReasoningTest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FLLM-ReasoningTest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FLLM-ReasoningTest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FLLM-ReasoningTest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FLLM-ReasoningTest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/declare-lab","download_url":"https://codeload.github.com/declare-lab/LLM-ReasoningTest/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FLLM-ReasoningTest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265793684,"owners_count":23829180,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gsm8k","humaneval","reasoning"],"created_at":"2024-11-08T01:59:33.786Z","updated_at":"2025-07-18T16:38:31.793Z","avatar_url":"https://github.com/declare-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions\n\nRecent advancements in Large Language Models (LLMs) have showcased striking results on existing logical reasoning benchmarks, with some models even surpassing human performance. However, the true depth of their competencies and robustness in reasoning tasks remains an open question. To this end, in this paper, we focus on two popular reasoning tasks: arithmetic reasoning and code generation. Particularly, we introduce (i) a general ontology of perturbations for math and coding questions, (ii) a semi-automatic method to apply these perturbations, and (iii) two datasets, GSMore and HUMANEVAL-CORE, respectively, of perturbed math and coding problems to probe LLM capabilities in numeric reasoning and coding tasks. Through comprehensive evaluations of both closed-source and open-source\nLLMs, we show a significant performance drop across all the models against the perturbed questions, suggesting that the current LLMs lack robust problem solving skills and structured reasoning abilities in many areas, as defined by our ontology. We open-source the datasets and source codes at: https://github.com/declare-lab/LLM-ReasoningTest.\n\nDatasets available at [MORE](https://huggingface.co/datasets/declare-lab/GSM8k_MORE) and [CORE](https://huggingface.co/datasets/declare-lab/HumanEval_CORE)\n\nThe pipeline of LLMs' reasoning tests for math and coding problems is as follows:\n\n![Alt text](llmrobustness.png)\n\n### Install libraries\n```\npip install -r requirements.txt\n```\n\n### Run our evaluation\nTo run evaluation on our closed-format questions using gpt4o, you should first export your openai_api as\n```\nexport gpt4o_api=\"XXX\"\n```\n\n#### Generate the model outputs\n\n```\n# Math(gsmore.csv)\npython generate_output.py math --filename datasets/gsmore.csv --model_name chiayewken/llama3-8b-gsm8k-rpo\n# Code(human_eval_core.csv)\npython generate_output.py code --filename datasets/human_eval_core.csv --model_name chiayewken/llama3-8b-gsm8k-rpo\n```\n\n#### Print result\n\n```\n# Math(gsmore.csv)\npython print_result.py math_result --filename datasets/gsmore.csv\n# Code(human_eval_core.csv)\npython print_result.py code_result --filename datasets/human_eval_core.csv\n```\n\n\n### Cite\n```\n@inproceedings{Hong2024EvaluatingLM,\n  title={Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions},\n  author={Pengfei Hong and Navonil Majumder and Deepanway Ghosal and Somak Aditya and Rada Mihalcea and Soujanya Poria},\n  year={2024},\n  url={https://api.semanticscholar.org/CorpusID:267028311}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fllm-reasoningtest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeclare-lab%2Fllm-reasoningtest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fllm-reasoningtest/lists"}