{"id":19520179,"url":"https://github.com/osu-nlp-group/groundcocoa","last_synced_at":"2026-06-19T14:32:02.969Z","repository":{"id":221665037,"uuid":"754999443","full_name":"OSU-NLP-Group/GroundCocoa","owner":"OSU-NLP-Group","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-08T21:45:27.000Z","size":18072,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-09T03:24:31.904Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://osu-nlp-group.github.io/GroundCocoa/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-09T08:04:42.000Z","updated_at":"2024-04-15T07:59:05.329Z","dependencies_parsed_at":"2024-02-09T10:47:21.529Z","dependency_job_id":"0ded7c7e-550d-49b3-90f8-81996036e395","html_url":"https://github.com/OSU-NLP-Group/GroundCocoa","commit_stats":null,"previous_names":["osu-nlp-group/groundcocoa"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OSU-NLP-Group/GroundCocoa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FGroundCocoa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FGroundCocoa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FGroundCocoa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FGroundCocoa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/GroundCocoa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FGroundCocoa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34536275,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-19T02:00:06.005Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:24:12.392Z","updated_at":"2026-06-19T14:32:02.947Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GroundCocoa\n\nThis is the original implementation of the dataset generation as outlined in our paper -\n\n[GroundCocoa: A Benchmark for Evaluating Compositional \u0026 Conditional Reasoning in Language Models](https://arxiv.org/abs/2404.04237)\n\nThis paper has been accepted at NAACL '25 (main). The final dataset (both test and validation splits) are provided [here](flight_data/groundcocoa.json), and also can be downloaded from [Huggingface🤗](https://huggingface.co/datasets/harsh147/GroundCocoa). GroundCocoa is now also integrated with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) for ease of use. \n\nAn overview of the query generation process is depicted in the figure below:\n\n![Query Generation Process](files/ground_cocoa_generation.png)\n\n## Dependencies\n\nPrimary dependencies are included in the requirements.txt . To install, run - \n\n`pip install -r requirements.txt`\n\n## How to Run (Step-by-Step)\n\nThe individual scripts correspond to various stages of the data creation pipeline as outlined in our paper. Paths corresponding to the intermediate files and other parameters are defined in [constants.py](constants.py). All of the following steps must be run in order.\n\n### Flight Data Scraper\n\nSet the *chrome_path* and *chromedriver_path* in the [constants.py](constants.py) and then run:\n\n`python scrape_flight_data.py`\n\nSource and target airports are randomly selected from the [list of busiest airports](flight_data/busiest_airports.csv) and scrapes flight data separately for each of the three classes - economy, business, and first class. The script requires that you have chromedriver corresponding to your google chrome version downloaded in the main directory of this project.\n\nParts of the code are adapted from the [flight-analysis](https://github.com/celebi-pkg/flight-analysis) project developed by Kaya Celebi. Due to Selenium's dependency on UI elements, small changes to the front-end might cause this code to fail and the element xpaths may have to be updated from time to time. \n\n### Product-of-Sums (POS) \u0026 Primitive Generation\n\n`python requirements_generator.py`\n\nThis script randomly selects flight features and relies on [SymPy](https://www.sympy.org/en/index.html) to generate a POS Expression corresponding to a randomly generated minterms table.The various (slot, minterm) configurations are defined in [constants.py](constants.py) . Primitive constraints are then generated for each occurence of a slot within the POS expression.\n\n### Paraphrasing using GPT-4 Turbo\n\n`python llm_q_generator.py`\n\nWe use GPT-4 Turbo to rephrase our generated primitives and combine them to form the final user requirement. For this script to work, the 'OPENAI_API_KEY' must be set in your environment variables.\n\n### Option Matching\n\n`python construct_final_dataset.py`\n\nFlight options corresponding to each route (scraped in the first step) are matched with the user requirement. Final dataset is created with 5 options (one matching and 4 not matching) for each user query.\n\n## How to Run (Full Pipeline)\n\nWe also provide a method to run the entire pipeline, once the individual scripts are properly configured:\n\n`python run_all.py`\n\n## How to Evaluate\n\nWe provide a sample evaluation script which could run evaluation on GroundCocoa and output the accuracy. The method is currently set up to evaluate GPT-4 Turbo. To evaluate a different model, change the get_model_response() method in the file to query a model of your choice with the \"final_prompt\" input. You can also alter the \"prompting_strategy\" to prompt with partial or full CoT as described in our paper.\n\n`python run_eval.py`\n\nGroundCocoa has been integrated into [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), the most popular framework for language model evaluation, and will be available there soon!\n\n## Citation\n\nIf our find this work useful, please cite our paper:\n\n```\n@misc{kohli2025groundcocoabenchmarkevaluatingcompositional,\n      title={GroundCocoa: A Benchmark for Evaluating Compositional \u0026 Conditional Reasoning in Language Models}, \n      author={Harsh Kohli and Sachin Kumar and Huan Sun},\n      year={2025},\n      eprint={2404.04237},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2404.04237}, \n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fgroundcocoa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Fgroundcocoa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fgroundcocoa/lists"}