{"id":27891136,"url":"https://github.com/dmis-lab/ethic","last_synced_at":"2025-10-12T14:16:46.988Z","repository":{"id":269913721,"uuid":"874670230","full_name":"dmis-lab/ETHIC","owner":"dmis-lab","description":"[NAACL 2025] ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage","archived":false,"fork":false,"pushed_at":"2025-01-17T00:33:07.000Z","size":376,"stargazers_count":15,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-17T10:08:58.930Z","etag":null,"topics":["benchmark","evaluation","long-context"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2410.16848","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmis-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-18T08:50:28.000Z","updated_at":"2025-06-01T12:43:29.000Z","dependencies_parsed_at":"2024-12-27T06:19:22.604Z","dependency_job_id":"177cfa12-983b-4c64-836b-afa0709a99dc","html_url":"https://github.com/dmis-lab/ETHIC","commit_stats":null,"previous_names":["dmis-lab/ethic"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmis-lab/ETHIC","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FETHIC","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FETHIC/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FETHIC/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FETHIC/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmis-lab","download_url":"https://codeload.github.com/dmis-lab/ETHIC/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FETHIC/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279011597,"owners_count":26084963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-12T02:00:06.719Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","evaluation","long-context"],"created_at":"2025-05-05T11:53:37.226Z","updated_at":"2025-10-12T14:16:46.983Z","avatar_url":"https://github.com/dmis-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage\n\n\u003cp align=\"center\"\u003e\n    📃 \u003ca href=\"https://arxiv.org/abs/2410.16848\" target=\"_blank\"\u003ePaper\u003c/a\u003e | 🤗 \u003ca href=\"https://huggingface.co/datasets/dmis-lab/ETHIC\" target=\"_blank\"\u003eDataset\u003c/a\u003e\n\u003c/p\u003e\n\n## 📋 Introduction\n**ETHIC** is a long-context benchmark designed to assess whether LLMs can fully utilize the provided information. ETHIC comprises tasks with high **Information Coverage (IC)** scores (~91%), i.e. the proportion of input context necessary for answering queries.   \n\n![](figs/long_context_figure.JPG)\n\n## ⚒️ Setup\nWe recommend using the following versions for compatibility.\n* PyTorch 2.4.0\n* Cuda 12.1\n```shell\n# create a new environment\nconda create -n ethic python==3.9.19\nconda activate ethic\n\n# install required packages\npip install -r requirements.txt\n```\n## ⏩ Quickstart\nTo use our dataset directly, simply download it using 🤗 Datasets:\n\n```python\nfrom datasets import load_dataset\n\ntask = \"Recalling\" # Choose from \"Recalling\", \"Summarizing\", \"Organizing\", \"Attributing\"\ndataset = load_dataset(\"dmis-lab/ETHIC\", task)[\"test\"]\n```\n\nFor model inference and evaluation, prepare your OpenAI API key (or other keys for authorization) in _api_config.py_, as we utilize `gpt-4o` in the _Summarizing_ task. Also, Qwen2.5 recommends utilizing YaRN for inputs exceeding 32,768 tokens. Make sure to run inference twice: {_use_yarn_=True, _over_32k_only_=True} and {_use_yarn_=False, _under_32k_only_=True}.\n```shell\n# run.sh\n\nCUDA_VISIBLE_DEVICES=1\nexport VLLM_WORKER_MULTIPROC_METHOD=spawn\n\ntask=Attributing # Recalling, Summarizing, Organizing, Attributing\nmodel_name_or_path=meta-llama/Meta-Llama-3.1-8B-Instruct\n\n# use_yarn=True\n# under_32k_only=True\n# over_32k_only=True\n# domain=Medicine\n\ncmd=\"CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES python inference.py \\\n    --task $task \\\n    --model_name_or_path $model_name_or_path\"\n\nif [ \"$use_yarn\" = \"True\" ]; then\n    cmd=\"$cmd --use_yarn\"\nfi\nif [ \"$under_32k_only\" = \"True\" ]; then\n    cmd=\"$cmd --under_32k_only\"\nfi\nif [ \"$over_32k_only\" = \"True\" ]; then\n    cmd=\"$cmd --over_32k_only\"\nfi\nif [ -n \"$domain\" ]; then\n    cmd=\"$cmd --domain $domain\"\nfi\nif [ -n \"$cache_dir\" ]; then\n    cmd=\"$cmd --cache_dir $cache_dir\"\nfi\n\neval $cmd\n```\n\n\n## Citation\n```\n@article{lee2024ethic,\n  title={ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage},\n  author={Lee, Taewhoo and Yoon, Chanwoong and Jang, Kyochul and Lee, Donghyeon and Song, Minju and Kim, Hyunjae and Kang, Jaewoo},\n  journal={arXiv preprint arXiv:2410.16848},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fethic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmis-lab%2Fethic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fethic/lists"}