{"id":28183989,"url":"https://github.com/ad-freiburg/elevant","last_synced_at":"2025-10-29T04:47:34.594Z","repository":{"id":44457767,"uuid":"277812644","full_name":"ad-freiburg/elevant","owner":"ad-freiburg","description":"Entity linking evaluation and analysis tool","archived":false,"fork":false,"pushed_at":"2025-04-14T10:31:34.000Z","size":148957,"stargazers_count":23,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-16T05:11:55.492Z","etag":null,"topics":["entity-disambiguation","entity-linking","evaluation-framework"],"latest_commit_sha":null,"homepage":"https://elevant.cs.uni-freiburg.de/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ad-freiburg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-07T12:39:14.000Z","updated_at":"2025-04-29T13:46:06.000Z","dependencies_parsed_at":"2023-02-09T03:15:30.326Z","dependency_job_id":"679779bc-b53a-4dbd-9b91-0366ef43acd6","html_url":"https://github.com/ad-freiburg/elevant","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ad-freiburg/elevant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad-freiburg%2Felevant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad-freiburg%2Felevant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad-freiburg%2Felevant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad-freiburg%2Felevant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ad-freiburg","download_url":"https://codeload.github.com/ad-freiburg/elevant/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ad-freiburg%2Felevant/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263074780,"owners_count":23409820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["entity-disambiguation","entity-linking","evaluation-framework"],"created_at":"2025-05-16T05:11:51.340Z","updated_at":"2025-10-29T04:47:29.540Z","avatar_url":"https://github.com/ad-freiburg.png","language":"Python","readme":"# ELEVANT: Entity Linking Evaluation \u0026 Analysis Tool\n\nELEVANT is a tool that helps you evaluate, analyse and compare entity linking systems in detail. You can explore a\n demo instance of the ELEVANT web app at https://elevant.cs.uni-freiburg.de/. If you are using ELEVANT for your\n research please cite our paper\n [\"ELEVANT: A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool\"](https://aclanthology.org/2022.emnlp-demos.8.pdf).\n\nFor the ELEVANT instance of the EMNLP 2023 paper\n[\"A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems\"](https://arxiv.org/abs/2305.14937)\n see https://elevant.cs.uni-freiburg.de/emnlp2023.\n\nWe summarized the most important information and instructions in this README. For further information, please check\nour [Wiki](https://github.com/ad-freiburg/elevant/wiki).\nFor a quick setup guide without lengthy explanations see [Quick Start](https://github.com/ad-freiburg/elevant/wiki/A-Quick-Start).\n\n## Docker Instructions\nGet the code, and build and start the docker container:\n\n    git clone https://github.com/ad-freiburg/elevant.git .\n    docker build -t elevant .\n    docker run -it -p 8000:8000 \\\n        -v \u003cdata_directory\u003e:/data \\\n        -v $(pwd)/evaluation-results/:/home/evaluation-results \\\n        -v $(pwd)/benchmarks/:/home/benchmarks \\\n        -v /var/run/docker.sock:/var/run/docker.sock \\\n        -v $(pwd)/wikidata-types/:/home/wikidata-types \\\n        -e WIKIDATA_TYPES_PATH=$(pwd) elevant\n\nwhere `\u003cdata_directory\u003e` is the directory in which the required data files will be stored. What these data files are\n and how they are generated is explained in section [Get the Data](#get-the-data). Make sure you can read from and\n write to all directories that are being mounted as volumes from within the docker container (i.e. your\n `\u003cdata_directory\u003e`, `evaluation-results` and `benchmarks`), for example (if security is not an issue) by giving all\n users read and write permissions to the directories in question with:\n\n    chmod a+rw -R \u003cdata_directory\u003e evaluation-results/ benchmarks/ wikidata-types/\n\n\nAll the following commands should be run inside the docker container. If you want to use the  system without docker, \n follow the instructions in [Setup without Docker](https://github.com/ad-freiburg/elevant/wiki/Setup-Without-Docker) \nbefore continuing with the next section.\n\n## Get the Data\n\nNote: If you want to use a custom knowledge base instead of Wikidata/Wikipedia/DBpedia you can skip this\n step and instead follow the instructions in [Using a Custom Knowledge Base](https://github.com/ad-freiburg/elevant/wiki/Using-A-Custom-Knowledge-Base).\n\nFor linking entities in text or evaluating the output of a linker, our system needs information about entities and\n mention texts, e.g. entity names, aliases, popularity scores, types, the frequency with which a mention is linked\n to a certain article in Wikipedia, etc. This information is stored in and read from several files. Since these files\n are too large to upload them on GitHub, you can either download them from our servers (fast) or build them yourself\n (slow, RAM intensive, but the resulting files will be based on recent Wikidata and Wikipedia dumps).\n\nTo download the files from our servers, simply run\n\n    make download-all\n\nThis will automatically run `make download-wikidata-mappings`, `make download-wikipedia-mappings` and\n `make download-entity-types-mapping` which will download the compressed files, extract them and move them to the\n correct location. See [Mapping Files](https://github.com/ad-freiburg/elevant/wiki/Mapping-Files) for a description of files downloaded in these steps.\n\nNOTE: This will overwrite existing Wikidata and Wikipedia mappings in your `\u003cdata_directory\u003e` so make sure this is what \n you want to do.\n\nIf you rather want to build the mappings yourself, you can run `make generate-all` to generate all required files, or\n alternatively, replace only a specific *download* command by the corresponding *generate* command.\nSee [Data Generation](https://github.com/ad-freiburg/elevant/wiki/Generating-Data) for more details.\n\n## Start the Web App\n\nTo start the evaluation web app, run\n\n    make start-webapp\n\nYou can then access the webapp at \u003chttp://0.0.0.0:8000/\u003e.\n\nThe evaluation results table contains one row for each experiment. In ELEVANT, an experiment is a run of a\n particular entity linker with particular linker settings on a particular benchmark. We already added a few experiments,\n including oracle predictions (perfect linking results generated from the ground truth), so you can start exploring\n the web app right away. The section [Add a Benchmark](#add-a-benchmark) explains how you can add more benchmarks\n and the section [Add an Experiment](#add-an-experiment) explains how you can add more experiments yourself.\n\nSee [Evaluation Web App](https://github.com/ad-freiburg/elevant/wiki/Web-App) for a detailed overview of the web app's features.\n\n## Add a Benchmark\n\nYou can easily add a benchmark if you have a benchmark file that is in the\n [JSONL format ELEVANT uses internally](https://github.com/ad-freiburg/elevant/wiki/Internally-Used-JSONL-Format), in the common NLP Interchange Format (NIF), in the IOB-based format\n used by Hoffart et al. for their AIDA/CoNLL benchmark or in a very simple JSONL format. Benchmarks in other formats\n have to be converted into one of these formats first.\n\nTo add a benchmark, simply run\n\n    python3 add_benchmark.py \u003cbenchmark_name\u003e -bfile \u003cbenchmark_file\u003e -bformat \u003cours|nif|aida-conll|simple-jsonl\u003e\n\nThis converts the `\u003cbenchmark_file\u003e` into our JSONL format (if it is not in this format already), annotates ground\n truth labels with their Wikidata label and Wikidata types and writes the result to the file\n `benchmarks/\u003cbenchmark_name\u003e.benchmark.jsonl`.\n\nThe benchmark can now be linked with a linker of your choice using the `link_benchmark.py` script with the\n parameter `-b \u003cbenchmark_name\u003e`. See section [Add an Experiment](#add-an-experiment) for details on how to link a\n benchmark and the supported formats.\n\nSee [How To Add A Benchmark](https://github.com/ad-freiburg/elevant/wiki/How-To-Add-A-Benchmark) for more details on adding a benchmark including a description of the\n supported file formats.\n\nMany popular entity linking benchmarks are already included in ELEVANT and can be used with ELEVANT's scripts out of the\n box. See [Benchmarks](https://github.com/ad-freiburg/elevant/wiki/Benchmarks) for a list of these benchmarks.\n\n## Add an Experiment\n\nYou can add an experiment, i.e. a row in the table for a particular benchmark, in two steps: 1) link the benchmark\n articles and 2) evaluate the linking results. Both steps are explained in the following two sections.\n\n### Link Benchmark Articles\nTo link the articles of a benchmark with a single linker configuration, use the script `link_benchmark.py`:\n\n    python3 link_benchmark.py \u003cexperiment_name\u003e -l \u003clinker_name\u003e -b \u003cbenchmark_name\u003e\n\nThe linking results will be written to\n `evaluation-results/\u003clinker_name\u003e/\u003cadjusted_experiment_name\u003e.\u003cbenchmark_name\u003e.linked_articles.jsonl` where\n `\u003cadjusted_experiment_name\u003e` is `\u003cexperiment_name\u003e` in lowercase and characters other than `[a-z0-9-]` replaced by\n `_`.\nFor example\n\n    python3 link_benchmark.py Baseline -l baseline -b kore50\n\nwill create the file `evaluation-results/baseline/baseline.kore50.linked_articles.jsonl`. The result file contains\n one article as JSON object per line. Each JSON object contains benchmark article information such as the article\n title, text, and ground truth labels, as well as the entity mentions produced by the specified linker.\n `\u003cexperiment_name\u003e` is the name that will be displayed in the first column of the evaluation results table in the\n web app.\n\nAlternatively, you can use a NIF API as is used by GERBIL to link benchmark articles. For this, you need to provide\n the URL of the NIF API endpoint as follows:\n\n    python3 link_benchmark.py \u003cexperiment_name\u003e -api \u003capi_url\u003e -pname \u003clinker_name\u003e -b \u003cbenchmark_name\u003e\n\nSee [Linking Benchmark Articles](https://github.com/ad-freiburg/elevant/wiki/How-To-Add-An-Experiment#linking-benchmark-articles) for more information on how to link benchmark articles, including information on how\n you can transform your existing linking result files into our format, and instructions for how to link multiple\n benchmarks using multiple linkers with a single command.\n\nSee [Linkers](https://github.com/ad-freiburg/elevant/wiki/Linkers) for a list of linkers that can be used out of the box with ELEVANT.\n These are for example *ReFinED*, OpenAI's *GPT* (you'll need an OpenAI API key for that), *REL*, *TagMe* (you'll\n need an access token for that which can be obtained easily and free of cost) and *DBpediaSpotlight*.\n\n### Evaluate Linking Results\n\nTo evaluate a linker's predictions use the script `evaluate.py`:\n\n    python3 evaluate.py \u003cpath_to_linking_result_file\u003e\n\nThis will print precision, recall and F1 scores and create two new files where the `.linked_articles.jsonl` file\n extension is replaced by `.eval_cases.jsonl` and `.eval_results.json` respectively. For example\n\n    python3 evaluate.py evaluation-results/baseline/baseline.kore50.linked_articles.jsonl\n\nwill create the files `evaluation-results/baseline/baseline.kore50.eval_cases.jsonl` and\n`evaluation-results/baseline/baseline.kore50.eval_results.json`. The `eval_cases` file contains information about\n each true positive, false positive and false negative case. The `eval_results` file contains the scores that are shown\n in the web app's evaluation results table.\n\nIn the web app, simply reload the page (you might have to disable caching) and the experiment will show up as a row in\n the evaluation results table for the corresponding benchmark.\n\nSee [Evaluating Linking Results](https://github.com/ad-freiburg/elevant/wiki/How-To-Add-An-Experiment#evaluating-linking-results) for instructions on how to evaluate multiple linking\n results with a single command.\n\n\n## Remove an Experiment\nIf you want to remove an experiment from the web app, simply (re)move the corresponding `.linked_articles.jsonl`,\n `.eval_cases.jsonl` and `.eval_results.json` files from the `evaluation-results/\u003clinker_name\u003e/` directory and reload\n the web app (again disabling caching).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fad-freiburg%2Felevant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fad-freiburg%2Felevant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fad-freiburg%2Felevant/lists"}