https://github.com/dobraczka/kiez-benchmarking
Configurations and results of kiez paper
https://github.com/dobraczka/kiez-benchmarking
Last synced: 8 months ago
JSON representation
Configurations and results of kiez paper
- Host: GitHub
- URL: https://github.com/dobraczka/kiez-benchmarking
- Owner: dobraczka
- Created: 2021-05-17T17:04:02.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2023-03-28T12:36:55.000Z (over 2 years ago)
- Last Synced: 2025-01-01T12:28:06.723Z (10 months ago)
- Language: Python
- Size: 2.43 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# kiez-benchmarking
Configurations and results of kiez paper
## Install dependencies
You can find the necessary depencies in the `pyproject.toml`.
If you have [poetry](https://github.com/python-poetry/poetry) installed simply execute:
```
poetry install
```
## Pre-calculated knowledge graph embeddings
The knowledge graph embeddings produced for and used in our study are available via [Zenodo](https://zenodo.org/record/6258620).
## Pre-calculated results
We make our results available in `results/max_all.csv`.
Our plots can be created with:
```
poetry run python kiezbenchmarking/create_plots.py --use-csv output
```
We make a lot more plots available in `results/additional_report.pdf`.
To recreate this document:
```
poetry run python kiezbenchmarking/create_plots.py --extensive plot_dir
poetry run python kiezbenchmarking/create_report.py plot_dir results/
cd results
pdflatex additional_report.tex
```
## Reproduce results
It is easy to run a single experiment:
```
poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 faiss --candidates 100 --index-key Flat --no-gpu ls --method nicdm
```
This will automatically download any data if necessary.
You can also track your results with [wandb](https://wandb.ai/) using the `--use-wandb` flag, e.g.:
```
poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 --use-wandb faiss --candidates 100 --index-key Flat --no-gpu ls --method nicdm
```
This command shows you the necessary arguments to run an experiment:
```
poetry run python kiezbenchmarking/experiment.py --help
```
The individual nearest neighbor algorithm and hubness reduction method are declared via subcommand, for which you can also get help (after supplying the required arguments of the base command):
```
poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 faiss --help
```
or for the hubness reduction method:
```
poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 faiss --candidates 100 --index-key Flat --use-gpu False ls --help
```
## For archival purposes: Using SEML
We originally used [SEML](https://github.com/TUM-DAML/seml) to keep track of results. Please refer to their instructions to set everything up.
Install the necessary packages inside a conda env (because seml wants you to use a conda env):
```
conda create -n kiez python=3.7.1
conda activate
poetry install
conda deactivate
```
### Queue the experiments
```
seml [db_name] add configs/[path_to_config]
```
### Run them
```
seml [db_name] run
```
Which starts a SLURM job with all the experiments and saves the results in your MongoDB using [Sacred](https://github.com/IDSIA/sacred).