{"id":21807130,"url":"https://github.com/cblearn/cblearn-benchmark","last_synced_at":"2026-04-18T13:39:30.505Z","repository":{"id":187072178,"uuid":"511054178","full_name":"cblearn/cblearn-benchmark","owner":"cblearn","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-12T20:47:18.000Z","size":83189,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-09T23:15:08.993Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cblearn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-06T08:32:05.000Z","updated_at":"2024-05-12T20:47:24.000Z","dependencies_parsed_at":"2024-11-27T12:47:54.600Z","dependency_job_id":null,"html_url":"https://github.com/cblearn/cblearn-benchmark","commit_stats":null,"previous_names":["cblearn/cblearn-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cblearn/cblearn-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cblearn%2Fcblearn-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cblearn%2Fcblearn-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cblearn%2Fcblearn-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cblearn%2Fcblearn-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cblearn","download_url":"https://codeload.github.com/cblearn/cblearn-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cblearn%2Fcblearn-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31971488,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-27T12:37:14.074Z","updated_at":"2026-04-18T13:39:30.487Z","avatar_url":"https://github.com/cblearn.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cblearn-benchmark\n\nThis repository contains a small empirical comparison of algorithm implementations in [cblearn](https://github.com/dekuenstle/cblearn)\nwith each other and with implementations in different libraries.\n\nAt the moment, only ordinal embedding algorithms are evaluated.\n\n\nHere you find some [results](./results.md).\n\n## Preparation\n\n### Setup a conda environment\n\n```sh\nconda create -n cblearn python==3.10\nconda activate cblearn\n\nconda install h5py seaborn tqdm pandas\nconda install -c conda-forge adjusttext\npip install git+https://github.com/dekuenstle/cblearn.git#egg=cblearn[torch]\npip install jupyterlab\n```\n\n### Download the datasets\n\nThe data will be stored in `./datasets`; the path can be customized with the environment variable `CBLEARN_DATA`.\n(this might take a few minutes)\n\n```sh\nconda activate cblearn\npython scripts/datasets.py\n```\n\n## Run benchmark\n\n\n### On a compute cluster (recommended)\n\n#### Python (via conda)\n\n1. `conda activate cblearn`\n2. `cat runs/py.sh | xargs -L1 sbatch slurm/batchjob.sh`\n\n#### Matlab (via singularity)\n\n1. Download [van der Maaten's STE scripts](https://lvdmaaten.github.io/ste/Stochastic_Triplet_Embedding.html) and extract to `lib/vanderMaaten_STE`\n2. Adjust matlab license file/server in `slurm/mat-batchjob.sh`\n3. `cat runs/mat.sh | tr '\\n' '\\0' | xargs -0n1 sbatch slurm/mat-batchjob.sh` or\n    `singularity run --bind ${PWD}:/home/docker --pwd /home/docker --env MLM_LICENSE_FILE=27000@matlab-campus.uni-tuebingen.de docker://mathworks/matlab:r2022a matlab -sd scripts/ -batch \"embedding('STE', 'car');\"` or (if matlab is available on your system) `sh runs/mat.sh`\n\nsh /mnt/qb/work/wichmann/dkuenstle56/cblearn-benchmark/slurm/mat-batchjob.sh \"matlab -sd scripts/ -batch 'disp(\\\"GNMDS\\\", \\\"car\\\");'\"\nsbatch slurm/mat-batchjob.sh matlab -sd scripts/ -batch \"'embedding(\\\"STE\\\", \\\"material\\\");'\"\n\n\nsingularity run --bind ${PWD}:/home/docker --pwd /home/docker --env MLM_LICENSE_FILE=27000@matlab-campus.uni-tuebingen.de docker://mathworks/matlab:r2022a matlab -sd scripts/ -batch \"embedding('STE', 'car');\"\n#### R\n\n1. Start R and install dependencies. If you are asked, if you want to use a personal library, respond \"yes\".\n\n    ```sh\n    R\n    \u003e install.packages(c('docopt', 'jsonlite', 'MLDS', 'loe'), dependencies=TRUE, repos='http://cran.r-project.org/')\n    ... yes\n    ... yes\n    \u003e q()\n    ```\n3. `cat runs/r.sh | xargs -L1 sbatch slurm/batchjob.sh` or `sh runs/r.sh` or `Rscript scripts/embedding.R SOE car`\n\nWorkaround on our HPC:\n```\necho $SCRATCH\n    /scratch_local/\u003cfoo\u003e\nmkdir $SCRATCH/r-lib\nR\n\u003e install.packages(c('docopt', 'jsonlite', 'MLDS', 'loe'), dependencies=TRUE, repos='http://cran.r-project.org/', lib='/scratch_local/\u003cfoo\u003e/r-lib')\n\u003e q()\ncp -a $SCRATCH/r-lib/* ~/R/x86_64-redhat-linux-gnu-library/3.6/\n```\n\n### Manual\n\n#### Python\n1. Install python environment as described above.\n2. `conda activate cblearn`\n2. Run a single model, e.g. `python scripts/embedding.py SOE car`, or all models `sh runs/py.sh`\n\n#### R\n\n1. Install R (tested with 4.2)\n2. Install dependencies in R.\n    ```R\n    install.packages(c('docopt', 'rjson', 'MLDS', 'loe'), dependencies=TRUE, repos='http://cran.rstudio.com/')\n    ```\n3. Run a single model, e.g. `Rscript scripts/embedding.R SOE car`, or all models `sh runs/r.sh`\n\n#### Matlab (in a container)\n```\n# singularity:\nsingularity run --env MLM_LICENSE_FILE=27000@matlab-campus.uni-tuebingen.de docker://mathworks/matlab:r2022a\n\n# or docker:\ndocker run -it --rm -p 8888:8888 -e MLM_LICENSE_FILE=27000@matlab-campus.uni-tuebingen.de --shm-size=512M mathworks/matlab:r2022a\n```\n\n## Plotting\n\nPlots that visualize the datasets and the comparison's results, like the ones in the paper, are generated with jupyter notebooks.\n\nStart jupyter `jupyter lab .`, and then run the following notebooks:\n\n* `scripts/plot_datasets.ipynb` ![Datasets plot](plots/datasets.png)\n\n\n## Libraries and Algorithms:\n\n**R-language** `R embedding.R \u003calgo\u003e \u003cdataset\u003e \u003cresult\u003e`\n\n* [MLDS](https://cran.r-project.org/web/packages/MLDS/index.html): MLDS algorithm\n* [loe](https://cran.r-project.org/web/packages/loe/index.html): SOE algorithm\n\n**Matlab** `matlab embedding.m -r \"embedding \u003calgo\u003e \u003cdataset\u003e \u003cresult\u003e\"`\n\n* [STE](https://lvdmaaten.github.io/ste/Stochastic_Triplet_Embedding.html): CKL[-K], GNMDS[-K], STE[-K], and tSTE algorithms.\n\n**Python**\n\n\n* [cblearn](https://github.com/dekuenstle/cblearn): MLDS, CKL-X, GNMDS-X, SOE, STE-X, tSTE CKL-GPU[-K], FORTE-GPU[-K], GNMDS-GPU[-K], SOE-GPU, STE-GPU, tSTE-GPU\n\n## Dependencies\n\n### R Dependencies\n\n* [docopt.R](https://github.com/docopt/docopt.R): Command line interface\n* [rjson](https://cran.r-project.org/web/packages/rjson/index.html): JSON loading\n* [MLDS](https://cran.r-project.org/web/packages/MLDS/index.html): MLDS Algorithm\n* [loe](https://cran.r-project.org/web/packages/loe/index.html): SOE Algorithm\n\nIf you don't run the scripts with containers, you can manually install\nthese dependencies to your local R instance with `install.packages(...)`.\n\n\n## Missing data\n\nWe run each algorithm and dataset on a separate cluster entity with 96GB RAM and maximum 24h runtime. Runs that exceeded these limitations failed intentionally. For example, our *FORTE-GPU* algorithm requires too much memory and thus fails on the large *imagenet-v2* dataset.\nSimilarly, the *tSTE* algorithm of vanderMaaten  timed out on the *things* and *imagenet-v2* datasets.\nThe *R* implementation of *SOE* crashed for *imagenet-v2* because \"long vectors\" are not supported by some internal function.\n\n# License\n\nThe scripts in this library are free to use under the MIT License conditions.\nThe plots are shared under [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/) and require attribution.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcblearn%2Fcblearn-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcblearn%2Fcblearn-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcblearn%2Fcblearn-benchmark/lists"}