{"id":13677270,"url":"https://github.com/sisinflab/elliot","last_synced_at":"2026-02-01T07:31:13.947Z","repository":{"id":43292848,"uuid":"306702552","full_name":"sisinflab/elliot","owner":"sisinflab","description":"Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation","archived":false,"fork":false,"pushed_at":"2026-01-28T19:00:43.000Z","size":82596,"stargazers_count":295,"open_issues_count":16,"forks_count":54,"subscribers_count":11,"default_branch":"master","last_synced_at":"2026-01-29T09:38:41.079Z","etag":null,"topics":["bprmf","collaborative-filtering","content-based-recommendation","deepfm","funksvd","k-nearest-neighbours","k-nn","keras","knowledge-graph","machine-learning","matrix-factorization","neural-collaborative-filtering","nfm","recommendations","recommender-system","slim","svdpp","tensorflow2","vae"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sisinflab.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-10-23T17:23:45.000Z","updated_at":"2026-01-15T14:37:08.000Z","dependencies_parsed_at":"2023-01-23T20:00:21.941Z","dependency_job_id":"4c3c100c-db26-4944-967b-7822d605ab40","html_url":"https://github.com/sisinflab/elliot","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/sisinflab/elliot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sisinflab%2Felliot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sisinflab%2Felliot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sisinflab%2Felliot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sisinflab%2Felliot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sisinflab","download_url":"https://codeload.github.com/sisinflab/elliot/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sisinflab%2Felliot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28972542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T06:46:42.625Z","status":"ssl_error","status_checked_at":"2026-02-01T06:44:56.173Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bprmf","collaborative-filtering","content-based-recommendation","deepfm","funksvd","k-nearest-neighbours","k-nn","keras","knowledge-graph","machine-learning","matrix-factorization","neural-collaborative-filtering","nfm","recommendations","recommender-system","slim","svdpp","tensorflow2","vae"],"created_at":"2024-08-02T13:00:39.688Z","updated_at":"2026-02-01T07:31:13.941Z","avatar_url":"https://github.com/sisinflab.png","language":"Python","funding_links":[],"categories":["Python","其他_推荐系统"],"sub_categories":["网络服务_其他"],"readme":"# ELLIOT\n\n![PyPI - Python Version](https://img.shields.io/badge/python-3.6%7C3.7%7C3.8-blue) [![Version](https://img.shields.io/badge/version-v0.2.1-green)](https://github.com/sisinflab/elliot) ![GitHub repo size](https://img.shields.io/github/repo-size/sisinflab/elliot) ![GitHub](https://img.shields.io/github/license/sisinflab/elliot.svg)\n\n[Docs] | [Paper]\n\n[Docs]: https://elliot.readthedocs.io/en/latest/\n\n[Paper]: https://doi.org/10.1145/3404835.3463245\n\n### A NEW VERSION OF [ELLIOT(v0.3.1) IS OUT](https://github.com/sisinflab/elliot/releases/)\n\nElliot is a comprehensive recommendation framework that analyzes the recommendation problem from the researcher's perspective.\nIt conducts a whole experiment, from dataset loading to results gathering.\nThe core idea is to feed the system with a simple and straightforward configuration file that drives the framework \nthrough the experimental setting choices.\nElliot untangles the complexity of combining splitting strategies, hyperparameter model optimization, model training, \nand the generation of reports of the experimental results.\n\n![system schema](img/elliot_img.png)\n\nThe framework loads, filters, and splits the data considering a vast set of strategies (splitting methods and filtering \napproaches, from temporal training-test splitting to nested K-folds Cross-Validation).\nElliot optimizes hyperparameters for several recommendation algorithms, selects the best models, compares them with the \nbaselines providing intra-model statistics, computes metrics spanning from accuracy to beyond-accuracy, bias, and fairness, \nand conducts statistical analysis (Wilcoxon and Paired t-test).\n\nElliot aims to keep the entire experiment reproducible and put the user in control of the framework.\n\nWe did our best to put all the bibliographic information of the methods, techniques, and metrics available in Elliot to give the right credits to their authors. Please, remember to cite them when you use Elliot in your research.\n\nActually, the same holds also for Elliot :-) If you find Elliot useful for your research or development, remember to cite the following [paper](https://doi.org/10.1145/3404835.3463245):\n\n```bibtex\n@inproceedings{DBLP:conf/sigir/AnelliBFMMPDN21,\n  author    = {Vito Walter Anelli and\n               Alejandro Bellog{\\'{\\i}}n and\n               Antonio Ferrara and\n               Daniele Malitesta and\n               Felice Antonio Merra and\n               Claudio Pomo and\n               Francesco Maria Donini and\n               Tommaso Di Noia},\n  editor    = {Fernando Diaz and\n               Chirag Shah and\n               Torsten Suel and\n               Pablo Castells and\n               Rosie Jones and\n               Tetsuya Sakai},\n  title     = {Elliot: {A} Comprehensive and Rigorous Framework for Reproducible\n               Recommender Systems Evaluation},\n  booktitle = {{SIGIR} '21: The 44th International {ACM} {SIGIR} Conference on Research\n               and Development in Information Retrieval, Virtual Event, Canada, July\n               11-15, 2021},\n  pages     = {2405--2414},\n  publisher = {{ACM}},\n  year      = {2021},\n  url       = {https://doi.org/10.1145/3404835.3463245},\n  doi       = {10.1145/3404835.3463245},\n  timestamp = {Thu, 15 Jul 2021 15:30:48 +0200},\n  biburl    = {https://dblp.org/rec/conf/sigir/AnelliBFMMPDN21.bib},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n```\n\n\n## Installation\nElliot works with the following operating systems:\n\n* Linux\n* Windows 10\n* macOS X\n\nElliot requires Python version 3.6 or later.\n\nElliot requires tensorflow version 2.3.2 or later. If you want to use Elliot with GPU,\nplease ensure that CUDA or cudatoolkit version is 7.6 or later.\nThis requires NVIDIA driver version \u003e= 10.1 (for Linux and Windows10).\n\nPlease refer to this [document](https://www.tensorflow.org/install/source#gpu) for further \nworking configurations.\n\n\n### Install from source\n\n#### CONDA\n```bash\ngit clone https://github.com//sisinflab/elliot.git \u0026\u0026 cd elliot\nconda create --name elliot_env python=3.8\nconda activate elliot_env\npip install --upgrade pip\npip install -e . --verbose\n```\n\n#### VIRTUALENV\n```bash\ngit clone https://github.com//sisinflab/elliot.git \u0026\u0026 cd elliot\nvirtualenv -p /usr/bin/python3.6 venv # your python location and version\nsource venv/bin/activate\npip install --upgrade pip\npip install -e . --verbose\n```\n\n## Quick Start\n\nElliot's entry point is the function `run_experiment`, which accepts a configuration file that drives the whole experiment. \nIn the following, a sample configuration file is shown to demonstrate how a sample and explicit structure can generate a rigorous experiment.\n\n```python\nfrom elliot.run import run_experiment\n\nrun_experiment(\"configuration/file/path\")\n```\n\nThe following file is a simple configuration for an experimental setup. It contains all the instructions to get \nthe MovieLens-1M catalog from a specific path and perform a train test split in a random sample way with a ratio of 20%.\n\nThis experiment provides a hyperparameter optimization with a grid search strategy for an Item-KNN model. Indeed, \nit is seen that the possible values of neighbors are closed in squared brackets. It indicates that two different models \nequipped with two different neighbors' values will be trained and compared to select the best configuration. Moreover, \nthis configuration obliges Elliot to save the recommendation lists with at most 10 items per user as suggest by top_k property.\n\nIn this basic experiment, only a simple metric is considered in the final evaluation study. The candidate metric is nDCG \nfor a cutoff equal to top_k, unless otherwise noted.\n\n```yaml\nexperiment:\n  dataset: movielens_1m\n  data_config:\n    strategy: dataset\n    dataset_path: ../data/movielens_1m/dataset.tsv\n  splitting:\n    test_splitting:\n      strategy: random_subsampling\n      test_ratio: 0.2\n  models:\n    ItemKNN:\n      meta:\n        hyper_opt_alg: grid\n        save_recs: True\n      neighbors: [50, 100]\n      similarity: cosine\n  evaluation:\n    simple_metrics: [nDCG]\n  top_k: 10\n```\n\nIf you want to explore a basic configuration, and an advanced configuration, please refer to:\n\n[basic_configuration](basic_configuration.md)\n\n[advanced_configuration](advanced_configuration.md)\n\nYou can find the full description of the two experiments in the [paper](https://arxiv.org/abs/2103.02590).\n\n## Contributing\n\nThere are many ways to contribute to Elliot! You can contribute code, make improvements to the documentation, report or investigate [bugs and issues](https://github.com/sisinflab/elliot/issues)\n\nWe welcome all contributions from bug fixes to new features and extensions.\n\nFeel free to share with us your custom configuration files. We are creating a vault of reproducible experiments, and we would be glad of mentioning your contribution.\n\nReference Elliot in your blogs, papers, and articles.\n\nTalk about Elliot on social media with the hashtag **#elliotrs**.\n\n\n## The Team\nElliot is developed by\n* Vito Walter Anelli\u003csup id=\"a1\"\u003e[*](#f1)\u003c/sup\u003e (vitowalter.anelli@poliba.it)\n* Alejandro Bellogín (alejandro.bellogin@uam.es)\n* Antonio Ferrara (antonio.ferrara@poliba.it)\n* Daniele Malitesta (daniele.malitesta@poliba.it)\n* Felice Antonio Merra (felice.merra@poliba.it)\n* Claudio Pomo\u003csup id=\"a1\"\u003e[*](#f1)\u003c/sup\u003e (claudio.pomo@poliba.it)\n* Francesco Maria Donini (donini@unitus.it)\n* Tommaso Di Noia (tommaso.dinoia@poliba.it)\n\nIt is maintained by [SisInfLab Group](http://sisinflab.poliba.it/) and [Information Retrieval Group](http://ir.ii.uam.es/).\n\n\u003cb id=\"f1\"\u003e\u003csup\u003e*\u003c/sup\u003e\u003c/b\u003e Corresponding authors\n## License\nELLIOT uses [APACHE2 License](./LICENSE).\n\n## Acknowledgements\n\nSliM and an alternative KNN-CF implementation refer to [RecSys2019_DeepLearning_Evaluation](https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsisinflab%2Felliot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsisinflab%2Felliot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsisinflab%2Felliot/lists"}