{"id":21543854,"url":"https://github.com/stefanoghinelli/salton","last_synced_at":"2026-07-11T01:32:05.429Z","repository":{"id":185943563,"uuid":"668879666","full_name":"stefanoghinelli/salton","owner":"stefanoghinelli","description":"Information retrieval system based on Okapi BM25 ranking function.","archived":false,"fork":false,"pushed_at":"2025-09-03T16:44:02.000Z","size":226,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-18T20:34:05.695Z","etag":null,"topics":["bm25","information-retrival","nltk","unimore-informatica","whoosh"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefanoghinelli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-07-20T20:11:35.000Z","updated_at":"2025-10-17T14:22:00.000Z","dependencies_parsed_at":"2024-11-24T05:15:52.884Z","dependency_job_id":"ac3e55c2-4b78-48c6-b6b8-8fb661460990","html_url":"https://github.com/stefanoghinelli/salton","commit_stats":null,"previous_names":["stefanoghinelli/everpaper","stefanoghinelli/salton","stefanoghinelli/just-information-retrieval","stefanoghinelli/salton-ir"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/stefanoghinelli/salton","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanoghinelli%2Fsalton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanoghinelli%2Fsalton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanoghinelli%2Fsalton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanoghinelli%2Fsalton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefanoghinelli","download_url":"https://codeload.github.com/stefanoghinelli/salton/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanoghinelli%2Fsalton/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35348376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bm25","information-retrival","nltk","unimore-informatica","whoosh"],"created_at":"2024-11-24T05:15:51.570Z","updated_at":"2026-07-11T01:32:05.395Z","avatar_url":"https://github.com/stefanoghinelli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [salton](https://en.wikipedia.org/wiki/Gerard_Salton)\n\n🚧 \u003cimg src=\"https://img.shields.io/badge/under%20construction-FF8C00\" /\u003e \u003cimg src=\"https://img.shields.io/badge/beta-blue\"/\u003e 🚧\n\n## Project description\n\nSalton is a vertical search engine built on a corpus of documents sourced from [CORE](https://core.ac.uk), a public repository of open-access research papers.\nThe goal is to provide a more refined search experience than the CORE portal.\nIt uses the [Okapi BM25](https://en.wikipedia.org/wiki/Okapi_BM25) ranking function to estimate the relevance of documents.\nEnd users can formulate queries based on a defined language, results are ranked by relevance with title, score, URL and summary of the abstract.\n\n## Architecture\n\n\u003cimg src=\"assets/diagram.png\" alt=\"diagram\" width=\"600\"/\u003e\n\n## Running the project\n\nThis project runs using Python 3 and pip. To install it as a Python package, do the following:\n\n1. Clone the repository and change directory\n\n```bash\n$ git clone https://github.com/stefanoghinelli/salton.git\n$ cd salton\n```\n\n2. Install from source\n\n```bash\n$ python3 -m venv venv\n$ source venv/bin/activate\n$ pip install -e .\n```\n\n3. Download NLTK corpora\n\n```bash\n$ python3 -c \"import ssl, nltk; ssl._create_default_https_context=ssl._create_unverified_context; [nltk.download(p) for p in ['punkt','stopwords','omw-1.4','wordnet','averaged_perceptron_tagger']]\"\n```\n\n4. Setup environment\n\n```bash\n$ sh setup_scripts/01.prepare_environment.sh\n```\n\n## Command details\n\n```bash\nUsage: salton [OPTIONS] COMMAND [ARGS]...\n\n  Salton: your tool for retrieving papers faster\n\nOptions:\n  --help  Show this and exit.\n\nCommands:\n  fetch       Fetch papers from CORE repository\n  preprocess  Tokenize, lemmatize, remove stopwords\n  index       Build the index\n  search      Query papers by keyword\n  stats       Show statistics\n  benchmark   Run benchmarks\n```\n\n### Usage\n\nThe project builds Salton locally for command line running.\n\nTo fetch papers (100 by default):\n\n```bash\n$ salton fetch -l [number of papers]\n```\n\nTo preprocess papers:\n\n```bash\n$ salton preprocess [--wsd]\n```\n\n`--wsd`: enables word sense disambiguation (off by default)\n\n\u003e [!NOTE]\n\u003e The word sense disambiguation computes similarity between word senses and compares each term against multiple context. This quadratic operation can be highly time consuming.\n\nTo build the index:\n\n```bash\n$ salton index\n```\n\nTo query papers by keyword:\n\n```bash\n$ salton search -q \"[your query]\" -l [number of results]\n```\n\nTo view the statistics:\n\n```bash\n$ salton stats\n\nIndex statistics:\n-Documents indexed: 55\n-Unique terms: 17041\n-Index size: 8.41 MB\n\nData statistics:\n-Raw papers: 61\n-Processed papers: 55\n\nBenchmark statistics:\n-Available query sets: 3\n```\n\n## Evaluation\n\n### Setup benchmarks\nTo run benchmarks, you'll need aset of test queries in the `evaluation` directory:\n   - `query_natural_lang.txt`: natural language queries\n   - `query_benchmark.txt`: structured queries\n   - `query_relevance.txt`: relevance data\n\n### Benchmark metrics\nThe currently supported metrics are precision, recall, NDCG, MAP.\n\nTo run benchmarks:\n\n```bash\n$ salton benchmark [--save/--no-save] [--detailed/--simple]\n```\n\n`--save/--no-save`: results to file (default: save)\n\n`--detailed/--simple`: level of detail in results (default: simple)\n\n## Results\n\n```bash\nsalton search -q \"artificial intelligence\" -l 3\n\n==================================================\n  Results for: artificial intelligence\n==================================================\n\n1. Title: SIR A New Wireless Sensor Network Routing Protocol Based on Artificial Intelligence\n   Score: 23.8540\n   Abstract: Currently, Wireless Sensor Networks (WSNs) are formed by hundreds of\n             low energy and low cost micro-electro-mechanical systems. However,\n             conventional Quality of Service routing models, are not suitable for\n             ad hoc sensor networks, due to the dynamic nature of such systems.\n   URL: https://core.ac.uk/download/161255615.pdf\n\n2. Title: A motion system for social and animated robots\n   Score: 9.9541\n   Abstract: The social robot Probo is used to study Human-Robot Interactions\n             (HRI), with a special focus on Robot Assisted Therapy (RAT). The\n             motion system has a Combination Engine, which combines motion commands\n             that are triggered by a human operator with motions that originate\n             from different units of the cognitive control architecture of the\n             robot.\n   URL: https://core.ac.uk/download/55844762.pdf\n\n3. Title: On the Collaboration of an Automatic Path-Planner and a Human User for Path-Finding in Virtual Industrial Scenes\n   Score: 6.3338\n   Abstract: This paper describes a global interactive framework enabling an\n             automatic path-planner and a user to collaborate for finding a path in\n             cluttered virtual environments. The user can then influence the\n             planner by not following the path and automatically order a new path\n             research.\n   URL: https://core.ac.uk/download/12043111.pdf\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanoghinelli%2Fsalton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefanoghinelli%2Fsalton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanoghinelli%2Fsalton/lists"}