{"id":19307397,"url":"https://github.com/artefactory/redis-team-thm","last_synced_at":"2026-05-12T23:07:16.797Z","repository":{"id":62646529,"uuid":"551889217","full_name":"artefactory/redis-team-THM","owner":"artefactory","description":"The ultimate CLI tool to help researchers and technical writers","archived":false,"fork":false,"pushed_at":"2022-12-15T05:16:57.000Z","size":4147,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-24T02:49:53.522Z","etag":null,"topics":["arxiv","cli","redis","sentence-transformers"],"latest_commit_sha":null,"homepage":"https://artefactory.github.io/redis-team-THM/","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/artefactory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-15T10:18:06.000Z","updated_at":"2024-01-01T12:44:45.000Z","dependencies_parsed_at":"2023-01-29T02:01:23.049Z","dependency_job_id":null,"html_url":"https://github.com/artefactory/redis-team-THM","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/artefactory/redis-team-THM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fredis-team-THM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fredis-team-THM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fredis-team-THM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fredis-team-THM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/artefactory","download_url":"https://codeload.github.com/artefactory/redis-team-THM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fredis-team-THM/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32960367,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-12T09:19:52.626Z","status":"ssl_error","status_checked_at":"2026-05-12T09:17:33.438Z","response_time":102,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arxiv","cli","redis","sentence-transformers"],"created_at":"2024-11-10T00:10:47.302Z","updated_at":"2026-05-12T23:07:16.783Z","avatar_url":"https://github.com/artefactory.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Team THM Submission\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"backend/thm/data/redis-logo.png\" height=\"25\"/\u003e \u003cbr/\u003e\n    \u003cimg src=\"backend/thm/data/artefact-logo.png\" height=\"20\"/\u003e\u003cbr/\u003e\n    Tom, Henrique, Michel, Corentin\u003cbr/\u003e\n    Oct. - Nov. 2022\u003cbr/\u003e\n    https://artefactory.github.io/redis-team-THM/\u003cbr/\u003e\n    https://thm-cli.community.saturnenterprise.io/api/docs\u003cbr/\u003e\u003cbr/\u003e\n\u003c/div\u003e\n\nThis demo showcases the _vector search similarity_ feature of Redis Enterprise.\n\nRediSearch enables developers to add documents and their embeddings indexes to the database, turning Redis into a vector database that can be used for modern data web applications.\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://asciinema.org/a/3CMfA6anck7oqgw0ZShznDVhR\" target=\"_blank\"\u003e\u003cimg src=\"https://asciinema.org/a/3CMfA6anck7oqgw0ZShznDVhR.svg\" height=500/\u003e\u003c/a\u003e\n\u003c/div\u003e\n\nSee [Architecture](#architecture) to see how it works, and [User Workflow](#user-workflow) to see how it can be used.\n\n------------------------------\n\n## Table of Contents\n\n- [Documentation](#documentation)\n- [History of Changes](#history-of-changes)\n- [Machine Setup](#machine-setup)\n- [Architecture](#architecture)\n- [User Workflow](#user-workflow)\n- [Running The Application](#running-the-application)\n- [Benchmarks](#benchmarks)\n\n------------------------------\n\n## Documentation\n\n- [Basic Demo](https://docsearch.redisventures.com) | [GitHub](https://github.com/RedisVentures/redis-arXiv-search)\n- [Redis Vector Similarity Search](https://redis.io/docs/stack/search/reference/vectors)\n- [Huggingface Tokenizers + Models](https://huggingface.co/sentence-transformers)\n- [Cornell University - arXiv dataset](https://www.kaggle.com/Cornell-University/arxiv), `arxiv-metadata-oai-snapshot.json` file is used\n- [`FastAPI`](https://fastapi.tiangolo.com/), [`pydantic`](https://pydantic-docs.helpmanual.io/), [`redis-om`](https://redis.io/docs/stack/get-started/tutorials/stack-python/)\n- [`redis`](https://redis.io/docs/stack/) see Vector database and JSON storage\n\n## History of Changes\n\n- 1/11 - Added a multi-category classifier, a Question Answering engine and a CLI HTTP client to the backend\n- 31/10 - Draft blog posts and CLI ETL tool\n- 30/10 - Refactored `RedisVentures/redis-arXiv-search` project\n- 27/10 - Setup Redis Cloud Enterprise and Saturn Cloud accounts and [organized](https://github.com/orgs/artefactory/projects/7) within the team\n- 15/10 - Added a blog based on [Pelican](https://getpelican.com)\n- 15/10 - Added CI/CD script\n- 15/10 - Forked from [`RedisVentures/redis-arXiv-search`](https://github.com/RedisVentures/redis-arXiv-search)\n\n## Machine Setup\n\n```sh\nbrew install yarn redis\npip install -r backend/requirements.txt\npip install -r scripts/requirements.txt\n```\n\n## Architecture\n\nThe user will perform searches to the Redis database through a REST API HTTP Server.\n\nWe wrote a small interactive CLI client tool that performs calls to the HTTP Server and returns papers matching the user queries.\n\n```txt\n                        writes pickle and loads index\n+-------------------+      +----------------+\n|                   |      |                |\n|  Redis            +\u003c-----+  ETL CLI       |\n|                   |      |                |\n+--------+----------+      +----------------+\n         ^\n         |  reads search index\n+--------+----------+\n|                   |\n|  FastAPI          |\n|                   |\n+--------+----------+\n         ^\n         |  calls backend\n+--------+----------+      +---------------------+\n|                   |      |                     |\n|  THM CLI          +-----\u003e+  arxiv.org          |\n|                   |      |  wolfram.alpha.com  |\n+-------------------+      +---------------------+\n    researcher uses the THM CLI while writing research\n```\n\n## User Workflow\n\nThis CLI tool is a quick assistant for a researcher daily activities and helps him improves his efficiency.\n\nIt can be used with his text editor and browser and helps him in the process of:\n\n- building bibliography in Markdown or BibTeX formats,\n- checking the PDF papers using arXiv website,\n- checking scientific facts on Wolfram Alpha website.\n\n```mermaid\n  graph TD;\n      welcome_message--\u003echoose_activity;\n      welcome_message--\u003econfigure_parameters;\n      choose_activity--\u003esearch_keywords;\n      search_keywords--\u003eSearch_API;\n      choose_activity--\u003esearch_similar_to;\n      search_similar_to--\u003eSearch_API;\n      choose_activity--\u003efetch_paper_details;\n      fetch_paper_details--\u003eSearch_API;\n      choose_activity--\u003eask_open_question;\n      ask_open_question--\u003eHugginFacePipeline;\n      choose_activity--\u003efind_formula;\n      find_formula--\u003eWolfram_Alpha_API;\n```\n\n## Running The Application\n\n### Backend Application\n\nSetup your Redis Enterprise Cloud then,\n\n```sh\ncd backend/\n./start.sh\n\nopen http://0.0.0.0:8080/api/docs\n```\n\n[Deploy on Saturn Cloud](https://app.community.saturnenterprise.io/dash/resources?recipeUrl=https://github.com/artefactory/redis-team-THM/blob/main/backend/.saturn/thm-backend-deployment-recipe.json)\n\n### Train model\n\n```sh\ncd scripts/\npip install -r requirements.txt\nbash retrain_model.sh\n```\n\n### THM CLI\n\n```sh\ncd scripts/\npip install -r requirements.txt\n./thm-cli.py\n```\n\n### ETL CLI\n\n```sh\ncd scripts/\npip install -r requirements.txt\n./pipeline.sh\n```\n\n### Blog\n\n```sh\n# To preview files locally\npelican blog/content \u0026\u0026 pelican --listen\n\n# To publish on GitHub pages\nmake publish_blog\n```\n\n### Machine Learning Models\n\nThe project uses the [`UKPLab/sentence-transformers`](https://github.com/UKPLab/sentence-transformers) library to compute dense vector representations for sentences found in [Cornell's arXiv corpus](https://www.kaggle.com/Cornell-University/arxiv).\n\nWe found the following models interesting NLP models from the [leaderboard](https://huggingface.co/spaces/mteb/leaderboard) that community built.\n\n- `sentence-transformers/all-mpnet-base-v2` has embeddings of size 768 and relative good performance\n- `sentence-transformers/all-MiniLM-L6-v2`\n- `sentence-transformers/all-MiniLM-L12-v2` has embedding of size 384 and interesting for development as performing inference is faster\n\nWe also used [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/transformers/v3.0.2/model_doc/auto.html#automodelforsequenceclassification) for the problem of multi-category classification.\n\nFor the problem of Question Answering we used [`distilbert-base-cased-distilled-squad`](https://huggingface.co/distilbert-base-cased-distilled-squad).\n\n```mermaid\n  graph TD;\n      sentence-transformers/all-MiniLM-L12-v2--\u003eTHM_API;\n      transformers.AutoModelForSequenceClassification--\u003eTHM_API;\n      THM_API--\u003eTHM_CLI;\n      distilbert-base-cased-distilled-squad--\u003eTHM_CLI;\n```\n\n## Benchmarks\n\nSee on our blog for the [benchmarks we did](https://artefactory.github.io/redis-team-THM/cost-stack.html) to evaluate the full solution.\n\n## Contributions\n\nChanges and improvements are welcome! Feel free to fork and open a pull request into `main`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fredis-team-thm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fartefactory%2Fredis-team-thm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fredis-team-thm/lists"}