{"id":19307418,"url":"https://github.com/artefactory/areyouredis","last_synced_at":"2026-06-11T22:31:44.039Z","repository":{"id":103034617,"uuid":"557227049","full_name":"artefactory/AreYouRedis","owner":"artefactory","description":null,"archived":false,"fork":false,"pushed_at":"2022-11-07T19:46:19.000Z","size":1494,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-24T02:49:57.599Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/artefactory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-25T09:59:08.000Z","updated_at":"2023-03-09T15:45:43.000Z","dependencies_parsed_at":"2023-03-13T15:09:48.988Z","dependency_job_id":null,"html_url":"https://github.com/artefactory/AreYouRedis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/artefactory/AreYouRedis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2FAreYouRedis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2FAreYouRedis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2FAreYouRedis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2FAreYouRedis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/artefactory","download_url":"https://codeload.github.com/artefactory/AreYouRedis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2FAreYouRedis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34221150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T00:10:52.723Z","updated_at":"2026-06-11T22:31:44.021Z","avatar_url":"https://github.com/artefactory.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AreYouRedis - Vector Search Engineering Hackathon \n\nThis repository contains the code base built by the AreYouRedis team to create the \"Darwinian Paper Explorer\" app.\n\nThe hackathon - organised by the MLOps Community, in collaboration with Redis and Saturn Cloud - focused on Vector search engineering, on the arXiv dataset.\n\nFor more information on it, please visit the [hackathon's welcome page](https://hackathon.redisventures.com/)\n\n## Submission summary\n\nThe submission is a streamlit app, hosted on **Saturn Cloud**, and accessible by clicking on [this link](https://pd-youss-areyouredi-ca51af7d090f4f20ab60bfc3d0e70e18.community.saturnenterprise.io/). \n\nThe app's vision is to combine vector search and co-citations graph structures in order to:\n\n- Help the user find arXiv articles linked to a subject he's interested in\n\n- Provide the user with a view of a topic's trend, based on the number of related publications throughout time, along with a trend prediction for the next two years\n\n- Illustrate how the subject of interest has evolved throughout history, with an arc diagram highlighting the founding / most influential papers and co-citation relationships between papers\n\n- Recommend a curated reading list, based on a combination of vector similarity score and paper node degree in the citations graph\n\n\nIt takes as input a user's query - a sentence describing what type of arXiv papers a person is interested in searching - along with a set of filters on the year, categories, and number of similar papers to be retrieved.\n\nUsing **RedisSearch** capabilities, a set of similar articles, linked to the query, is obtained. \n\nFour output sections are then presented: \n\n1. Topic trend \u0026 future projection\n![Topic trend \u0026 future projection](data/topic_trend.png) \n\n\n2. Topic evolution\n\n![Topic evolution](data/topic_evolution.png) \n\n3. Reading list recommender - based on the papers' similarity scores and node degree in the citation graph\n\n![Recommendations](data/reading_list_recommended.png) \n\n4. Papers overview\n\n![Papers overview](data/papers_overview.png) \n\n\n\n## Running locally\n\n### 1 - Setting up the environment\n\nFirst, clone the repository: \n```\ngit clone https://github.com/artefactory/AreYouRedis.git\n```\n\nCreate a virtual environment at the root of your local repository:\n```\npython3 -m virtualenv .venv\nsource .venv/bin/activate\n```\n\nInstall dependencies:\n```\npip install -r requirements.txt\n```\n\nFinally, add the database's password as environment variable:\n```\nexport REDIS_PASSWORD = '{password}'\n```\n\n### 2 - Launching the streamlit\n\nTo launch the app locally, run the following command:\n```  \nstreamlit run app/app.py\n```\n\nThe page below should open in your web browser:\n![Darwinian paper searc](data/Darwinian_paper_explorer.png) \n\n\n## Repository in more details\n\n### 1 - Preprocessing \u0026 Loading data inside the Redis DB\n\nGoing from the raw arXiv dataset to the data uploaded in redis incured several steps:\n\n- The preprocessing operations, as long as utils function to handle the datasets, can be found in the [vector.py](https://github.com/artefactory/AreYouRedis/blob/master/src/vectors.py) script.\n\n- Functions related to data loading, index creation \u0026 similarity search are in [redis_db.py](https://github.com/artefactory/AreYouRedis/blob/master/src/redis_db.py).\n\n- Functions used to gather the citations (see next section) are written in [scholar_citations.py](https://github.com/artefactory/AreYouRedis/blob/master/src/scholar_citations.py).\n\nThose different modules are called in the [custom-single-gpu-arxiv-embeddings.ipynb](https://github.com/artefactory/AreYouRedis/blob/master/notebooks/custom-single-gpu-arxiv-embeddings.ipynb) notebook, where datasets are gathered, cleaned, merged and loaded into Redis.\n\n### 2 - Data enrichment\n\nTo better understand the origin \u0026 links between different articles, we chose to gather their citations - i.e. all other arXiv papers cited in one article.\n\nFor this, we used [Semantic Scholar](https://www.semanticscholar.org/), an online research tool focused on scientific literature.\nWe worked specifically with the [python API](https://pypi.org/project/semanticscholar/), to query the citations.\n\n\n### 3 - Topic evolution\n\nFinally, the code used to generate the different graphs (topic evolution, prediction of the trend \u0026 arc graph) can be found in the [topic evolution](https://github.com/artefactory/AreYouRedis/blob/master/app/features/topic_evolution.py) script.\n\n### Repository Structure\n\n```\n .\n├── LICENSE\n├── Makefile\n├── README.md\n├── app\n│   ├── app.py\n│   ├── config_files\n│   │   └── config.py\n│   ├── features\n│   │   └── topic_evolution.py\n│   ├── main_page.py\n│   ├── style\n│   │   ├── Artefact_logo.png\n│   │   ├── Artefact_small_logo.jpeg\n│   │   ├── Redis_logo.png\n│   │   ├── Saturncloud_logo.webp\n│   │   └── style.css\n│   └── utils\n│       ├── display.py\n│       ├── graph.py\n│       ├── load_css.py\n│       └── widgets.py\n├── entrypoint.sh\n├── notebooks\n│   ├── custom-single-gpu-arxiv-embeddings.ipynb\n│   ├── multi-gpu-arxiv-embeddings.ipynb\n│   └── single-gpu-arxiv-embeddings.ipynb\n├── requirements.in\n├── requirements.txt\n├── setup.py\n└── src\n    ├── categories.py\n    ├── config.py\n    ├── models.py\n    ├── redis_db.py\n    ├── scholar_citations.py\n    └── vectors.py\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fareyouredis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fartefactory%2Fareyouredis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fareyouredis/lists"}