{"id":48086245,"url":"https://github.com/tschaefer/semantic-rss-search","last_synced_at":"2026-04-04T15:20:20.033Z","repository":{"id":345277369,"uuid":"991206586","full_name":"tschaefer/semantic-rss-search","owner":"tschaefer","description":"A Solid Semantic Search for RSS Feeds 🗞️","archived":false,"fork":false,"pushed_at":"2026-04-03T11:40:26.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-03T16:34:06.631Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tschaefer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-27T09:30:16.000Z","updated_at":"2026-04-03T11:40:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tschaefer/semantic-rss-search","commit_stats":null,"previous_names":["tschaefer/semantic-rss-search"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tschaefer/semantic-rss-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tschaefer%2Fsemantic-rss-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tschaefer%2Fsemantic-rss-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tschaefer%2Fsemantic-rss-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tschaefer%2Fsemantic-rss-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tschaefer","download_url":"https://codeload.github.com/tschaefer/semantic-rss-search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tschaefer%2Fsemantic-rss-search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31403955,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T15:20:18.108Z","updated_at":"2026-04-04T15:20:19.986Z","avatar_url":"https://github.com/tschaefer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Solid Semantic Search for RSS Feeds 🗞️\n\nIndex and search RSS feeds using semantic search techniques.\nThis API provides endpoints to embed RSS feed entries into a vector database\nand search them based on semantic similarity.\n\n### Index RSS Feeds\n\nIndex RSS feeds by posting their URLs to the `/embed` endpoint. This will\nparse the feed entries and store title, summary, published date, link, and\nvector embeddings in the database. The vector embeddings are based on the\ncontent of the title and summary.\n\nThe endpoint requires authorization with a bearer token, which must be set\nas an environment variable `SEMANTIC_RSS_SEARCH_API_TOKEN`.\n\n### Search RSS Feeds\n\nSearch indexed RSS feeds by posting a query to the `/search` endpoint.\nThe search will return the top `k` entries that match the query based on\nsemantic similarity. The results include the title, summary, published date,\nlink, distance from the query vector, and the number of tokens of the indexed\ncontent.\n\n### Technology Stack\n\n- [**FastAPI**](https://fastapi.tiangolo.com): For building the API endpoints.\n- [**SQLite Vector**](https://alexgarcia.xyz/sqlite-vec/): For storing vector embeddings in a SQLite database.\n- [**Sentence Transformers**](https://www.sbert.net): For generating vector embeddings from text.\n- [**bge-base-en-v1.5**](https://huggingface.co/BAAI/bge-base-en-v1.5): A pre-trained model optimised for semantic search and passage retrieval (768-dimensional embeddings).\n\n## Getting Started\n\n### Prerequisites\n\nAn environment to run containerized applications, such as Docker or\nKubernetes.\n\nFor local development, you need [Python 3.11](https://www.python.org) or later.\nSet up the necessary environment by running the script `setup.sh`. This will\ncreate a [virtual environment](https://docs.python.org/3/library/venv.html)\nand install the required dependencies.\n\n```bash\nbash setup.sh\n```\n\nFor running tests execute the setup script using the `--with-tests` flag.\nThis will install additional dependencies.\n\n```bash\nbash setup.sh --with-tests\n```\n\n### Running with Docker\n\n```bash\ndocker run --detach \\\n  --name semantic-rss-search \\\n  --publish 8000:8000 \\\n  --environment SEMANTIC_RSS_SEARCH_API_TOKEN=your_token_here \\\n  --volume ./models:/app/models \\\n  --volume ./db:/app/db \\\n  ghcr.io/tschaefer/semantic-rss-search:latest\n```\n\n### Running Locally\n\nExport the required environment variables.\n\n```bash\nexport SEMANTIC_RSS_SEARCH_API_TOKEN=your_token_here\nexport NLTK_DATA=venv/nltk_data\nexport HF_HOME=models\n```\n\nActivate the virtual environment.\n\n```bash\nsource venv/bin/activate\n```\n\nRun the production server using FastAPI and Uvicorn.\n\n```bash\npython run.py\n```\n\nFor development purposes, you can run FastAPI with hot reloading and\nadditional logging information.\n\n```bash\nfastapi run app/main.py\n```\n\n\u003e [!NOTE]\n\u003e On first run, the application will download the pre-trained model.\n\n\n### Running Tests\n\nTo run the tests, ensure you have installed the additional dependencies, see\nthe **Prerequisites** section. Then execute the tests using `pytest` with a\ntemporary SQLite database.\n\n```bash\nSEMANTIC_RSS_SEARCH_DB=/tmp/semantic_rss_search.db pytest -v tests/\nrm -f /tmp/semantic_rss_search.db\n```\n\n## Usage\n\nFind the API documentation at http://localhost:8000/redoc.\n\n### Indexing RSS Feeds\n\n```bash\ncurl --silent --include \\\n    --header \"Authorization: Bearer token_48e46df9b9e3dc6251877724f8328c39da2158fc892846ab6710b1f29afe98eb\" \\\n    --json '{ \"url\": \"https://www.theregister.com/headlines.atom\" }' \\\n    http://localhost:8000/embed\n\nHTTP/1.1 201 Created\ndate: Wed, 28 May 2025 19:14:49 GMT\nserver: uvicorn\ncontent-length: 14\ncontent-type: application/json\n\n{\"entries\":50}\n```\n\n### Searching RSS Feeds\n\n```bash\ncurl --silent \\\n    --json '{ \"query\": \"Is using AI an energy waste?\", \"k\": 3 }' \\\n    http://localhost:8000/search\n{\n  \"query\": \"Is using AI an energy waste?\",\n  \"k\": 3,\n  \"results\": [\n    {\n      \"title\": \"AI's enormous energy appetite can be curbed, but only through lateral thinking\",\n      \"summary\": \"...\",\n      \"published\": 1748331006,\n      \"link\": \"https://go.theregister.com/feed/www.theregister.com/2025/05/27/opinion_column_ai_energy/\",\n      \"distance\": 0.7886583209037781,\n      \"token\": 117\n    },\n    {\n      \"title\": \"'Some Signs of AI Model Collapse Begin To Reveal Themselves'\",\n      \"summary\": \"...\",\n      \"published\": 1748433783,\n      \"link\": \"https://slashdot.org/story/25/05/28/0242240/some-signs-of-ai-model-collapse-begin-to-reveal-themselves?utm_source=rss1.0mainlinkanon\u0026utm_medium=feed\",\n      \"distance\": 0.8932643532752991,\n      \"token\": 525\n    },\n    {\n      \"title\": \"Nothing's Carl Pei Says Your Smartphone's OS Will Replace All of Its Apps\",\n      \"summary\": \"...\",\n      \"published\": 1748433783,\n      \"link\": \"https://mobile.slashdot.org/story/25/05/28/0316239/nothings-carl-pei-says-your-smartphones-os-will-replace-all-of-its-apps?utm_source=rss1.0mainlinkanon\u0026utm_medium=feed\",\n      \"distance\": 0.906494677066803,\n      \"token\": 421\n    }\n  ]\n}\n```\n\n## Contributing\n\nContributions are welcome! Please fork the repository and submit a pull request.\nFor major changes, open an issue first to discuss what you would like to change.\n\nEnsure that your code adheres to the existing style and includes appropriate tests.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftschaefer%2Fsemantic-rss-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftschaefer%2Fsemantic-rss-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftschaefer%2Fsemantic-rss-search/lists"}