{"id":47603611,"url":"https://github.com/rv64m/open-news-mcp","last_synced_at":"2026-04-01T19:00:51.086Z","repository":{"id":343011161,"uuid":"1175076903","full_name":"rv64m/open-news-mcp","owner":"rv64m","description":"MCP server for ingesting, storing, embedding, and semantically querying financial and macro news with SQLite/Postgres and Qdrant.","archived":false,"fork":false,"pushed_at":"2026-03-08T15:10:51.000Z","size":179,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-08T16:10:49.245Z","etag":null,"topics":["finacial","mcp","news","python","qdrant","rss","sematic-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rv64m.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-07T07:34:48.000Z","updated_at":"2026-03-08T15:10:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rv64m/open-news-mcp","commit_stats":null,"previous_names":["rv64m/open-news-mcp"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/rv64m/open-news-mcp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rv64m%2Fopen-news-mcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rv64m%2Fopen-news-mcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rv64m%2Fopen-news-mcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rv64m%2Fopen-news-mcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rv64m","download_url":"https://codeload.github.com/rv64m/open-news-mcp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rv64m%2Fopen-news-mcp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31291004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["finacial","mcp","news","python","qdrant","rss","sematic-search"],"created_at":"2026-04-01T19:00:16.928Z","updated_at":"2026-04-01T19:00:51.069Z","avatar_url":"https://github.com/rv64m.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Open News MCP\n\nOpen News MCP is a news-focused MCP server for collecting market and macro news, normalizing it into a local database, embedding it into a vector store, and exposing structured and semantic retrieval tools to LLM clients.\n\nIt is designed for workflows where an agent needs to:\n\n- discover available news sources,\n- browse recent news with explicit filters,\n- search semantically related news,\n- inspect clusters of related stories as a graph.\n\n## Why This Project Exists\n\nMost news APIs are optimized for either raw aggregation or UI consumption. MCP-oriented agents need something different:\n\n- stable, tool-friendly method contracts,\n- deterministic metadata such as source, tier, category, and language,\n- local persistence for repeatable queries,\n- semantic retrieval over already-ingested news,\n- operational simplicity for self-hosted setups.\n\nOpen News MCP fills that gap.\n\n## Core Capabilities\n\n- RSS-based ingestion from a curated source catalog in [`src/core/feeds.py`](src/core/feeds.py)\n- Persistent storage in SQLite or PostgreSQL\n- Alembic-based schema management\n- Local or pluggable embedding backend abstraction\n- Qdrant-backed vector search\n- MCP tools for source discovery, structured browsing, and related-news graph search\n- One-shot and long-running sync workers\n\n## System Design\n\n### Architecture\n\n```mermaid\nflowchart LR\n    A[Curated RSS Sources] --\u003e B[Sync Worker]\n    B --\u003e C[Normalized News Storage]\n    B --\u003e D[Source Catalog]\n    B --\u003e E[Embedding Pipeline]\n    E --\u003e F[Qdrant Vector Store]\n    C --\u003e G[MCP Server]\n    D --\u003e G\n    F --\u003e G\n    G --\u003e H[LLM / MCP Client]\n```\n\n### Data Flow\n\n1. `commands/sync.py` selects configured sources from the source catalog.\n2. RSS entries are fetched, normalized, and deduplicated by `url_hash`.\n3. New rows are stored in `news_articles`, and sources are upserted into `sources`.\n4. If `--embed` is enabled, newly inserted rows are embedded immediately.\n5. Vectors are written into Qdrant using a stable point id derived from the article URL.\n6. The MCP server exposes retrieval methods over the database and vector store.\n\n### Main Layers\n\n- Ingestion layer: [`commands/sync.py`](commands/sync.py)\n- Source definitions: [`src/core/feeds.py`](src/core/feeds.py)\n- Persistent storage: [`src/store/`](src/store)\n- Embedding abstraction: [`src/embedding/`](src/embedding)\n- Vector abstraction: [`src/vector/`](src/vector)\n- MCP tools: [`src/tools/`](src/tools)\n- Server entrypoint: [`server.py`](server.py)\n\n## Quick Start\n\n### Requirements\n\n- Python `3.13+`\n- [`uv`](https://github.com/astral-sh/uv)\n- A database:\n  - SQLite for local development, or\n  - PostgreSQL for a multi-process / production deployment\n- Qdrant if you want semantic retrieval and related-news graph search\n\n### 1. Install Dependencies\n\n```bash\nuv sync\n```\n\n### 2. Configure the Environment\n\nCreate your local environment file from the example:\n\n```bash\ncp .env.example .env\n```\n\nFor a minimal SQLite setup:\n\n```bash\nNEWS_HOST=127.0.0.1\nNEWS_PORT=10110\nNEWS_TRANSPORT=streamable-http\n\nNEWS_DATABASE_BACKEND=sqlite\nNEWS_SQLITE_PATH=data/news.db\n\nNEWS_EMBEDDING_BACKEND=local\nNEWS_EMBEDDING_MODEL=Qwen/Qwen3-Embedding-0.6B\nNEWS_EMBEDDING_DEVICE=cpu\n\nNEWS_VECTOR_BACKEND=qdrant\nNEWS_VECTOR_COLLECTION=news_articles\nNEWS_QDRANT_URL=http://127.0.0.1:6333\n```\n\nFor PostgreSQL:\n\n```bash\nNEWS_DATABASE_BACKEND=postgres\nNEWS_DATABASE_URL=postgresql+asyncpg://postgres:postgres@127.0.0.1:5432/open_news_mcp\n```\n\n### 3. Start Qdrant\n\nThe simplest local option is Docker. Qdrant’s official quickstart shows:\n\n```bash\ndocker pull qdrant/qdrant\n\ndocker run -p 6333:6333 -p 6334:6334 \\\n  -v \"$(pwd)/qdrant_storage:/qdrant/storage:z\" \\\n  qdrant/qdrant\n```\n\nOnce started:\n\n- REST API: `http://127.0.0.1:6333`\n- Web UI: `http://127.0.0.1:6333/dashboard`\n- gRPC: `127.0.0.1:6334`\n\nOfficial reference:\n\n- https://qdrant.tech/documentation/quick-start/\n\nIf you prefer embedded local storage instead of a remote Qdrant process, set:\n\n```bash\nNEWS_QDRANT_PATH=data/qdrant\n```\n\n### 4. Run Database Migrations\n\nFor SQLite:\n\n```bash\nuv run alembic upgrade head\n```\n\nFor PostgreSQL using the Postgres-oriented Alembic config:\n\n```bash\nuv run alembic -c alembic.postgres.ini upgrade head\n```\n\nYou can also let the MCP server auto-apply migrations at startup:\n\n```bash\nNEWS_DATABASE_AUTO_MIGRATE=true\n```\n\n### 5. Sync News\n\nOne-shot sync:\n\n```bash\nuv run python commands/sync.py --once\n```\n\nOne-shot sync plus embedding:\n\n```bash\nuv run python commands/sync.py --once --embed\n```\n\nLong-running sync loop:\n\n```bash\nuv run python commands/sync.py --loop\n```\n\nLong-running sync loop with embedding:\n\n```bash\nuv run python commands/sync.py --loop --embed\n```\n\nUseful flags:\n\n- `--categories markets crypto centralbanks`\n- `--sources \"Federal Reserve\" \"Reuters Markets\"`\n- `--verbose`\n\n### 6. Start the MCP Server\n\n```bash\nuv run python server.py\n```\n\nThe server exposes MCP tools over the configured transport. By default:\n\n- host: `127.0.0.1`\n- port: `10110`\n- transport: `streamable-http`\n\n## Running with PM2\n\nPM2 is useful when you want to keep both the MCP server and the sync loop alive on a single host.\n\n### Option A: Simple PM2 Commands\n\nStart the MCP server:\n\n```bash\npm2 start \"uv run python server.py\" --name open-news-mcp-server\n```\n\nStart the sync loop:\n\n```bash\npm2 start \"uv run python commands/sync.py --loop --embed\" --name open-news-mcp-sync\n```\n\nPersist the process list:\n\n```bash\npm2 save\npm2 startup\n```\n\n### Option B: PM2 Ecosystem File\n\nCreate `ecosystem.config.js`:\n\n```js\nmodule.exports = {\n  apps: [\n    {\n      name: \"open-news-mcp-server\",\n      cwd: \"/absolute/path/to/open-news-mcp\",\n      script: \"uv\",\n      args: \"run python server.py\",\n      interpreter: \"none\",\n      env: {\n        NEWS_HOST: \"127.0.0.1\",\n        NEWS_PORT: \"10110\",\n        NEWS_TRANSPORT: \"streamable-http\"\n      }\n    },\n    {\n      name: \"open-news-mcp-sync\",\n      cwd: \"/absolute/path/to/open-news-mcp\",\n      script: \"uv\",\n      args: \"run python commands/sync.py --loop --embed\",\n      interpreter: \"none\"\n    }\n  ]\n};\n```\n\nThen start it:\n\n```bash\npm2 start ecosystem.config.js\npm2 save\n```\n\n## MCP Features\n\nThe MCP server currently registers eight tools:\n\n### `list_sources`\n\nPurpose:\n\n- discover which sources are available in the local source catalog\n- filter by `categories`, `tiers`, `limit`, and `offset`\n- supports both single value and list for `categories`/`tiers` (for example, `\"markets\"` or `[\"markets\"]`)\n\nWhy it exists:\n\n- LLMs should not guess source names\n- source selection should be explicit and inspectable\n\nImplementation:\n\n- tool: [`src/tools/sources.py`](src/tools/sources.py)\n- store query: [`src/store/sources.py`](src/store/sources.py)\n\n### `search_news`\n\nPurpose:\n\n- browse normalized news articles with deterministic filters\n\nSupported filters:\n\n- `published_after`\n- `timespan`\n- `categories`\n- `sources`\n- `tiers`\n- `offset`\n- `sort`\n- supports both single value and list for `categories`/`sources`/`tiers`\n- `published_after` format: `YYYY-MM-DD` (example: `2026-03-01`)\n\nDesign intent:\n\n- this is the structured retrieval tool\n- it does not perform fuzzy or semantic matching\n- it is useful when an agent already knows the scope it wants\n\nOutput extras:\n\n- `query_diagnostics.matched_before_pagination`\n\nImplementation:\n\n- tool: [`src/tools/search.py`](src/tools/search.py)\n- repository: [`src/store/repository.py`](src/store/repository.py)\n\n### `query_news`\n\nPurpose:\n\n- retrieve semantically related news as a ranked list\n\nInput:\n\n- `query`\n- `limit`\n- `offset`\n- `min_score`\n- `published_after` (`YYYY-MM-DD`)\n- `timespan`\n- `categories`\n- `sources`\n- `tiers`\n\nOutput extras:\n\n- `query_diagnostics.top_score`\n- `query_diagnostics.threshold_used`\n- `query_diagnostics.candidate_count`\n- `query_diagnostics.applied_filters`\n\n### `query_related_news_graph`\n\nPurpose:\n\n- retrieve semantically related news from the vector store\n- return the result as a graph instead of a flat list\n\nInput:\n\n- `query`\n- `limit`\n- `offset`\n- `min_score`\n- `published_after`\n- `timespan`\n- `categories`\n- `sources`\n- `tiers`\n\nOutput shape:\n\n- `graph.nodes`\n- `graph.edges`\n\nGraph semantics:\n\n- one query node\n- article nodes returned from vector retrieval\n- `query_match` edges from the query to each article\n- `related` edges between articles that are semantically close to each other\n- duplicate content is deduplicated by title/domain/date fingerprinting\n\nDesign intent:\n\n- support agent reasoning over clusters, not just rankings\n- expose story neighborhoods that are easier to summarize, compare, and route to downstream logic\n\nImplementation:\n\n- tool: [`src/tools/query.py`](src/tools/query.py)\n\n### `list_categories`, `list_tiers`, `list_source_names`, `list_filter_metadata`\n\nPurpose:\n\n- expose valid filter metadata without forcing clients to materialize and deduplicate large `list_sources` responses\n- `list_filter_metadata` returns all categories/tiers/source names in one roundtrip for clean client bootstrap\n\nDesign intent:\n\n- improve discoverability and reduce client-side schema guessing\n\n## Storage Model\n\n### Database Tables\n\n- `sources`\n  - source catalog metadata\n  - category, tier, language, feed URL, tags\n- `news_articles`\n  - normalized article rows\n  - deduplicated by `url_hash`\n  - tracks `is_embedded` so newly inserted or updated rows can be re-embedded\n\n### Vector Storage\n\nQdrant points currently store:\n\n- `article_id`\n- `url`\n- `title`\n- `source`\n- `category`\n- `tier`\n- `domain`\n- `published_at`\n\nPoint ids are derived from the article URL, which keeps upserts stable across repeated syncs.\n\n## Sync and Scheduling Model\n\nThe sync worker uses tier-aware in-process scheduling:\n\n- tier `1`: every `5` minutes\n- tier `2`: every `10` minutes\n- tier `3+`: every `30` minutes\n\nThis keeps high-signal sources fresher without forcing the whole catalog into the same polling interval.\n\nWhen `--embed` is enabled:\n\n- only newly inserted articles from the current sync cycle are embedded\n- the worker does not rescan the full historical backlog on every loop\n\n## Local Development\n\n### Useful Commands\n\n```bash\nuv sync\nuv run alembic upgrade head\nuv run python commands/sync.py --once --embed\nuv run python server.py\nuv run python -m compileall src commands server.py\n```\n\n### Notebooks\n\nThe `playground/` directory contains example notebooks for local exploration:\n\n- [`playground/search_news.ipynb`](playground/search_news.ipynb)\n- [`playground/query_news.ipynb`](playground/query_news.ipynb)\n- [`playground/query_related_news_graph.ipynb`](playground/query_related_news_graph.ipynb)\n\n## Project Structure\n\n```text\n.\n├── alembic/                 # schema migrations\n├── commands/                # operational commands such as sync\n├── playground/              # notebooks for local experimentation\n├── src/\n│   ├── core/                # curated feeds and upstream helpers\n│   ├── embedding/           # embedding abstraction and providers\n│   ├── store/               # database models and repositories\n│   ├── tools/               # MCP tool implementations\n│   └── vector/              # vector store abstraction and providers\n├── .env.example\n├── server.py                # MCP server entrypoint\n└── README.md\n```\n\n## Operational Notes\n\n### SQLite vs PostgreSQL\n\nUse SQLite when:\n\n- you are developing locally\n- you run a single process\n- you want the simplest setup\n\nUse PostgreSQL when:\n\n- you want a more production-oriented database\n- you run multiple processes\n- you want clearer operational separation between app and storage\n\n### Local Embedding Models\n\nThe default local example uses:\n\n- `Qwen/Qwen3-Embedding-0.6B`\n\nThis is a reasonable balance between local deployability and retrieval quality. You can swap it by changing `NEWS_EMBEDDING_MODEL`.\n\n### Logging\n\nThe sync worker defaults to `INFO`.\n\n- `--verbose` enables `DEBUG`\n- `httpx` request logs are suppressed by default\n- transformer and sentence-transformers startup noise is reduced in normal runs\n\n## Limitations\n\n- Remote embedding providers are reserved in config but not implemented yet\n- RSS availability and quality vary by upstream source\n- There is no dedicated reranker yet\n- Related-news graph edges are currently built from post-retrieval semantic similarity rather than a separate graph index\n- The old `playground/query_news.ipynb` reflects an earlier list-based semantic query workflow; the graph notebook is the preferred path for related-news exploration\n\n## Contributing\n\nIssues and pull requests are welcome. Good contribution targets include:\n\n- new source catalog entries,\n- better semantic clustering or reranking,\n- more vector backends,\n- improved filtering in vector search,\n- production deployment recipes,\n- richer MCP tool metadata and examples.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frv64m%2Fopen-news-mcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frv64m%2Fopen-news-mcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frv64m%2Fopen-news-mcp/lists"}