{"id":47737923,"url":"https://github.com/codex-crusader/le_market_intelligence_platform","last_synced_at":"2026-04-06T02:01:10.913Z","repository":{"id":348456817,"uuid":"1198186507","full_name":"Codex-Crusader/le_Market_Intelligence_Platform","owner":"Codex-Crusader","description":"An explainable market analysis system that combines technical indicators and news sentiment to generate clear buy/sell signals with reasoning through an interactive dashboard","archived":false,"fork":false,"pushed_at":"2026-04-04T13:05:24.000Z","size":573,"stargazers_count":11,"open_issues_count":12,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-04-05T01:10:03.142Z","etag":null,"topics":["cryptocurrency","data-analysis","data-pipeline","financial-analysis","market-analysis","python","sentiment-analysis","stock-market","streamlit","trading-signals"],"latest_commit_sha":null,"homepage":"https://pulseengine.streamlit.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Codex-Crusader.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"Docs/ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"ko_fi":"bhargavaramkrishnapur"}},"created_at":"2026-04-01T07:34:04.000Z","updated_at":"2026-04-04T13:05:29.000Z","dependencies_parsed_at":"2026-04-05T01:01:19.258Z","dependency_job_id":null,"html_url":"https://github.com/Codex-Crusader/le_Market_Intelligence_Platform","commit_stats":null,"previous_names":["codex-crusader/le_market_intelligence_platform"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/Codex-Crusader/le_Market_Intelligence_Platform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Codex-Crusader%2Fle_Market_Intelligence_Platform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Codex-Crusader%2Fle_Market_Intelligence_Platform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Codex-Crusader%2Fle_Market_Intelligence_Platform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Codex-Crusader%2Fle_Market_Intelligence_Platform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Codex-Crusader","download_url":"https://codeload.github.com/Codex-Crusader/le_Market_Intelligence_Platform/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Codex-Crusader%2Fle_Market_Intelligence_Platform/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31456664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"online","status_checked_at":"2026-04-06T02:00:07.287Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cryptocurrency","data-analysis","data-pipeline","financial-analysis","market-analysis","python","sentiment-analysis","stock-market","streamlit","trading-signals"],"created_at":"2026-04-02T23:02:43.433Z","updated_at":"2026-04-06T02:01:10.789Z","avatar_url":"https://github.com/Codex-Crusader.png","language":"Python","funding_links":["https://ko-fi.com/bhargavaramkrishnapur"],"categories":[],"sub_categories":[],"readme":"# Market Intelligence Platform\n\n[![Python](https://img.shields.io/badge/Python-3.9%2B-3776AB?style=flat-square\u0026logo=python\u0026logoColor=white)](https://www.python.org/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-22c55e?style=flat-square)](LICENSE)\n[![Streamlit](https://img.shields.io/badge/Streamlit-1.30%2B-FF4B4B?style=flat-square\u0026logo=streamlit\u0026logoColor=white)](https://streamlit.io/)\n[![pandas](https://img.shields.io/badge/pandas-2.0%2B-150458?style=flat-square\u0026logo=pandas\u0026logoColor=white)](https://pandas.pydata.org/)\n[![Plotly](https://img.shields.io/badge/Plotly-5.18%2B-3F4F75?style=flat-square\u0026logo=plotly\u0026logoColor=white)](https://plotly.com/)\n[![Data: Yahoo Finance](https://img.shields.io/badge/Data-Yahoo%20Finance-6001D2?style=flat-square)](https://finance.yahoo.com/)\n[![News: RSS Feeds](https://img.shields.io/badge/News-12%20RSS%20Feeds-FFA500?style=flat-square)](https://en.wikipedia.org/wiki/RSS)\n[![Assets: 24 Tracked](https://img.shields.io/badge/Assets-24%20Tracked-0ea5e9?style=flat-square)]()\n[![PRs Welcome](https://img.shields.io/badge/PRs-Welcome-brightgreen?style=flat-square)](CONTRIBUTING.md)\n[![Disclaimer](https://img.shields.io/badge/Disclaimer-Not%20Financial%20Advice-red?style=flat-square)](DISCLAIMER.md)\n\n_LINK TO LIVE DEPLOYMENT: [HERE!!!](https://lemarketintelligenceplatform-vsr4ernrtqqduydam2onapp.streamlit.app/)_\n\n![dashboard1](https://github.com/user-attachments/assets/efe48c7c-8acb-40c7-8cea-2ece6d4ab30c)\n\n---\n\nA real-time market analysis dashboard that combines technical price indicators, multi-source news sentiment, and event-driven signal generation into a single composite score for 24 tracked assets across commodities, cryptocurrencies, technology equities, and market indices.\n\nAll data is sourced from free, publicly available feeds. No proprietary APIs, no paid data subscriptions, and no trading execution. The platform is an analytical tool only.\n\n---\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Architecture](#architecture)\n- [Asset Coverage](#asset-coverage)\n- [Quick Start](#quick-start)\n- [Installation](#installation)\n- [Running the Dashboard](#running-the-dashboard)\n- [Running a Full Scan](#running-a-full-scan)\n- [Configuration](#configuration)\n- [Data Storage](#data-storage)\n- [Backtesting](#backtesting)\n- [Project Structure](#project-structure)\n- [Documentation](#documentation)\n- [Contributing](#contributing)\n- [Disclaimer](#disclaimer)\n- [License](#license)\n\n---\n\n## Overview\n\nThe platform processes data from two independent sources on each analysis cycle:\n\n- **Price data** — 30-day OHLCV history fetched from Yahoo Finance via `yfinance`\n- **News data** — up to 300 articles ingested in parallel from 12 curated RSS feeds, deduplicated using Jaccard similarity, and scored using VADER sentiment analysis augmented with a financial lexicon\n\nThese two streams are merged into a composite signal score ranging from -10 (Strong Bearish) to +10 (Strong Bullish) using weighted contributions from six sub-components: trend direction, price momentum, RSI oscillator reading, news sentiment, trend strength, and market/sector context. Each asset class applies its own weighting profile to reflect how different market types respond to different signal types.\n\nHistorical snapshots are persisted to compressed JSON files on disk. A background scan thread processes all 24 assets every 30 minutes without blocking the dashboard UI.\n\n---\n\n## Features\n\n| Feature | Detail |\n|---|---|\n| Signal scoring | Composite score -10 to +10 across 6 weighted components |\n| Asset classes | Commodities, Cryptocurrency, Tech Stocks, Market Indices |\n| News ingestion | 12 public RSS feeds, parallel fetch, Jaccard deduplication |\n| Sentiment engine | VADER with injected financial lexicon, keyword fallback |\n| Event detection | 8 event categories (central bank, geopolitical, earnings, etc.) |\n| Market context | Sector peer comparison and benchmark alignment analysis |\n| Background scan | Full 24-asset scan every 30 minutes via daemon thread |\n| Historical storage | Compressed per-asset JSON snapshots with tiered retention |\n| Backtesting | Hit-rate evaluation by signal strength and label |\n| Retention policy | Full detail 7 days, reduced detail 30 days, deleted after 60 days |\n| Dashboard | Streamlit wide-layout with auto-refresh every 90 seconds |\n| Top movers | Live 24h gainers and losers in sidebar |\n| Market heatmap | Category-level 24h change heatmap |\n| Category overview | Tabular summary of all assets in selected category |\n\n---\n\n## Architecture\n\n```mermaid\nflowchart TD\n    A([Dashboard Start]) --\u003e B[_maybe_trigger_scan]\n    B --\u003e C{Scan due?}\n    C --\u003e|No| D[Serve cached data]\n    C --\u003e|Yes| E[Acquire singleton lock]\n    E --\u003e F{Lock free?}\n    F --\u003e|No| D\n    F --\u003e|Yes| G[Spawn daemon thread]\n    G --\u003e H[run_scan — all 24 assets]\n    D --\u003e I[User selects asset]\n    H --\u003e J[fetch_news_articles]\n    I --\u003e K[cached_history]\n    J --\u003e L[analyse_asset loop]\n    K --\u003e M[compute_price_metrics]\n    M --\u003e N[compute_momentum_metrics]\n    N --\u003e O[correlate_news]\n    O --\u003e P[compute_signal_score]\n    P --\u003e Q[build_explanation]\n    Q --\u003e R[save_snapshot]\n    L --\u003e R\n    R --\u003e S[Dashboard renders]\n    S --\u003e T{Auto-refresh 90s}\n    T --\u003e B\n```\n\n---\n\n## Asset Coverage\n\n[![Commodities](https://img.shields.io/badge/Commodities-8%20Assets-a16207?style=flat-square)]()\n[![Cryptocurrency](https://img.shields.io/badge/Cryptocurrency-5%20Assets-7c3aed?style=flat-square)]()\n[![Tech%20Stocks](https://img.shields.io/badge/Tech%20Stocks-7%20Assets-0284c7?style=flat-square)]()\n[![Market%20Indices](https://img.shields.io/badge/Market%20Indices-4%20Assets-059669?style=flat-square)]()\n\n| Category | Assets |\n|---|---|\n| Commodities | Gold, Silver, Crude Oil, Natural Gas, Copper, Platinum, Wheat, Corn |\n| Cryptocurrency | Bitcoin, Ethereum, Monero, Solana, Litecoin |\n| Tech Stocks | Apple, Microsoft, NVIDIA, Google, Amazon, Meta, Tesla |\n| Market Indices | S\u0026P 500, NASDAQ, Dow Jones, VIX (Fear Index) |\n\n---\n\n## Quick Start\n\n```bash\n# 1. Clone the repository\ngit clone https://github.com/Codex-Crusader/le_Market_Intelligence_Platform.git\ncd le_Market_Intelligence_Platform\n\n# 2. Create and activate a virtual environment\npython -m venv .venv\n# Windows\n.venv\\Scripts\\activate\n# macOS / Linux\nsource .venv/bin/activate\n\n# 3. Install dependencies\npip install -r requirements.txt\n\n# 4. Run the dashboard\nstreamlit run dashboard.py\n```\n\nThe dashboard opens at `http://localhost:8501`. A full-market background scan starts automatically on first load and repeats every 30 minutes.\n\n---\n\n## Quick Start (Docker)\n\n```bash\ndocker build -t market-intel .\ndocker run -p 8501:8501 market-intel\n```\n\nDashboard available at `http://localhost:8501`.\n\n---\n\n## Installation\n\n### Requirements\n\n- Python 3.9 or later\n- Internet connection (Yahoo Finance and RSS feeds)\n\n### Dependencies\n\n| Package | Minimum Version | Purpose |\n|---|---|---|\n| streamlit | 1.30.0 | Dashboard framework |\n| yfinance | 0.2.30 | Price history via Yahoo Finance |\n| pandas | 2.0.0 | DataFrame operations |\n| plotly | 5.18.0 | Interactive charts and heatmap |\n| feedparser | 6.0.0 | RSS feed ingestion |\n| vaderSentiment | 3.3.2 | Sentiment analysis |\n\nInstall all dependencies in one command:\n\n```bash\npip install -r requirements.txt\n```\n\n---\n\n## Running the Dashboard\n\n```bash\nstreamlit run dashboard.py\n```\n\nOn startup the dashboard:\n\n1. Renders the sidebar with category and asset selectors\n2. Checks whether a full-market scan is due\n3. Launches a background daemon thread to scan all 24 assets if the last scan summary is missing or older than 30 minutes\n4. Displays `System initializing — full market scan running in background...` while the scan is active\n5. Fetches price and news data for the currently selected asset and renders the full analysis panel\n\nThe sidebar shows the scan status (running / N minutes ago / pending first run) and a manual `Run full scan now` button.\n\n---\n\n## Running a Full Scan\n\nThe scan pipeline can be executed independently of the dashboard:\n\n```bash\n# Full verbose scan — saves snapshots and summary\npython scan.py\n\n# Suppress per-asset log lines\npython scan.py --quiet\n\n# Validate pipeline without writing any files\npython scan.py --dry-run\n```\n\nOutput is written to:\n- `market_data/\u003cAssetName\u003e_YYYYMMDD.json.gz` — per-asset daily snapshot\n- `market_data/_scan_summary.json.gz` — hierarchical summary of latest scan\n\n---\n\n## Configuration\n\nAll tunable values are in `config.py`. No magic numbers exist anywhere else in the codebase.\n\n| Constant | Default | Description |\n|---|---|---|\n| `LOOKBACK_DAYS` | 30 | Price history window in days |\n| `NEWS_MAX_AGE_HOURS` | 96 | Maximum article age accepted |\n| `NEWS_MAX_ARTICLES` | 300 | Article pool cap before correlation |\n| `PRICE_CACHE_TTL` | 90 s | Dashboard price cache lifetime |\n| `NEWS_CACHE_TTL` | 300 s | Dashboard news cache lifetime |\n| `SCAN_INTERVAL_MINUTES` | 30 | Background scan frequency |\n| `MAX_WORKERS` | 8 | Parallel threads for data fetching |\n| `RSI_PERIOD` | 14 | RSI calculation window |\n| `MOMENTUM_PERIOD` | 10 | Rate-of-change calculation window |\n| `DEDUP_SIMILARITY_THRESHOLD` | 0.65 | Jaccard cutoff for deduplication |\n| `PRICE_CHANGE_THRESHOLD` | 2.0 % | Threshold for significant move alert |\n| `RELEVANCE_HIGH` | 6 | Score threshold for high-relevance news |\n| `RELEVANCE_MEDIUM` | 3 | Score threshold for medium-relevance news |\n| `STORAGE_FULL_DETAIL_DAYS` | 7 | Days to retain full snapshots |\n| `STORAGE_REDUCED_DETAIL_DAYS` | 30 | Days to retain reduced snapshots |\n| `STORAGE_MAX_DAYS` | 60 | Days before snapshot deletion |\n\n### Signal Thresholds\n\n| Label | Score Range |\n|---|---|\n| Strong Bullish | \u003e= 6.0 |\n| Bullish | \u003e= 3.0 |\n| Slightly Bullish | \u003e= 1.0 |\n| Neutral | -1.0 to 1.0 |\n| Slightly Bearish | \u003e= -3.0 |\n| Bearish | \u003e= -6.0 |\n| Strong Bearish | \u003c -6.0 |\n\n### Per-Asset-Class Signal Weights\n\n| Component | Crypto | Tech Stocks | Commodities | Indices |\n|---|---|---|---|---|\n| Trend | 1.2x | 1.2x | 1.3x | 1.5x |\n| Momentum | 1.8x | 1.0x | 1.0x | 1.2x |\n| RSI | 0.8x | 1.0x | 0.8x | 0.5x |\n| Sentiment | 1.2x | 1.6x | 1.2x | 1.0x |\n| Trend Strength | 1.2x | 1.0x | 1.0x | 1.2x |\n| Context | 0.5x | 1.2x | 1.2x | 1.5x |\n\n---\n\n## Data Storage\n\nSnapshots are stored as gzip-compressed JSON files under `market_data/`:\n\n```\nmarket_data/\n  Gold_20260401.json.gz\n  Bitcoin_20260401.json.gz\n  ...\n  _scan_summary.json.gz\n```\n\n### Retention Tiers\n\n| Age | Fields Retained |\n|---|---|\n| 0 – 7 days | Full snapshot including top 5 headlines |\n| 8 – 30 days | Reduced: price, change_1d, signal_score, signal_label, trend, rsi, roc_10d, trend_strength |\n| \u003e 60 days | Deleted automatically after each scan |\n\n---\n\n## Backtesting\n\nThe backtesting module evaluates historical signal accuracy by comparing the signal score on day N with the actual price direction from day N to day N+1.\n\n```python\nfrom backtest import evaluate_signal_accuracy\nresult = evaluate_signal_accuracy(\"Gold\", lookback=20)\nprint(result[\"hit_rate\"])\n```\n\nResults are broken down by:\n- Overall hit rate\n- Accuracy by signal strength (strong / moderate / weak)\n- Accuracy per signal label\n- Current win or loss streak\n\n---\n\n## Project Structure\n\n```\nmarket-intelligence-platform/\n  app.py              Core analysis engine (price, news, signals, explanation)\n  dashboard.py        Streamlit dashboard with background scan management\n  config.py           All configuration constants\n  scan.py             Full-market batch scan pipeline\n  storage.py          Compressed snapshot persistence and retention\n  backtest.py         Historical signal accuracy evaluation\n  requirements.txt    Python dependencies\n  README.md           This file\n  CONTRIBUTING.md     Contribution guidelines\n  DISCLAIMER.md       Legal and financial disclaimer\n  LICENSE             MIT License\n  .gitignore          Git ignore rules\n  Docs/\n    code_flow.md      Detailed execution flow diagrams\n    variable_list.md  Complete variable and constant reference\n  market_data/        Runtime snapshot directory (git-ignored)\n```\n\n---\n\n## Documentation\n\n| Document | Description |\n|---|---|\n| [Docs/code_flow.md](Docs/code_flow.md) | Step-by-step execution flow for every pipeline with Mermaid diagrams |\n| [Docs/variable_list.md](Docs/variable_list.md) | Complete reference of all variables, constants, and return structures |\n| [CONTRIBUTING.md](CONTRIBUTING.md) | How to report issues, propose changes, and submit pull requests |\n| [DISCLAIMER.md](Docs/DISCLAIMER.md) | Financial, legal, and data accuracy disclaimers |\n| [CHANGELOG.md](Docs/CHANGELOG.md) | Financial, legal, and data accuracy disclaimers |\n\n---\n\n## Contributing\n\nContributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request. All contributors must follow the code style and testing requirements described there.\n\n[![Open Issues](https://img.shields.io/github/issues/Codex-Crusader/le_Market_Intelligence_Platform?style=flat-square)](https://github.com/Codex-Crusader/le_Market_Intelligence_Platform/issues)\n[![Pull Requests](https://img.shields.io/github/issues-pr/Codex-Crusader/le_Market_Intelligence_Platform?style=flat-square)](https://github.com/Codex-Crusader/le_Market_Intelligence_Platform/pulls)\n\n---\n\n## Disclaimer\n\nThis software is provided for informational and educational purposes only. It does not constitute financial advice, investment advice, trading advice, or any other form of advice. See [DISCLAIMER.md](Docs/DISCLAIMER.md) for the full disclaimer.\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See [LICENSE](LICENSE) for the full text.\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-22c55e?style=flat-square)](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodex-crusader%2Fle_market_intelligence_platform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodex-crusader%2Fle_market_intelligence_platform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodex-crusader%2Fle_market_intelligence_platform/lists"}