{"id":30654934,"url":"https://github.com/devjosef/shannon-entropy","last_synced_at":"2026-07-03T19:01:34.820Z","repository":{"id":310439478,"uuid":"1039834651","full_name":"Devjosef/Shannon-Entropy","owner":"Devjosef","description":"Quantify uncertainty or randomness in a probability distribution.","archived":false,"fork":false,"pushed_at":"2026-02-17T07:23:00.000Z","size":2656,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-17T12:48:09.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Devjosef.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-18T03:55:27.000Z","updated_at":"2026-02-17T07:23:03.000Z","dependencies_parsed_at":"2025-08-18T06:24:52.041Z","dependency_job_id":null,"html_url":"https://github.com/Devjosef/Shannon-Entropy","commit_stats":null,"previous_names":["devjosef/shannon-entropy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Devjosef/Shannon-Entropy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Devjosef%2FShannon-Entropy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Devjosef%2FShannon-Entropy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Devjosef%2FShannon-Entropy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Devjosef%2FShannon-Entropy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Devjosef","download_url":"https://codeload.github.com/Devjosef/Shannon-Entropy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Devjosef%2FShannon-Entropy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35097760,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-31T09:10:27.347Z","updated_at":"2026-07-03T19:01:34.812Z","avatar_url":"https://github.com/Devjosef.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shannon Entropy Market Analysis\n\n**Live Demo** (60 windows, r=-0.193):\n[![Kaggle Dataset](https://img.shields.io/badge/Kaggle-Dataset-blueviolet)](https://www.kaggle.com/datasets/josefmath/shannon-entropy-market-simulation-60-windows)\n[![Kaggle Notebook](https://img.shields.io/badge/Kaggle-Notebook-orange)](https://www.kaggle.com/code/josefmath/notebook-for-shannon-dataset)\n\n## Purpose \u0026 Goal\n\n**Purpose**: Apply Shannon entropy from information theory to analyze trader behavior patterns and their correlation with market volatility.\n\n**Goal**: Develop a mathematical framework for predicting market volatility through behavioral complexity analysis.\n\n# Theory \u0026 Approach\n\n### Shannon Entropy in market analysis\nShannon entropy quantifies how unpredictable trader actions are.\n\n**Low Entropy (0–0.5 bits)**: Predictable behavior (mass buying or selling)  \n**Medium Entropy (0.5–1.2 bits)**: Mixed behavior patterns  \n**High Entropy (1.2+ bits)**: Unpredictable, chaotic behavior\n\n### Core Hypothesis\n**Thesis**: Trader behavior entropy correlates with market volatility and may provide early warning signals of market stress.\n\n## Methodology\n\n### Data Collection\n\n## Phase 1: Synthetic trader actions test\n*Controlled experiment* Validating the entropy volatility relationship\n\n| Time_Window | Trader_Actions  | Shannon_Entropy | Market_Volatility | Interpretation             |\n| ----------- | --------------- | --------------- | ----------------- | -------------------------- |\n| 1           | [0,0,1,2,2,0,1] | 1.251           | 2.5               | Mixed, Moderate Volatility |\n| 2           | [0,0,0,0,0,0,0] | 0.000           | 0.2               | Predictable, Stable Market |\n| 3           | [0,1,2,0,1,2,0] | 1.585           | 3.0               | Unpredictable, High Vol    |\n\n## Phase 2: Live Market validation\n*SPY portfolio proxy (312 quotes, Feb 16, 2026 15:30-21:09):\n\n| Index | Timestamp | Price (Avg) | Volatility | Entropy | Condition            | Assets            |\n| ----- | --------- | ----------- | ---------- | ------- | -------------------- | ----------------- |\n| 0     | 15:30:58  | 681.27      | 0.00       | 0.722   | Stable (Predictable) | SPY/QQQ/AAPL/TSLA |\n| 50    | 15:35:31  | 681.27      | 173.57     | 1.468   | Chaotic High Vol     | SPY/QQQ/AAPL/TSLA |\n| 100   | 17:34:00  | 417.07      | 178.54     | 1.441   | Chaotic High Vol     | SPY/QQQ/AAPL/TSLA |\n| 150   | 18:31:45  | 417.07      | 147.99     | 1.467   | Moderate Vol         | SPY/QQQ/AAPL/TSLA |\n| 200   | 21:00:03  | 681.27      | 147.95     | 1.508   | Moderate Vol         | SPY/QQQ/AAPL/TSLA |\n| 250   | 21:04:36  | 681.27      | 173.57     | 1.490   | Chaotic High Vol     | SPY/QQQ/AAPL/TSLA |\n| 300   | 21:09:24  | 600.64      | 156.10     | 1.479   | Moderate Vol         | SPY/QQQ/AAPL/TSLA |       |\n\n\n**Trader Actions**: 0 (hold), 1 (buy), 2 (sell)  \n**Time Window**: Sequential trading periods  \n**Entropy Calculation**: `H = -Σ(p_i * log2(p_i))`\n\n### Testing Framework\n**Unit Tests** Validate entropy calculations against known distrubutions.\n\n**Robustness Tests** Edge cases such as empty data, all identical actions, and random or algorithmic trading patterns.\n\n**Market Simulation**: Realistic scenarios (Bull/Bear markets, crashes, recovery)\n\n**Visual Validation**: Comphrensive plotting and correlation analysis\n\n### Implementation\n```cpp\ndouble shannon_entropy(std::vector\u003cint\u003e actions) {\n    std::map\u003cint, int\u003e counts;\n    int total = actions.size();\n    for (int action : actions) {\n        counts[action]++;\n    }\n    double entropy = 0.0;\n    for (auto\u0026 pair : counts) {\n        double p = (double)pair.second / total;\n        if (p \u003e 0) entropy -= p * std::log2(p);\n    }\n    return entropy;\n}\n```\n\n## Testing \u0026 Validation\n\n### Results\n\n**Unit Tests**: 100% pass rate (exact entropy calculations) *automate.test.cpp*\n\n**Robustness Tests**: 15/15 edge cases handled gracefully *robustness.test.cpp*\n \n**Market Validation**: 60 windows across 5 market scenarios *market_validation.test.cpp*\n\n**Table Validation**: Phase 1. tables  *realistic.data.test.cpp*\n\n**Visual Inspection**: Comprehensive correlation analysis *visual_inspection.test.cpp*\n\n### Key Test Scenarios\n**Bull Market**: High entropy (1.497 bits) + Moderate volatility (2.945)\n\n**Bear Market**: Mixed entropy (0.944 bits) + High volatility (3.216)\n\n**Market Crash**: Low entropy (0.599 bits) + Very high volatility (4.999)\n\n**Sideways Market**: Mixed entropy (0.925 bits) + Moderate volatility (2.788)\n\n**Recovery**: Mixed entropy (0.868 bits) + High volatility (3.902)\n\n##  Validating the Thesis\n\n### Primary Finding\n**Live Data Correlation**: r = **-0.193** (312 quotes, `visualize_entropy.py`)\n\n**Simulation Correlation**: r = **-0.601** (`market_validation.test.cpp`)\n\n**Unexpected vs Hypothesis**: Initially the thesis predicted *positive* entropy-vol correlation. Data shows **negative** -high entropy during volatile periods, low entropy during panic crashes.\n\n**Phase 2 Evidence** (table samples):  \n`Idx 50: entropy=1.468 → vol=173.57 (High)`  \n`Idx 100: entropy=1.441 → vol=178.54 (High)`  \n`Idx 0: entropy=0.722 → vol=0.000 (Stable)`\n\n### Market Behavior Patterns Observed:\n\n### Market Behavior Patterns Observed\n\n1. **Panic/Crash Scenarios**  \n   **Low Entropy + Extreme Volatility**  \n   - Mass coordinated selling → predictable (0.599 bits)  \n   - Results in 4.999+ vol spikes  \n   *Pattern*: **Predictable panic leads to market crash**\n\n2. **Normal/Chaotic Trading**  \n   **High Entropy + High Volatility**  \n   - Diverse actions (1.44-1.50 bits, Phase 2)  \n   - 173-178 vol range  \n   *Pattern*: **Chaotic trading  leads to volatile markets**\n\n3. **Stable Periods**  \n   **Low-Medium Entropy + Low Vol**  \n   - Predictable/mixed (0.72 bits, Idx 0)  \n   - 0.000 vol  \n   *Pattern*: **Orderly trading leads to stable prices**\n\n\n## Conclusions\n\n### Thesis Status: **Surprising \u0026 Valid**\n\n**Live Data**: r = **-0.193** (Phase 2, 312 quotes)  \n**Simulation**: r = **-0.601** (60 windows)  \n\n**What We Learned**: Predicted that: \"entropy rises with volatility\"! which was wrong.\n- **Negative correlation confirmed**: High entropy = volatile markets\n- **Panic crashes show low entropy** Predictable mass selling = 0.599 bits\n- **Entropy excels at regime detection** Stable / Chaotic / Panic\n- **Needs microstructure data** In other words it needs access to high frequency feeds, for real prediction power\n\n**Bottom Line**: Entropy seems to quantify behavior brilliantly. **With access to high frequency feeds it can potentially be used for crash detection, and not for volume forecasting.**\n\n\n### Key Observations\n1. **Lower entropy during crashes** (0.599 bits) to extreme volatility (4.999+)\n2. **Higher entropy during volatile periods** (1.44-1.50 bits) ca: 173-178 vol range  \n3. **Negative correlation is stable**: r=-0.193 (Phase 2) / r=-0.601 (simulation)\n4. **Clear behavioral regimes** But requrie high-frequency data for prediction\n\n### Research Insights\n- **Entropy Analysis**: Quantifies trader action distribution across market regimes [1]\n- **Pattern Recognition**: Captures distinct behavioral signatures (crashes: 0.599 bits vs. normal: 1.44+ bits) [2]\n- **Volatility Prediction**: Negative correlation (r=-0.193) but needs high-frequency data + ML models + methods such as: \n  - Semi-supervised learning that captures the noise/signal mix\n  - Pruning via regression \u0026 or decision trees  \n  - Classical Reinforcement Learning identifies regimes [3] [4]\n- **Risk Management**: Can provide behavioral context as a supporting signal, and not  as a standalone predictor [5]\n\n\n\n## Usage\n\n### Quick Start\n```bash\n# Setup environment\n./setup.sh\n\n# Compile \u0026 run main entropy analyzer\ng++ -std=c++17 -o entropy entropy.cpp \u0026\u0026 ./entropy\n\n# Run robustness tests\ng++ -std=c++17 -o test_suite tests/robustness.test.cpp \u0026\u0026 ./test_suite\n\n# Run visual inspection tests\ng++ -std=c++17 -o visual_test tests/visual_inspection.test.cpp \u0026\u0026 ./visual_test\n\n# Generate visualizations\npython visualize_entropy.py\n```\n\n## Technical Specifications\n- **Language**: C++17 core + Python visualization\n- **Entropy Range**: 0.0-1.585 bits (theoretical max for 3 trader actions: BUY/HOLD/SELL)\n- **Test Coverage**: Edge cases + 60 market scenarios (Phase 2 data)\n- **Correlation**: Pearson r=-0.193 (entropy vs. volatility)\n- **Visualization**: **3-panel charts** scatter correlation, entropy time-series, volatility time-series + summary\n\n\n\n ## References\n- [1]: Shannon (1948) - https://ieeexplore.ieee.org/document/6773024\n- - [2]: Shefrin, H. (2002). *Beyond Greed and Fear: Understanding Behavioral Finance and the Psychology of Investing*\n- [3]: Market Microstructure and Entropy Analysis - [https://pmc.ncbi.nlm.nih.gov/articles/PMC9073522/]\n[1]: https://www.linkedin.com/pulse/volume-volatillity-imesh-fernando-0lhyc  \n[2]: https://pmc.ncbi.nlm.nih.gov/articles/PMC9318915/  \n[3]: https://pmc.ncbi.nlm.nih.gov/articles/PMC12748890/  \n[4]: http://upubscience.com/upload/20250314101323.pdf  \n[5]: https://pmc.ncbi.nlm.nih.gov/articles/PMC11675851/\n---\n\n## Final Insight\nEntropy quantifies behavioral patterns effectively. The negative correlation (r=-0.193) challenges traditional hypotheses but reveals **crash predictability** (low entropy leads to high volatility). \n\n*Due to algorithmic trading (m/s: e.g. milisecond executions), requires co-located high-frequency model near exchange to confirm new findings.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevjosef%2Fshannon-entropy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevjosef%2Fshannon-entropy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevjosef%2Fshannon-entropy/lists"}