{"id":50324485,"url":"https://github.com/mindees/crypto-ai","last_synced_at":"2026-05-29T05:02:47.570Z","repository":{"id":360891223,"uuid":"1252160976","full_name":"mindees/crypto-ai","owner":"mindees","description":"Reproducible, decision-support BTC/ETH multi-task ML pipeline — free data, causal features, honest baselines, purged walk-forward backtests. Not financial advice.","archived":false,"fork":false,"pushed_at":"2026-05-28T09:19:52.000Z","size":1478,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T10:26:30.875Z","etag":null,"topics":["algorithmic-trading","backtesting","bitcoin","crypto","cryptocurrency","data-pipeline","deep-learning","ethereum","feature-engineering","financial-machine-learning","fintech","keras","machine-learning","mlops","python","quantitative-finance","tensorflow","time-series","trading-bot","transformer"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mindees.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-28T08:46:35.000Z","updated_at":"2026-05-28T09:19:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mindees/crypto-ai","commit_stats":null,"previous_names":["mindees/crypto-ai"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mindees/crypto-ai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindees%2Fcrypto-ai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindees%2Fcrypto-ai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindees%2Fcrypto-ai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindees%2Fcrypto-ai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mindees","download_url":"https://codeload.github.com/mindees/crypto-ai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindees%2Fcrypto-ai/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33637485,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithmic-trading","backtesting","bitcoin","crypto","cryptocurrency","data-pipeline","deep-learning","ethereum","feature-engineering","financial-machine-learning","fintech","keras","machine-learning","mlops","python","quantitative-finance","tensorflow","time-series","trading-bot","transformer"],"created_at":"2026-05-29T05:02:46.912Z","updated_at":"2026-05-29T05:02:47.561Z","avatar_url":"https://github.com/mindees.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🪙 mindees · crypto-ai\n\n### BTC/ETH multi-task, multi-timeframe market-intelligence pipeline\n\n*Free data in. Causal features. Honest baselines. A probabilistic decision-support model out.*\n\n[![CI](https://github.com/mindees/crypto-ai/actions/workflows/smoke_tests.yml/badge.svg)](https://github.com/mindees/crypto-ai/actions/workflows/smoke_tests.yml)\n[![python](https://img.shields.io/badge/python-3.11%20%7C%203.13-blue.svg)](https://www.python.org/)\n[![tensorflow](https://img.shields.io/badge/tensorflow-2.21-orange.svg)](https://www.tensorflow.org/)\n[![tests](https://img.shields.io/badge/tests-88%20passing-brightgreen.svg)](#testing)\n[![license](https://img.shields.io/badge/license-MIT-lightgrey.svg)](#license)\n[![status](https://img.shields.io/badge/build-8%2F8%20phases%20complete-success.svg)](#build-phases)\n[![stars](https://img.shields.io/github/stars/mindees/crypto-ai?style=social)](https://github.com/mindees/crypto-ai/stargazers)\n\n**Repository:** \u003chttps://github.com/mindees/crypto-ai\u003e\n\n\u003c/div\u003e\n\n\u003e [!WARNING]\n\u003e **Decision-support only. Not financial advice.**\n\u003e Markets are noisy and adversarial. **No model reliably predicts BTC/ETH prices all the time.**\n\u003e A model that fails to beat honest baselines after fees and slippage **must not be used for trading.**\n\u003e Backtests are not promises. You are responsible for your own capital.\n\n---\n\n## Table of contents\n\n- [What this is](#what-this-is)\n- [What it does — and does not — do](#what-it-does--and-does-not--do)\n- [Why \"deepest verified free history\"](#why-deepest-verified-free-history)\n- [Pipeline at a glance](#pipeline-at-a-glance)\n- [Quickstart](#quickstart)\n- [Data sources](#data-sources)\n- [Features \u0026 the rule-based scorecard](#features--the-rule-based-scorecard)\n- [Labels](#labels)\n- [Feature selection](#feature-selection)\n- [Class imbalance](#class-imbalance)\n- [Model architecture](#model-architecture)\n- [PlantGuard-style training](#plantguard-style-training)\n- [Evaluation \u0026 honest baselines](#evaluation--honest-baselines)\n- [Backtesting](#backtesting)\n- [Prediction output \u0026 how to read it](#prediction-output--how-to-read-it)\n- [Serving (FastAPI) \u0026 alerts](#serving-fastapi--alerts)\n- [Model registry, promotion \u0026 rollback](#model-registry-promotion--rollback)\n- [Drift, retraining \u0026 shadow A/B](#drift-retraining--shadow-ab)\n- [Compute: Kaggle, Colab \u0026 GitHub Actions](#compute-kaggle-colab--github-actions)\n- [Command reference](#command-reference)\n- [Repository layout](#repository-layout)\n- [Configuration \u0026 API keys](#configuration--api-keys)\n- [Testing](#testing)\n- [Build phases](#build-phases)\n- [Limitations \u0026 known gaps](#limitations--known-gaps)\n- [License \u0026 disclaimer](#license--disclaimer)\n\n---\n\n## What this is\n\n`crypto-ai` ingests the deepest **free, verifiable** historical data for **BTCUSDT** and\n**ETHUSDT** (spot + USDT-margined perpetual futures), engineers strictly causal features,\nand trains a **multi-task TensorFlow model** with four heads:\n\n| Head | Type | Classes |\n|---|---|---|\n| **direction** | 3-class softmax | `down` · `sideways` · `up` |\n| **regime** | 6-class softmax | `trending_up` · `trending_down` · `ranging_low_vol` · `ranging_high_vol` · `breakout` · `capitulation` |\n| **cycle** | 4-class softmax | `accumulation` · `bull` · `distribution` · `bear` |\n| **trade_quality** | binary sigmoid | would the trade reach ≥2R before stop, after fees? |\n\nEverything is validated with **purged + embargoed walk-forward** splits (never random shuffle),\nbacktested with realistic fees + slippage, and compared against **honest baselines**. If the\nmodel can't beat buy-and-hold / EMA / RSI-MACD / majority-class after costs, the reports say so\nplainly.\n\n## What it does — and does not — do\n\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003e✅ Does\u003c/th\u003e\u003cth\u003e🚫 Does not\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd valign=\"top\"\u003e\n\n- Free, public data only (Binance, blockchain.info, Alternative.me, CoinGecko, FRED/yfinance)\n- Idempotent, checksum-verified ingestion\n- Causal feature engineering (no lookahead — enforced by tests)\n- Multi-timeframe transformer with cross-timeframe attention\n- Honest evaluation + event-driven backtest vs baselines\n- Model registry with gated promotion, rollback, shadow A/B\n- PSI drift detection + visual dashboard + retrain recommendations\n- FastAPI serving + disabled-by-default alerts\n\n\u003c/td\u003e\u003ctd valign=\"top\"\u003e\n\n- Claim coverage \"from asset genesis\" (it can't — see below)\n- Require any paid data source (paid adapters are disabled stubs)\n- Auto-promote or auto-trade — promotion needs an explicit command\n- Send alerts by default — every channel ships disabled\n- Emit hard buy/sell calls — signals are hedged biases\n- Pretend a weak model is strong — failure to beat baselines is reported\n\n\u003c/td\u003e\u003c/tr\u003e\n\u003c/table\u003e\n\n## Why \"deepest verified free history\"\n\nBinance does **not** have BTC from 2009 or ETH from Ethereum genesis. So this project never\nclaims \"zero to today.\" Instead it **discovers and reports the first verified candle per\nsource**, and uses that wording everywhere:\n\n\u003e *\"Deepest free verified history available from each configured source.\"*\n\nObserved free coverage (auto-detected, written to `metadata/watermarks.json`):\n\n| Market | Symbol | First verified candle |\n|---|---|---|\n| spot | BTCUSDT / ETHUSDT | **2017-08-17** |\n| futures (USDT-M) | BTCUSDT / ETHUSDT | **2020-01-01** |\n| on-chain (BTC, blockchain.info) | — | **2009** (daily) |\n| Fear \u0026 Greed | — | **2018-02-01** |\n\n\u003e Binance launched USDT-M futures in Sep 2019, but the public bulk archives only start\n\u003e 2020-01 — so the pipeline reports 4 leading 404s and starts there. That honesty is the point.\n\n## Pipeline at a glance\n\n```\n                 ┌──────────────────────────────────────────────────────────────┐\n  FREE SOURCES   │  Binance bulk · ccxt · derivatives REST · blockchain.info ·   │\n                 │  Alternative.me F\u0026G · CoinGecko · yfinance/FRED               │\n                 └───────────────┬──────────────────────────────────────────────┘\n                                 ▼\n   ingest/  ──►  data/processed/*.parquet  (idempotent, checksum-verified, UTC)\n                                 ▼\n  features/ ──►  causal indicators · structure · patterns · flow · onchain ·\n                 sentiment · macro · scorecard            (no lookahead)\n                                 ▼\n   labels/  ──►  triple-barrier direction · regime · cycle · trade-quality\n                                 ▼\n datasets/  ──►  purged+embargoed walk-forward windows · train-only scaler\n                                 ▼\n  models/   ──►  MTF Transformer (4 heads)  ──►  train · PlantGuard 2-phase\n                                 ▼\n            evaluate (vs baselines) · backtest (fees+slippage) · thresholds · predict\n                                 ▼\n  registry · promote · rollback · drift · drift_viz · shadow A/B · retrain_check\n                                 ▼\n  serve/    ──►  FastAPI · scheduler · drift dashboard · disabled-by-default alerts\n```\n\n---\n\n## Quickstart\n\n```bash\n# 1. Clone\ngit clone https://github.com/mindees/crypto-ai.git\ncd crypto-ai\n\n# 2. Create a venv (Python 3.11 recommended; 3.13 also works with TF 2.21)\npython -m venv .venv\n# Windows (PowerShell):\n.\\.venv\\Scripts\\Activate.ps1\n# macOS / Linux:\n# source .venv/bin/activate\n\n# 3. Install pinned dependencies\npip install -r requirements.txt\n\n# 4. Sanity checks\npython -c \"import tensorflow as tf; print(tf.__version__, tf.config.list_physical_devices('GPU'))\"\npython -m src.utils.hardware\npytest tests/ -q          # 88 passing\n```\n\nEnd-to-end smoke run (CPU, tiny sample windows — proves the whole pipeline):\n\n```bash\n# Ingest 1d OHLCV for all four combos (deepest verified history)\npython -m src.ingest.binance_bulk --symbols BTCUSDT ETHUSDT --market-types spot futures_um --timeframes 1d\n\n# Free context adapters\npython -m src.ingest.sentiment\npython -m src.ingest.coingecko\npython -m src.ingest.onchain\npython -m src.ingest.derivatives --symbols BTCUSDT ETHUSDT\n\n# Features → labels → dataset → train (sample) → evaluate → backtest → predict\npython -m src.features.build_matrix --symbols BTCUSDT ETHUSDT --timeframes 1h 4h --sample true\npython -m src.labels.labeling       --symbols BTCUSDT ETHUSDT --timeframes 1h 4h --sample true\npython -m src.datasets.build_dataset --symbols BTCUSDT ETHUSDT --timeframes 1h --sample true\npython -m src.models.train_like_plantguard --timeframes 1h --sample true --phase1-epochs 1 --phase2-epochs 1\npython -m src.models.evaluate --latest --sample true\npython -m src.backtest.engine --latest --sample true\npython -m src.models.predict  --latest --symbols BTCUSDT ETHUSDT --timeframes 1h 4h\n```\n\n\u003e Optional extras (TA-Lib, polars, vectorbt, SHAP) live in `requirements-optional.txt`.\n\u003e The codebase works **without** TA-Lib — indicators are pandas-native with a TA-Lib hook.\n\n---\n\n## Data sources\n\nAll free by default. Every adapter returns a dated DataFrame **or a clear \"unavailable\" log** —\nnothing silently fabricates data.\n\n| Source | Module | Data | Coverage / limitation |\n|---|---|---|---|\n| **Binance bulk** (`data.binance.vision`) | `ingest/binance_bulk.py` | spot + USDT-M OHLCV, all intervals | Primary OHLCV. Monthly + daily archives, SHA-256 verified. ms→µs timestamp switch (2025-01) handled; spot/futures column aliases normalized. |\n| ccxt | `ingest/ccxt_incremental.py` | recent OHLCV delta | Fallback only. |\n| Binance derivatives REST | `ingest/derivatives.py` | funding, OI, long/short, taker vol | Funding paginated to launch; **OI / ratios / taker are recent-only (~30 days)** — marked as such. |\n| blockchain.info | `ingest/onchain.py` | BTC hash-rate, difficulty, miner rev, txs, addresses, supply, fees | Daily, back to 2009. |\n| Etherscan | `ingest/onchain.py` | ETH supply / gas snapshots | **Disabled unless `ETHERSCAN_API_KEY` set**; snapshots only. |\n| Alternative.me | `ingest/sentiment.py` | Fear \u0026 Greed index | Daily since 2018-02; mostly BTC-driven. |\n| CoinGecko (free) | `ingest/coingecko.py` | BTC/ETH dominance, global mcap | Snapshot endpoint — accrues history per run; rate-limited. |\n| yfinance / FRED | `ingest/macro.py` | S\u0026P500, Nasdaq, DXY, VIX, FedFunds, CPI | yfinance is **frequently rate-limited (429)** from data-center IPs; FRED needs a free key + opt-in. |\n| Glassnode / CryptoQuant / CoinGlass / Coinalyze / Amberdata | `ingest/paid_stubs.py` | — | **Disabled stubs.** Bring your own key + implement to enable. |\n\nAn `onchain_coverage_score` per asset is written to `metadata/onchain_coverage.json` so the\nmodeling layer knows whether on-chain features are strong or mostly missing.\n\n---\n\n## Features \u0026 the rule-based scorecard\n\nAll features are **causal** — row `t` uses only data at or before `t`. This is enforced by\n`tests/test_no_lookahead.py`, which mutates future rows and asserts past features don't change.\n\n- **Indicators** (`features/indicators.py`): EMA 9/21/50/120/200 + stack score, EMA120 cycle signal, golden/death cross, RSI(14) + slope/zscore, MACD, Bollinger %B/bandwidth, ATR, OBV, VWAP, realized volatility, distance-from-ATH/52w.\n- **Structure** (`features/structure.py`): causal swing highs/lows, HH/HL/LH/LL, market-structure score, range/breakout, liquidity-sweep proxy, FVG / order-block / CHoCH **proxies** (clearly labelled).\n- **Patterns** (`features/patterns.py`): doji, hammer, shooting star, engulfing, inside/outside bar, pin bar.\n- **Flow** (`features/flow.py`): funding rate + z-score + extremes, OI change, price/OI quadrants, taker delta, **CVD proxy** (named a proxy — not true full-market CVD), basis, funding/OI governor.\n- **Sentiment / on-chain / macro**: causally joined via `merge_asof(direction=\"backward\")` so a bar never sees a value published after its close.\n\nThe **scorecard** (`features/scorecard.py`) is a transparent, rule-derived assessment **separate\nfrom the ML model**. Missing inputs are reported as the literal string `\"unavailable\"` — never\nguessed, never zero. It powers the `scorecard` field in the prediction JSON and alerts.\n\n---\n\n## Labels\n\n`labels/labeling.py` produces four targets (targets may look forward; **features may not**):\n\n- **Direction** — triple-barrier: upper = `close + k·ATR`, lower = `close − k·ATR`, vertical = `N` bars. Uses intrabar high/low path. Same-candle double-touch → `ambiguous` (excluded from training). Per-timeframe `k` and `N` in `configs/config.yaml`.\n- **Regime** — rule-based from EMA slope, EMA-stack, ATR percentile, realized-vol percentile, structure.\n- **Cycle** — anchored on BTC halving dates (`reference/halvings.csv`) + drawdown-from-ATH + 200-week-MA position. ETH inherits the BTC anchor plus its own drawdown.\n- **Trade quality** — binary: did the directional target reach **≥2R before a −1R stop**, after fees + slippage?\n\nValidation **purges overlapping label horizons** and applies an embargo so triple-barrier\nwindows don't leak across the train/val/test boundary.\n\n---\n\n## Feature selection\n\n`features/selection.py` runs **inside each training fold** (never on test):\n\n1. Drop features below `min_non_null_ratio` (0.85).\n2. Drop near-zero-variance features.\n3. Drop one of each highly-correlated pair (`max_pairwise_corr` = 0.95).\n4. Rank by **mutual information** (training window only).\n5. Rank by **permutation importance** of a light GBM (validation window only).\n6. Keep `always_keep` features + top-K (`final_top_k` = 120).\n\n`tests/test_feature_selection_no_leakage.py` proves selection is unchanged when the test split\nis mutated — i.e., the test set is never read during selection.\n\n---\n\n## Class imbalance\n\n- **Class weights** by default: `weight = 1/√frequency`, normalized to mean 1.0 (no SMOTE — synthetic candle windows are unrealistic).\n- Optional **focal loss** flag.\n- **Decision-threshold tuning** (`models/thresholds.py`) on validation only — maximizes macro F1 subject to per-trade-class precision floors, and allows a `no_trade` zone when confidence is low.\n- Stratified reporting by regime.\n\n---\n\n## Model architecture\n\nDefault: **MTF Transformer with cross-timeframe attention** (`models/multitask_model.py`).\n\n```\nfast_seq ─┐\nmain_seq ─┼─► per-tf encoder (LayerNorm → Dense → PosEnc → 3× Transformer block → attention-pool)\nslow_seq ─┘                    │\n                               ▼\n                 cross-timeframe MultiHeadAttention  ◄── asset \u0026 timeframe embeddings\n                               │\n            context branch ───►├─► shared trunk (Dense 256 → 128, BN + dropout)\n                               ▼\n        ┌──────────┬───────────┬──────────────┬────────────────┐\n   direction(3)  regime(6)   cycle(4)   trade_quality(1, sigmoid)\n```\n\n- Optimizer AdamW (lr 3e-4, weight-decay 1e-4, clipnorm 1.0), cosine decay + warmup.\n- Mixed precision auto-enabled on GPU; **`tf.distribute.MirroredStrategy()` auto-selected when ≥2 GPUs** (Kaggle dual-T4).\n- A config switch `use_multi_timeframe_fusion: false` falls back to a compact single-timeframe transformer for CPU smoke tests.\n- Custom layers are serialization-safe (reload + predict verified).\n\n---\n\n## PlantGuard-style training\n\n`models/train_like_plantguard.py` adapts the polished Plant Guard image-classification workflow\nto **causal time series** (no image augmentation, no MobileNet). Two phases:\n\n1. **Phase 1 — head warmup.** If an SSL-pretrained encoder exists, freeze it and train fusion/trunk/heads at a higher LR. If not, run a lower-LR warmup of the full model (spec-compliant fallback).\n2. **Phase 2 — fine-tune.** Unfreeze the last N transformer blocks at a much lower LR.\n\nArtifacts per run (`artifacts/runs/\u003cid\u003e_plantguard/`): `model.keras`, `phase{1,2}_best.keras`,\n`training_curves.png`, `confusion_{direction,regime,cycle}.png`, `classification_report.json`,\n`prediction_demo.json`, `class_indices.json`, `dataset_spec.json`, + a reload + sanity-predict check.\n\n```bash\npython -m src.models.train_like_plantguard \\\n  --symbols BTCUSDT ETHUSDT --timeframes 1h \\\n  --phase1-epochs 10 --phase2-epochs 25\n```\n\n| Plant Guard concept | Time-series equivalent here |\n|---|---|\n| image class distribution | label distribution (direction/regime/cycle/quality) |\n| sample images | sample market windows |\n| MobileNet pretrained base | optional self-supervised time-series encoder |\n| frozen base → fine-tune | freeze encoder → unfreeze last blocks |\n| confusion matrix | direction / regime / cycle confusion matrices |\n| single-image prediction demo | latest-market prediction demo |\n| Flask inference | FastAPI prediction endpoint |\n\n---\n\n## Evaluation \u0026 honest baselines\n\n`models/evaluate.py` reports per-head metrics on the held-out split and compares the direction\nhead against **majority-class** and **random** baselines, tunes thresholds on validation, and\nwrites `reports/eval_\u003cid\u003e.md` + `.json`. The report **states plainly whether the model beats\nbaselines** — and on a 1–2 epoch CPU smoke run it correctly says it does **not**.\n\n## Backtesting\n\n`backtest/` is a real **event-driven** simulator (not just classification metrics):\n\n- Risk-based sizing, ATR stops, **TP1/TP2/TP3 partials** (33/33/34%), SL→breakeven after TP1, SL→TP1 after TP2, vertical-barrier exit.\n- Fees (bps/side) + **ATR/volume-aware slippage**, intrabar high/low logic, max-daily-loss cap.\n- Metrics: total return, CAGR, max drawdown, profit factor, expectancy, Sharpe, Sortino, win rate, avg/median R, fee + slippage drag, long/short split, worst-10 trades.\n- Compares **model vs buy-and-hold / EMA-trend / RSI-MACD / random / no-trade**.\n- Default leverage **1×** for evaluation honesty.\n\n```bash\npython -m src.backtest.engine --latest --sample true\n# → reports/backtest_\u003cid\u003e.md · backtest_\u003cid\u003e.json · trades_\u003cid\u003e.csv\n```\n\n\u003e **Futures leverage / margin / liquidation simulation** is specified as optional and ships\n\u003e **disabled** (`backtest.futures_margin.enabled: false`). The default evaluation is spot/1×.\n\n---\n\n## Prediction output \u0026 how to read it\n\n`models/predict.py` loads the production model, rebuilds the latest feature window with the\n**saved scaler/imputer**, applies tuned thresholds + the funding/OI governor, attaches the\nscorecard, and writes `reports/latest_predictions.json`:\n\n```jsonc\n{\n  \"asset\": \"BTCUSDT\", \"timeframe\": \"1h\", \"model_id\": \"...\",\n  \"model_outputs\": {\n    \"direction\": { \"down\": 0.21, \"sideways\": 0.28, \"up\": 0.51 },\n    \"regime\":    { \"predicted\": \"trending_up\", \"confidence\": 0.63 },\n    \"cycle\":     { \"predicted\": \"bull\", \"confidence\": 0.58 },\n    \"trade_quality\": { \"probability\": 0.62 }\n  },\n  \"signal\": { \"action\": \"no_trade\", \"reason\": \"confidence below threshold\",\n              \"long_threshold\": 0.58, \"short_threshold\": 0.58, \"quality_threshold\": 0.60 },\n  \"scorecard\": { \"trend_direction\": \"up\", \"rsi_14\": 61.2, \"funding_state\": \"slightly_positive\", \"...\": \"...\" },\n  \"risk_warning\": \"Decision-support only. Not financial advice. Validate manually before trading.\"\n}\n```\n\n**Signal vocabulary is intentionally hedged** — never a hard buy/sell:\n`long_bias` · `short_bias` · `no_trade` · `range_wait` · `high_risk`.\n\n---\n\n## Serving (FastAPI) \u0026 alerts\n\nThe API (`serve/api.py`) is deliberately **TensorFlow-free** — it serves the JSON `predict.py`\nwrites, keeping the web tier light.\n\n```bash\nuvicorn src.serve.api:app --host 0.0.0.0 --port 8000\npython -m src.serve.api --smoke-test     # hits every route via TestClient\n```\n\n| Endpoint | Returns |\n|---|---|\n| `GET /health` | service status + disclaimer |\n| `GET /model/current` | production model pointer |\n| `GET /registry` | full model registry |\n| `GET /predict/latest` | latest predictions for all combos |\n| `GET /predict/{asset}/{timeframe}` | one prediction |\n| `GET /scorecard/{asset}/{timeframe}` | the rule-based scorecard |\n| `GET /drift/latest` | most recent drift dashboard (HTML) |\n| `POST /predict/refresh` | re-read predictions (`?run_predict=true` to regenerate) |\n\n**Alerts** (`serve/alert_templates.py`, `serve/alerts.py`) are **disabled by default** and require\nboth a config flag **and** environment credentials. The canonical payload includes `asset,\ntimeframe, signal, direction_confidence, trade_quality_probability, regime, cycle_phase,\nentry/stop/tp1/tp2/tp3_reference, estimated_rr, risk_per_trade_pct, leverage,\nliquidation_buffer_pct, scorecard{...}, warnings[], cooldown_minutes`. Alerts fire **only** for\n`long_bias`/`short_bias` that clear confidence + trade-quality + cooldown gates and a non-stale\nmodel — never for `no_trade`/`range_wait`/`high_risk`.\n\n```bash\npython -m src.serve.alert_templates --sample true   # preview payload + Telegram/email render\npython -m src.serve.scheduler --refresh-minutes 15   # local predict→alert loop (--once for one tick)\n```\n\n---\n\n## Model registry, promotion \u0026 rollback\n\n`metadata/model_registry.json` tracks every run (`candidate` → `production` → `archived` /\n`rejected` / `rolled_back`). **Artifacts are never overwritten** — only pointers move.\n\n```bash\npython -m src.models.registry --list          # sync + list all runs\npython -m src.models.promote  --latest --dry-run   # show gate decision, change nothing\npython -m src.models.promote  --model-id \u003cid\u003e      # apply (gated)\npython -m src.models.rollback --model-id \u003cid\u003e      # restore a previous production model\n```\n\nPromotion is **gated** (and never silent): beats production on direction macro F1 by ≥ threshold,\npositive expectancy after costs, profit factor ≥ minimum, drawdown not materially worse.\n\n## Drift, retraining \u0026 shadow A/B\n\n- **Drift** (`models/drift.py`): PSI per feature — `\u003c0.10` stable · `0.10–0.25` moderate · `≥0.25` significant.\n- **Retrain check** (`models/retrain_check.py`): new-bars + PSI + performance triggers → `metadata/retrain_status.json` + report. **Recommends only — never auto-trains.**\n- **Drift dashboard** (`models/drift_viz.py`): 6 charts + HTML dashboard under `reports/`.\n- **Shadow A/B** (`models/shadow.py`, `models/ab_compare.py`): run a candidate beside production on identical bars, log both, compare agreement / signal counts / confidence. Promotion after shadow still requires an explicit command.\n\n```bash\npython -m src.models.retrain_check\npython -m src.models.drift_viz   --sample true\npython -m src.models.shadow      --candidate latest --sample true\npython -m src.models.ab_compare  --candidate latest --sample true\n```\n\n---\n\n## Compute: Kaggle, Colab \u0026 GitHub Actions\n\n| Environment | Role | Notes |\n|---|---|---|\n| **Kaggle** | primary training | dual-T4 → `MirroredStrategy` auto-selected. Notebooks: `notebooks/kaggle_train.ipynb`, `notebooks/kaggle_train_plantguard_style.ipynb`. |\n| **Colab** | secondary training | T4 single-GPU; mount Drive for resume across disconnects. Notebooks: `notebooks/colab_train.ipynb`, `notebooks/colab_train_plantguard_style.ipynb`. |\n| **GitHub Actions** | automation only | **Never trains.** Uses `requirements-ci.txt` (no TF). |\n| **Local** | CPU smoke + serving | Windows + TF ≥ 2.11 is CPU-only (use WSL2 for GPU). |\n\nAll four notebooks import the real `src/` modules (no duplicated logic) and support **resume**\nfrom existing checkpoints. They are **self-contained**: each clones this repo, installs only the\nmissing extras, ingests free data, builds features/labels, then trains.\n\n### Running on Kaggle\n\n1. Create a notebook and upload/import `notebooks/kaggle_train.ipynb` (or the PlantGuard one).\n2. In the right-hand **Settings**: **Internet → On** (needed to clone + ingest) and\n   **Accelerator → GPU T4 x2** (auto-detected → `MirroredStrategy`).\n3. **Run All.** Defaults bound ingestion with `START_YEAR=2022`; set `SAMPLE=True` for a ~2-min smoke.\n\n\u003e The notebooks install only `requirements-notebook.txt` (ccxt, fredapi, pandas-ta-classic,\n\u003e yfinance) and **reuse Kaggle's GPU-matched TensorFlow** — installing the fully-pinned\n\u003e `requirements.txt` there would overwrite the host TF and can break GPU detection.\n\n### Running on Colab\n\nOpen `notebooks/colab_train.ipynb`, set **Runtime → T4 GPU**, Run All. Mount Drive (first cell)\nto persist checkpoints across disconnects.\n\n**Workflows** (`.github/workflows/`):\n- `daily_data.yml` — daily delta → validation → retrain check → conditional Kaggle push → commits **only metadata/reports** (large data is never committed).\n- `smoke_tests.yml` — `pytest tests/` on push/PR + a guard that fails if large data is git-tracked.\n- `weekly_retrain_notice.yml` — weekly retrain **recommendation** (no training).\n\nIf Kaggle secrets are absent, the daily job still runs local validation and logs that the push was skipped.\n\n---\n\n## Command reference\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003eClick to expand the full CLI\u003c/b\u003e\u003c/summary\u003e\n\n```bash\n# Ingestion\npython -m src.ingest.binance_bulk --symbols BTCUSDT ETHUSDT --market-types spot futures_um --timeframes 1h 4h 1d\npython -m src.ingest.derivatives  --symbols BTCUSDT ETHUSDT\npython -m src.ingest.sentiment\npython -m src.ingest.coingecko\npython -m src.ingest.onchain\npython -m src.ingest.macro\npython -m src.ingest.daily_update --symbols BTCUSDT ETHUSDT --timeframes 1h 4h 1d\n\n# Features / labels / selection\npython -m src.features.build_matrix --symbols BTCUSDT ETHUSDT --timeframes 1h 4h\npython -m src.labels.labeling       --symbols BTCUSDT ETHUSDT --timeframes 1h 4h\npython -m src.features.selection    --symbol BTCUSDT --timeframe 1h\n\n# Dataset / train\npython -m src.datasets.build_dataset --symbols BTCUSDT ETHUSDT --timeframes 1h\npython -m src.models.train                  --timeframe 1h --epochs 60\npython -m src.models.train_like_plantguard  --timeframes 1h --phase1-epochs 10 --phase2-epochs 25\n\n# Evaluate / backtest / predict\npython -m src.models.evaluate --latest\npython -m src.backtest.engine --latest\npython -m src.models.predict  --latest --symbols BTCUSDT ETHUSDT --timeframes 1h 4h\n\n# Lifecycle\npython -m src.models.registry --list\npython -m src.models.promote  --latest --dry-run\npython -m src.models.rollback --model-id \u003cid\u003e\npython -m src.models.retrain_check\npython -m src.models.drift_viz  --sample true\npython -m src.models.shadow     --candidate latest --sample true\npython -m src.models.ab_compare --candidate latest --sample true\n\n# Serve\nuvicorn src.serve.api:app --host 0.0.0.0 --port 8000\npython -m src.serve.scheduler --refresh-minutes 15\npython -m src.utils.hardware\npython -m src.utils.validation_cli\n```\n\n\u003c/details\u003e\n\n## Repository layout\n\n```text\nconfigs/            main YAML config\ndata/               raw / interim / processed / features / labels (gitignored except samples)\nmetadata/           source registry, watermarks, checksums, model registry, retrain/coverage\nreference/          halvings.csv\nsrc/\n  ingest/           binance_bulk, ccxt, derivatives, onchain, sentiment, macro, coingecko,\n                    paid_stubs, daily_update\n  features/         indicators, structure, patterns, flow, onchain, sentiment, macro,\n                    scorecard, selection, build_matrix\n  labels/           labeling (triple-barrier, regime, cycle, trade-quality)\n  datasets/         build_dataset (windowing, purged walk-forward, train-only scaler)\n  models/           multitask_model, train, train_like_plantguard, evaluate, predict,\n                    thresholds, registry, promote, rollback, drift, drift_viz,\n                    shadow, ab_compare, retrain_check\n  backtest/         engine, broker, metrics, strategies, costs\n  serve/            api, scheduler, drift_dashboard, alert_templates, alerts\n  utils/            io, time, logging, seeds, validation, validation_cli, hardware\nnotebooks/          kaggle/colab train + PlantGuard-style notebooks\n.github/workflows/  daily_data, smoke_tests, weekly_retrain_notice\ntests/              88 tests (no-lookahead, labeling, idempotency, schema, scorecard,\n                    selection-no-leakage, backtest, registry, drift, shadow, serving, alerts)\nartifacts/          training runs + production pointer\nreports/            eval / backtest / drift / shadow / retrain reports\n```\n\n## Configuration \u0026 API keys\n\nEverything is driven by [`configs/config.yaml`](configs/config.yaml). No keys are required for\nthe free defaults. Optional keys go in `.env` (gitignored):\n\n| Variable | Needed for |\n|---|---|\n| `KAGGLE_USERNAME`, `KAGGLE_KEY` | Kaggle dataset push (daily workflow) |\n| `FRED_API_KEY` | FRED macro series (also set `sources.enable_fred: true`) |\n| `ETHERSCAN_API_KEY` | ETH on-chain (also set `sources.enable_etherscan: true`) |\n| `TELEGRAM_BOT_TOKEN`, `TELEGRAM_CHAT_ID` | Telegram alerts (also `enable_telegram: true`) |\n| `DISCORD_WEBHOOK_URL` | Discord alerts |\n| `SMTP_HOST/PORT/USER/PASS/FROM/TO` | email alerts |\n\n## Testing\n\n```bash\npytest tests/ -q        # 88 passing\n```\n\nCoverage highlights: no-lookahead causality, triple-barrier label cases, ingestion idempotency\n(+ ms→µs + futures-alias handling), feature schema, scorecard \"unavailable\" handling,\nfeature-selection no-leakage, backtest mechanics, registry/promote/rollback, PSI + drift viz,\nshadow A/B, FastAPI contract, alert gating.\n\n## Build phases\n\n| Phase | Scope | Status |\n|---|---|:--:|\n| 0 | Scaffold, config, utils, hardware detection | ✅ |\n| 1 | Binance bulk ingestion (idempotent, checksum-verified) | ✅ |\n| 2 | Derivatives, sentiment, CoinGecko, on-chain, macro, paid stubs | ✅ |\n| 3 | Features, labels, feature selection | ✅ |\n| 4 | Dataset builder, MTF model, train + PlantGuard two-phase | ✅ |\n| 5 | Evaluation, backtest, thresholds, prediction | ✅ |\n| 6 | Registry, promotion, rollback, drift viz, shadow A/B | ✅ |\n| 7 | FastAPI serving, scheduler, alerts, drift dashboard | ✅ |\n| 8 | GitHub Actions + Kaggle/Colab notebooks | ✅ |\n\n## Limitations \u0026 known gaps\n\n- **No GPU locally** (Windows + TF ≥ 2.11). Shipped models are CPU smoke runs that **honestly do not beat baselines** — real training is the Kaggle/Colab notebooks' job.\n- **yfinance/Yahoo rate-limits** macro pulls from data-center IPs; FRED is the reliable macro path.\n- **Derivatives ratios/OI/taker are recent-only (~30 days)** via free REST; only funding has deep history.\n- **`src/strategies/` research comparators** (ema120_cycle, wyckoff_proxy, ict_smc_proxy, funding_arbitrage_research, …) are not yet built — the core honest baselines live in `src/backtest/strategies.py`.\n- **Futures margin/liquidation/funding simulation** is specified-but-disabled; default eval is spot/1×.\n- **SSL encoder pretraining** (masked-window modeling) is optional; PlantGuard Phase 1 currently runs as a full-model warmup.\n- SMC/ICT/Wyckoff and CVD features are **proxies**, labelled as such — not institutional-grade.\n\n## License \u0026 disclaimer\n\nReleased under the **MIT License**.\n\n\u003e This software is provided for research and educational purposes. It is **not investment\n\u003e advice** and must not be relied upon for trading decisions. Crypto markets are extremely\n\u003e volatile and adversarial. Past performance and backtest results do not predict future results.\n\u003e The authors and contributors accept no liability for any loss arising from use of this software.\n\n\u003cdiv align=\"center\"\u003e\n\n**[⬆ back to top](#-mindees--crypto-ai)** · \u003chttps://github.com/mindees/crypto-ai\u003e\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindees%2Fcrypto-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmindees%2Fcrypto-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindees%2Fcrypto-ai/lists"}