{"id":51081456,"url":"https://github.com/royxlead/production-drift-detection","last_synced_at":"2026-06-23T18:32:40.783Z","repository":{"id":364821434,"uuid":"1260998950","full_name":"royxlead/production-drift-detection","owner":"royxlead","description":"Production ML monitoring library - KL, PSI, MMD, and ADWIN drift detectors with empirical benchmarks, confidence tracking, and a 6-page FastAPI dashboard.","archived":false,"fork":false,"pushed_at":"2026-06-14T15:50:47.000Z","size":213,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-14T17:23:33.244Z","etag":null,"topics":["data-drift","drift-detection","fastapi","kl-divergence","mlops","mmd","model-monitoring","production-ml","psi","pytorch","scikit-learn","uncertainty-quantification"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/royxlead.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-06T05:39:36.000Z","updated_at":"2026-06-14T15:50:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/royxlead/production-drift-detection","commit_stats":null,"previous_names":["royxlead/production-drift-detection"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/royxlead/production-drift-detection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royxlead%2Fproduction-drift-detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royxlead%2Fproduction-drift-detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royxlead%2Fproduction-drift-detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royxlead%2Fproduction-drift-detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/royxlead","download_url":"https://codeload.github.com/royxlead/production-drift-detection/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/royxlead%2Fproduction-drift-detection/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34702913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-drift","drift-detection","fastapi","kl-divergence","mlops","mmd","model-monitoring","production-ml","psi","pytorch","scikit-learn","uncertainty-quantification"],"created_at":"2026-06-23T18:32:39.695Z","updated_at":"2026-06-23T18:32:40.776Z","avatar_url":"https://github.com/royxlead.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Production Drift Detection\n### Real-Time Data Drift Detection for Production ML Systems\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Python-3.9%2B-3776AB?style=flat-square\u0026logo=python\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/PyTorch-EE4C2C?style=flat-square\u0026logo=pytorch\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/HuggingFace-FFD21E?style=flat-square\u0026logo=huggingface\u0026logoColor=black\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Scikit--learn-F7931E?style=flat-square\u0026logo=scikitlearn\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Dashboard-FastAPI-009688?style=flat-square\u0026logo=fastapi\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/License-MIT-6366f1?style=flat-square\" /\u003e\n\u003c/p\u003e\n\n\u003e A lightweight, pip-installable Python library for monitoring data drift and model confidence in production ML systems. Detects when real-world data begins diverging from the training distribution - before model performance visibly degrades.\n\n---\n\n## Table of Contents\n\n- [The Problem](#the-problem)\n- [What This Does](#what-this-does)\n- [Benchmark Results](#benchmark-results)\n- [Research: Confidence-Drift Correlation](#research-confidence-drift-correlation)\n- [Drift Detection Methods](#drift-detection-methods)\n- [Architecture](#architecture)\n- [Repository Structure](#repository-structure)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Confidence Monitoring](#confidence-monitoring)\n- [Model Integrations](#model-integrations)\n- [Synthetic Drift Generation](#synthetic-drift-generation)\n- [Dashboard](#dashboard)\n- [Evaluation](#evaluation)\n- [Roadmap](#roadmap)\n- [Related Work](#related-work)\n- [Citation](#citation)\n\n---\n\n## The Problem\n\nGround truth labels arrive late in almost every production ML system:\n\n- Credit risk: defaults take months to confirm\n- Medical diagnosis: pathology results take days to weeks\n- Fraud detection: chargebacks take weeks to materialize\n- Recommendations: satisfaction is measured indirectly\n\nBy the time accuracy degradation is confirmed, the model may have been producing poor predictions for thousands of inferences. **Unsupervised monitoring - detecting degradation without labels - is essential for production ML reliability.**\n\n---\n\n## What This Does\n\nProduction Drift Detection provides:\n\n1. **Four drift detectors** (KL Divergence, PSI, MMD, ADWIN) with a unified `fit/score/detect/summary` API\n2. **Stream monitoring** with batch ingestion and rolling statistics\n3. **Confidence monitoring** tracking entropy, margin, and trend signals\n4. **Confidence-Drift Correlation module** - empirically testing whether confidence degradation precedes drift detection\n5. **Alerting system** with four severity levels: Healthy, Watch, Warning, Critical\n6. **Interactive 6-page dashboard** (FastAPI + Chart.js)\n7. **Synthetic drift generator** with 8 drift types for reproducible experiments\n8. **Model integrations** for Scikit-learn, PyTorch, and HuggingFace\n\n---\n\n## Benchmark Results\n\nRigorous benchmark across **20 random seeds**, 5 features, 2000 reference samples, drift magnitude 2.0 (10-batch ramp + 40 full-strength batches). All metrics reported as mean ± 2×SEM (~95% CI).\n\n### Core Metrics\n\n| Detector | FPR (clean) | Detection Rate | ROC AUC | Cohen's d (strong) | Composite Rank |\n|---|---|---|---|---|---|\n| **MMD** | **0.0%** | 99.9% ± 0.2% | **1.0000** | 6.38 ± 0.43 | **#1 (0.9225)** |\n| PSI | 39.9% ± 3.0% | **100.0%** | **1.0000** | 4.62 ± 0.26 | #2 (0.8537) |\n| KL Divergence | **0.0%** | 95.2% ± 1.6% | 0.9995 ± 0.0008 | 1.18 ± 0.11 | #3 (0.7831) |\n| ADWIN | 46.5% ± 6.2% | 97.7% ± 1.0% | 0.9751 ± 0.0079 | 2.37 ± 0.13 | #4 (0.7187) |\n\n### Detection Latency\n\nAll four detectors achieve **median latency of 0 batches** - drift is flagged in the same batch it begins. KL Divergence averages 0.20 batches (max 1 batch across all seeds), making all methods effectively immediate.\n\n| Detector | Median Latency | Mean Latency | % Detected |\n|---|---|---|---|\n| KL Divergence | 0 batches | 0.20 batches | 100% |\n| PSI | 0 batches | 0.00 batches | 100% |\n| MMD | 0 batches | 0.00 batches | 100% |\n| ADWIN | 0 batches | 0.00 batches | 100% |\n\n### Threshold Calibration (Youden's F1-Max)\n\nDefault thresholds are well-calibrated for KL and MMD. PSI and ADWIN benefit from recalibration:\n\n| Detector | Default Threshold | Optimal Threshold | F1 (default) | F1 (optimal) | Gain |\n|---|---|---|---|---|---|\n| KL Divergence | 0.1000 | 0.1000 | 0.9708 | 0.9708 | +0.0000 |\n| PSI | 0.1000 | 0.1898 | 0.7567 | 0.9499 | **+0.1932** |\n| MMD | 0.0500 | 0.0108 | 0.9994 | **1.0000** | +0.0006 |\n| ADWIN | 0.1000 | 0.3778 | 0.7700 | 0.9573 | **+0.1872** |\n\n**Recommendation:** Use MMD as the primary detector (zero FPR, perfect AUC, immediate detection). Pair with KL Divergence as a secondary signal. Recalibrate PSI and ADWIN thresholds before production deployment.\n\n---\n\n## Research: Confidence-Drift Correlation\n\n### Hypothesis\n\n\u003e **H1:** Under gradual covariate shift, changes in model confidence precede changes in drift scores by k time steps.\n\u003e\n\u003e **H2:** The lead-lag relationship is asymmetric - confidence is more likely to lead drift than vice versa.\n\n### Empirical Validation\n\nTested across **30 trials** (3 experiment types x 10 seeds), two model classes (LogisticRegression, PyTorch NN), two drift types (covariate shift, perturbation).\n\n| Experiment | H1 Support (KL) | H1 Support (PSI) | H1 Support (MMD) | Avg EWS | Conf Drop | Acc Drop |\n|---|---|---|---|---|---|---|\n| sklearn - covariate shift | 4/10 (40%) | 3/10 (30%) | 2/10 (20%) | 42/100 | -2.69% | -1.45% |\n| PyTorch NN - covariate shift | 3/10 (30%) | 3/10 (30%) | 3/10 (30%) | 42/100 | -1.42% | +6.01% |\n| sklearn - perturbation | 2/10 (20%) | 5/10 (50%) | 1/10 (10%) | 44/100 | -2.35% | -1.45% |\n\n### Conclusion\n\n**H1 is not supported.** Confidence precedes drift in only 20-40% of trials depending on detector and model class - not reliably enough to serve as a universal early warning signal. Across all 30 trials, roughly 27-30% of detector-seed pairs show confidence leading drift.\n\nThis is an honest empirical result. The confidence-drift lead-lag relationship is **detector-dependent and non-deterministic** under gradual covariate shift. The Confidence-Drift Correlation module remains useful as a diagnostic and monitoring signal, but should not be relied upon as a consistent early tripwire without further investigation into conditions where leading behaviour emerges.\n\nCross-correlation analysis does show meaningful **co-movement** between confidence and drift signals (max correlations 0.38-0.77 across detectors), confirming that confidence tracks drift even when it does not reliably precede it.\n\n```python\nfrom production-drift-detection.correlation.confidence_drift import ConfidenceDriftCorrelation\n\ncorrelator = ConfidenceDriftCorrelation(max_lag=10)\nresult = correlator.analyze(confidence_history, drift_history)\n\nprint(f\"Lead-lag: {result['lead_lag_steps']} steps\")\nprint(f\"Early warning score: {result['early_warning_score']}/100\")\n```\n\n---\n\n## Drift Detection Methods\n\n### KL Divergence\nMeasures divergence between reference and current probability distributions. Best for categorical features and probability outputs. Laplace smoothing for numerical stability. **Zero FPR in benchmarks, ROC AUC 0.9995.** Default threshold (0.1) is optimally calibrated - no recalibration needed.\n\n### Population Stability Index (PSI)\nMeasures feature distribution stability over time using binned proportions. Standard convention: PSI \u003c 0.1 (stable), 0.1-0.25 (moderate shift), \u003e 0.25 (significant shift). **Perfect detection rate (100%) but high FPR (39.9%)** - recalibrate threshold to 0.1898 for production use.\n\n### Maximum Mean Discrepancy (MMD)\nKernel-based two-sample test using RBF kernels. Supports multivariate distributions. Median heuristic for automatic bandwidth selection. **Top-ranked detector: zero FPR, perfect AUC (1.0000), Cohen's d of 6.38 under strong drift.** Optimal threshold is 0.0108, not the default 0.05.\n\n### ADWIN-Style Adaptive Windowing\nOnline drift detection for streaming data. Adaptive window resizing based on Hoeffding-bound change detection. No need to store full historical data. **High FPR (46.5%) and weaker perturbation AUC (0.65)** - requires threshold recalibration to 0.3778. Best suited for scenarios requiring online, memory-efficient detection.\n\n---\n\n## Architecture\n\n```\nProduction Data Stream\n         |\n         v\n+---------------------+\n|   Stream Monitor    |   Batch ingestion + rolling statistics\n|                     |   Coordinates all detectors\n+----------+----------+\n           |\n    +------+------+\n    v             v\n+---------+  +------------------+\n| Drift   |  |   Confidence     |\n|Detectors|  |   Monitor        |\n|KL/PSI/  |  | Entropy, Margin, |\n|MMD/ADWIN|  | Trend Analysis   |\n+----+----+  +--------+---------+\n     |                |\n     +-------+--------+\n             |\n             v\n+---------------------+\n| Confidence-Drift    |   Lead-lag correlation analysis\n| Correlation Module  |   Early warning scoring\n+----------+----------+\n           |\n           v\n+---------------------+\n|   Alert Engine      |   Threshold + rolling window rules\n|                     |   Healthy / Watch / Warning / Critical\n+----------+----------+\n           |\n           v\n    Dashboard + Alerts\n```\n\n---\n\n## Repository Structure\n\n```\nproduction-drift-detection/\n|\n+-- src/                         # Core library source\n|   +-- detectors/               # KL, PSI, MMD, ADWIN\n|   +-- monitors/                # StreamMonitor, ConfidenceMonitor\n|   +-- alerts/                  # Alert schemas, rules, engine\n|   +-- correlation/             # Confidence-drift correlation module\n|   +-- data/                    # Synthetic drift generator, loaders\n|   +-- integrations/            # Sklearn, PyTorch, HuggingFace adapters\n|   +-- evaluation/              # Metrics, benchmarks\n|   +-- dashboard/               # FastAPI backend + Chart.js frontend\n|   +-- utils/                   # Validation, logging, statistics\n|\n+-- notebooks/                   # Experiment notebooks\n+-- tests/                       # Test suite\n+-- benchmarks.py                # Standard benchmark runner\n+-- benchmark_rigorous.py        # 20-seed rigorous benchmark suite\n+-- empirical_validation.py      # Confidence-drift correlation validation\n+-- demo.py                      # Demo + dashboard launcher\n+-- pyproject.toml\n+-- LICENSE\n```\n\n---\n\n## Installation\n\n```bash\n# From source\ngit clone https://github.com/royxlead/production-drift-detection.git\ncd production-drift-detection\npip install -e .\n\n# With optional dependencies\npip install -e \".[pytorch]\"        # PyTorch support\npip install -e \".[transformers]\"   # HuggingFace support\npip install -e \".[dev]\"            # Testing + notebooks\npip install -e \".[all]\"            # Everything\n```\n\n**Requirements:** Python 3.9+ · NumPy 2.4+ · Pandas 3.0+\n\n---\n\n## Quick Start\n\n```python\nimport numpy as np\nfrom production-drift-detection.detectors.mmd import MMDDetector      # Recommended\nfrom production-drift-detection.detectors.kl import KLDivergenceDetector\nfrom production-drift-detection.detectors.psi import PSIDetector\n\n# Reference distribution (training data)\nreference = np.random.normal(0, 1, (2000, 5))\n# Production data (shifted)\nproduction = np.random.normal(2, 1, (500, 5))\n\n# Unified API across all detectors\ndetector = MMDDetector(threshold=0.0108)   # Use calibrated threshold\ndetector.fit(reference)\nresult = detector.detect(production)\n\nprint(f\"Drift detected: {result['drift_detected']}\")\nprint(f\"Score: {result['score']:.4f}\")\n```\n\n**Stream monitoring:**\n\n```python\nfrom production-drift-detection.monitors.stream_monitor import StreamMonitor\n\nmonitor = StreamMonitor()\nmonitor.fit(reference)\n\nfor batch_idx in range(10):\n    batch = np.random.normal(batch_idx * 0.2, 1, (100, 5))\n    result = monitor.process_batch(batch)\n    print(f\"Batch {batch_idx}: Status={result['status']}\")\n```\n\n---\n\n## Confidence Monitoring\n\n```python\nfrom production-drift-detection.monitors.confidence_monitor import ConfidenceMonitor\n\nmonitor = ConfidenceMonitor()\n\nfor batch_predictions in prediction_stream:\n    status = monitor.update(batch_predictions)\n    print(f\"Mean confidence: {status['mean_confidence']:.3f}\")\n    print(f\"Entropy trend: {status['entropy_trend']}\")\n    print(f\"Degradation detected: {status['degradation_detected']}\")\n```\n\n**Tracked metrics:** Mean confidence, predictive entropy, margin (top-2 gap), trend direction, ECE approximation, over/underconfidence ratios.\n\n---\n\n## Model Integrations\n\n**Scikit-learn:**\n\n```python\nfrom production-drift-detection.integrations.sklearn_adapter import SklearnAdapter\n\nadapter = SklearnAdapter(sklearn_model)\nresult = adapter.predict(X_test, y_test)\nprint(f\"Drift: {result['drift']['drift_detected']}\")\nprint(f\"Confidence: {result['confidence']['mean_confidence']:.3f}\")\n```\n\n**PyTorch:**\n\n```python\nfrom production-drift-detection.integrations.pytorch_adapter import PyTorchAdapter\n\nadapter = PyTorchAdapter(torch_model)\nresult = adapter.predict(X_tensor)\n```\n\n**HuggingFace:**\n\n```python\nfrom production-drift-detection.integrations.hf_adapter import HFAdapter\n\nadapter = HFAdapter()   # DistilBERT SST-2 by default\nresult = adapter.predict_text([\"Great product!\", \"Terrible experience.\"])\n```\n\n---\n\n## Synthetic Drift Generation\n\n8 drift types for reproducible research:\n\n| Drift Type | Description |\n|---|---|\n| Covariate Shift | Shift feature means |\n| Prior Shift | Change class balance |\n| Gradual Drift | Slowly increasing shift over time |\n| Sudden Drift | Abrupt point shift |\n| Missingness Drift | Inject NaN values |\n| Feature Perturbation | Add noise to a subset of features |\n| Gaussian Noise | Add noise to all features |\n| Feature Corruption | Set features to constant values |\n\n```python\nfrom production-drift-detection.data.synthetic_drift import DriftGenerator\n\ngenerator = DriftGenerator(n_features=5, random_state=42)\nreference = generator.generate_reference(n_samples=2000)\nshifted = generator.gradual_drift(n_samples=500, drift_magnitude=2.0)\n```\n\n---\n\n## Dashboard\n\n```bash\npython -m production-drift-detection.dashboard.server\n# or\npython demo.py --dashboard\n```\n\n**6 pages:**\n1. Overview - current status, active alerts, recent scores\n2. Drift Monitoring - PSI, MMD, KL, ADWIN score trends\n3. Feature Analysis - per-feature drift heatmap\n4. Confidence Monitoring - confidence, entropy, margin history\n5. Confidence-Drift Correlation - lead-lag plots, cross-correlation\n6. Alerts - filterable log with severity indicators\n\n---\n\n## Evaluation\n\nRun the rigorous benchmark (20 seeds, ~45 seconds):\n\n```bash\npython benchmark_rigorous.py\n```\n\nRun the confidence-drift empirical validation:\n\n```bash\npython empirical_validation.py\n```\n\nRun standard benchmarks programmatically:\n\n```python\nfrom production-drift-detection.evaluation.benchmarks import BenchmarkFramework\nfrom production-drift-detection.detectors.kl import KLDivergenceDetector\nfrom production-drift-detection.detectors.mmd import MMDDetector\n\nframework = BenchmarkFramework(detectors=[\n    KLDivergenceDetector(),\n    MMDDetector(),\n])\n\nresults = framework.run_benchmark(drift_magnitude=2.0)\nsensitivity = framework.run_sensitivity_analysis(\n    magnitudes=[0.0, 0.5, 1.0, 2.0, 3.0]\n)\n```\n\n**Testing:**\n\n```bash\npytest                            # All tests\npytest --cov=production-drift-detection           # With coverage\npytest tests/test_detectors.py   # Specific module\npytest -m \"integration\"          # Integration tests only\n```\n\n---\n\n## Research Background\n\nProduction Drift Detection connects to established literature on distribution shift and uncertainty estimation:\n\n- **Confidence-accuracy gap under shift** - Guo et al. (2017) showed modern networks are systematically overconfident. Under distribution shift, ECE increases before accuracy degrades.\n- **Entropy as uncertainty signal** - High entropy predictions signal OOD inputs. Production Drift Detection tracks entropy trends as a population-level monitoring signal.\n- **Self-diagnosing models** - Leibig et al. (2017) demonstrated neural networks can estimate their own uncertainty. Production Drift Detection operationalizes this at the monitoring layer.\n- **Bayesian uncertainty** - MC Dropout (Gal and Ghahramani, 2016) approximates epistemic uncertainty. Production Drift Detection's ConfidenceMonitor is designed to be compatible with MC Dropout outputs.\n\n**Why confidence tracks drift even without leading it:**\nConfidence is continuous - it responds to small perturbations. Accuracy is discontinuous - a prediction is right or wrong. Both signals are informative; their relationship is model-dependent and drift-type-dependent rather than universally ordered.\n\n---\n\n## Roadmap\n\n- [ ] MC Dropout integration for improved epistemic uncertainty estimates\n- [ ] Deep kernel learning for MMD-based drift detection\n- [ ] Conformal prediction interval monitoring\n- [ ] Adaptive threshold calibration (auto-recalibration from validation data)\n- [ ] REST API for production deployment\n- [ ] Kubernetes-native deployment support\n- [ ] Real-time alerting (Slack, PagerDuty, email)\n- [ ] SQL/NoSQL storage for drift history\n- [ ] Multi-model orchestration\n- [ ] A/B test monitoring support\n\n---\n\n## Related Work\n\n- [Unsupervised Confidence Estimation](https://github.com/royxlead/unsupervised-confidence-estimation) - Unsupervised Confidence Estimation\n- [CURA](https://github.com/royxlead/cura-python) - RAG reliability and hallucination mitigation\n- [AutoLLM Forge](https://github.com/royxlead/autollmforge-python) - Efficient LLM fine-tuning\n\n---\n\n## Citation\n\n```bibtex\n@software{roy2026production-drift-detection,\n  author = {Roy, Sourav},\n  title  = {Production Drift Detection: Real-Time Data Drift Detection for Production ML Systems},\n  year   = {2026},\n  url    = {https://github.com/royxlead/production-drift-detection}\n}\n```\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003csub\u003eBuilt by \u003ca href=\"https://github.com/royxlead\"\u003eSourav Roy\u003c/a\u003e · Founding AI/ML Engineer · Yuga AI\u003c/sub\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froyxlead%2Fproduction-drift-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froyxlead%2Fproduction-drift-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froyxlead%2Fproduction-drift-detection/lists"}