{"id":24539286,"url":"https://github.com/statsim/confusionmatrix","last_synced_at":"2026-01-03T05:05:37.742Z","repository":{"id":270736298,"uuid":"891191619","full_name":"statsim/confusionmatrix","owner":"statsim","description":"Generate Confusion Matrix and Evaluation Metrics Online","archived":false,"fork":false,"pushed_at":"2025-01-02T18:06:02.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T16:16:01.125Z","etag":null,"topics":["classification","confusion-matrix","machine-learning"],"latest_commit_sha":null,"homepage":"https://statsim.com/confusionmatrix/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/statsim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-19T22:21:41.000Z","updated_at":"2025-01-02T18:06:06.000Z","dependencies_parsed_at":"2025-01-02T19:20:23.526Z","dependency_job_id":"b2dd1013-1352-4f1f-b92c-dbd8057cd9fb","html_url":"https://github.com/statsim/confusionmatrix","commit_stats":null,"previous_names":["statsim/confusionmatrix"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statsim%2Fconfusionmatrix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statsim%2Fconfusionmatrix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statsim%2Fconfusionmatrix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statsim%2Fconfusionmatrix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/statsim","download_url":"https://codeload.github.com/statsim/confusionmatrix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243822310,"owners_count":20353498,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["classification","confusion-matrix","machine-learning"],"created_at":"2025-01-22T16:16:13.145Z","updated_at":"2026-01-03T05:05:37.665Z","avatar_url":"https://github.com/statsim.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Generate Confusion Matrix and Evaluation Metrics Online\n\n*A confusion matrix is a useful tool for evaluating the performance of classification models. It provides a breakdown of predicted versus actual outcomes, allowing for a deeper understanding of model performance beyond just accuracy.*\n\n---\n\n## **Why Accuracy is Not Enough for Binary Classification?**\n\nAccuracy is the proportion of correctly classified instances among all instances:\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{Accuracy} = \\frac{\\text{TP} + \\text{TN}}{\\text{Total}}\n```\n\u003c/div\u003e\n\nWhere $$\\text{TP}$$ is the number of true positives, $$\\text{TN}$$ is the number of true negatives, and $$\\text{Total}$$ is the total number of instances.\n\n\n### **Limitations of Accuracy as a Metric**\n- **Class Imbalance.** In datasets with imbalanced classes (e.g., 95% negatives and 5% positives), a model that always predicts the majority class (negative) can achieve high accuracy but fails to capture the minority class.\n- **No Insight into Error Types.** Accuracy does not distinguish between types of errors (e.g., false positives vs. false negatives), which can have vastly different implications in real-world scenarios.\n\n---\n\n## **Confusion Matrix Terminology**\n\n|                | **Predicted Positive** | **Predicted Negative** |\n|----------------|-------------------------|-------------------------|\n| **Actual Positive** | True Positive (TP): Correctly identified positive cases | False Negative (FN): Missed positive cases |\n| **Actual Negative** | False Positive (FP): Incorrectly predicted positive cases | True Negative (TN): Correctly identified negative cases |\n\n### **Key Terms**:\n- **TP (True Positive)**: Correctly predicted positive instances.\n- **TN (True Negative)**: Correctly predicted negative instances.\n- **FP (False Positive)**: Negative instances incorrectly predicted as positive.\n- **FN (False Negative)**: Positive instances incorrectly predicted as negative.\n\n---\n\n## **Metrics and Their Meaning**\n\n### **1. Sensitivity (True Positive Rate, TPR)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{TPR} = \\frac{\\text{TP}}{\\text{TP} + \\text{FN}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of actual positives correctly identified.\n- **Importance**: Measures the ability to detect positive cases (useful in medical diagnosis, fraud detection).\n\n---\n\n### **2. Specificity (SPC)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{SPC} = \\frac{\\text{TN}}{\\text{FP} + \\text{TN}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of actual negatives correctly identified.\n- **Importance**: Measures the ability to avoid false alarms (useful in spam filters, anomaly detection).\n\n---\n\n### **3. Precision (Positive Predictive Value, PPV)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{PPV} = \\frac{\\text{TP}}{\\text{TP} + \\text{FP}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of positive predictions that are correct.\n- **Importance**: Indicates reliability of positive predictions.\n\n---\n\n### **4. Negative Predictive Value (NPV)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{NPV} = \\frac{\\text{TN}}{\\text{TN} + \\text{FN}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of negative predictions that are correct.\n- **Importance**: Indicates reliability of negative predictions.\n\n---\n\n### **5. False Positive Rate (FPR)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{FPR} = \\frac{\\text{FP}}{\\text{FP} + \\text{TN}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of actual negatives incorrectly predicted as positive.\n- **Importance**: Highlights the rate of false alarms.\n\n---\n\n### **6. False Discovery Rate (FDR)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{FDR} = \\frac{\\text{FP}}{\\text{FP} + \\text{TP}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of positive predictions that are incorrect.\n- **Importance**: Complements precision in evaluating prediction quality.\n\n---\n\n### **7. False Negative Rate (FNR)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{FNR} = \\frac{\\text{FN}}{\\text{FN} + \\text{TP}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of actual positives incorrectly predicted as negative.\n- **Importance**: Highlights the rate of missed positive cases.\n\n---\n\n### **8. F1 Score**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{F1} = \\frac{2 \\cdot \\text{TP}}{2 \\cdot \\text{TP} + \\text{FP} + \\text{FN}}\n```\n\u003c/div\u003e\n\n- **Definition**: Harmonic mean of precision and recall.\n- **Importance**: Balances precision and recall, especially useful in imbalanced datasets.\n\n---\n\n### **9. Accuracy**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{ACC} = \\frac{\\text{TP} + \\text{TN}}{\\text{P} + \\text{N}}\n```\n\u003c/div\u003e\n\n- **Definition**: Proportion of correct predictions.\n- **Importance**: Provides a general sense of model performance but is less reliable in imbalanced datasets.\n\n---\n\n### **10. Matthews Correlation Coefficient (MCC)**\n\n\u003cdiv style=\"text-align: center; font-size: 1.5em; margin:10px;\"\u003e\n```latex\n\\text{MCC} = \\frac{\\text{TP} \\cdot \\text{TN} - \\text{FP} \\cdot \\text{FN}}{\\sqrt{(\\text{TP} + \\text{FP})(\\text{TP} + \\text{FN})(\\text{TN} + \\text{FP})(\\text{TN} + \\text{FN})}}\n```\n\u003c/div\u003e\n\n- **Definition**: A balanced measure that accounts for TP, TN, FP, and FN.\n- **Importance**: Considered a robust metric for imbalanced datasets. Ranges from -1 (inverse prediction) to +1 (perfect prediction), with 0 indicating random performance.\n\n---\n\n## **Key Takeaways**\n\n1. **Class Imbalance Requires Caution**: Accuracy alone can be misleading when classes are imbalanced.\n2. **Use Multiple Metrics**: Evaluate sensitivity, specificity, precision, and other metrics to understand the trade-offs in your model.\n3. **MCC for Imbalanced Datasets**: Use Matthews Correlation Coefficient for a single comprehensive measure.\n4. **Domain-Specific Importance**: Choose metrics based on the problem domain (e.g., sensitivity for medical tests, precision for legal applications).\n\n--- \n\nBy understanding and applying these metrics, you can better assess and improve your model's performance, ensuring it meets the requirements of the task at hand.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatsim%2Fconfusionmatrix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstatsim%2Fconfusionmatrix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatsim%2Fconfusionmatrix/lists"}