{"id":51491900,"url":"https://github.com/idanmoradarthas/datascienceutils","last_synced_at":"2026-07-07T12:00:44.091Z","repository":{"id":36238971,"uuid":"138488993","full_name":"idanmoradarthas/DataScienceUtils","owner":"idanmoradarthas","description":"Data Science Utils: Frequently Used Methods for Data Science","archived":false,"fork":false,"pushed_at":"2026-05-20T19:00:15.000Z","size":18341,"stargazers_count":37,"open_issues_count":7,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-05-21T00:09:45.773Z","etag":null,"topics":["conda-packages","correlation","data-science","features-correlated","machine-learning","matplotlib","plot-confusion-matrix","plotly","plots","pypi-package","python","scikit-learn","utilities","visualize-features"],"latest_commit_sha":null,"homepage":"https://datascienceutils.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idanmoradarthas.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-06-24T14:32:58.000Z","updated_at":"2026-05-02T07:51:49.000Z","dependencies_parsed_at":"2024-02-11T10:29:31.627Z","dependency_job_id":"64474ae7-b543-4033-a46a-2e32a3524845","html_url":"https://github.com/idanmoradarthas/DataScienceUtils","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/idanmoradarthas/DataScienceUtils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idanmoradarthas%2FDataScienceUtils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idanmoradarthas%2FDataScienceUtils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idanmoradarthas%2FDataScienceUtils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idanmoradarthas%2FDataScienceUtils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idanmoradarthas","download_url":"https://codeload.github.com/idanmoradarthas/DataScienceUtils/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idanmoradarthas%2FDataScienceUtils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35226918,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conda-packages","correlation","data-science","features-correlated","machine-learning","matplotlib","plot-confusion-matrix","plotly","plots","pypi-package","python","scikit-learn","utilities","visualize-features"],"created_at":"2026-07-07T12:00:38.452Z","updated_at":"2026-07-07T12:00:44.084Z","avatar_url":"https://github.com/idanmoradarthas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Science Utils: Frequently Used Methods for Data Science\n\n[![License: MIT](https://img.shields.io/github/license/idanmoradarthas/DataScienceUtils)](https://opensource.org/licenses/MIT)\n![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/idanmoradarthas/DataScienceUtils)\n[![GitHub issues](https://img.shields.io/github/issues/idanmoradarthas/DataScienceUtils)](https://github.com/idanmoradarthas/DataScienceUtils/issues)\n[![Documentation Status](https://readthedocs.org/projects/datascienceutils/badge/?version=latest)](https://datascienceutils.readthedocs.io/en/latest/?badge=latest)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/data-science-utils)\n![PyPI - Wheel](https://img.shields.io/pypi/wheel/data-science-utils)\n[![PyPI version](https://badge.fury.io/py/data-science-utils.svg)](https://badge.fury.io/py/data-science-utils)\n[![Anaconda-Server Badge](https://anaconda.org/idanmorad/data-science-utils/badges/version.svg)](https://anaconda.org/idanmorad/data-science-utils)\n![Build Status](https://github.com/idanmoradarthas/DataScienceUtils/actions/workflows/test.yml/badge.svg?branch=master)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![codecov](https://codecov.io/gh/idanmoradarthas/DataScienceUtils/graph/badge.svg?token=GRQ86SNBNY)](https://codecov.io/gh/idanmoradarthas/DataScienceUtils)\n\nData Science Utils extends the Scikit-Learn API and Matplotlib API to provide simple methods that simplify tasks and\nvisualizations for data science projects.\n\n# Code Examples and Documentation\n\nThis README provides several examples of the package's capabilities. For complete documentation, including more methods and additional examples, please visit:\n**[https://datascienceutils.readthedocs.io/en/stable/](https://datascienceutils.readthedocs.io/en/stable/)**\n\nThe API of the package is built to work with the Scikit-Learn API and Matplotlib API.\n\n## Metrics\n\nThe metrics module is organized into focused submodules:\n- **confusion_matrix** - Confusion matrix visualization and analysis\n- **curves** - ROC and Precision-Recall curves\n- **regression** - Regression Error Characteristic (REC) curves and Regression AUC\n- **time_series** - Time-series and forecasting directional metrics\n- **learning_curves** - Learning curve visualization\n- **probability_analysis** - Probability calibration and accuracy analysis\n\n### Plot Confusion Matrix\n\nComputes and plots a confusion matrix, False Positive Rate, False Negative Rate, Accuracy, and F1 score of a\nclassification.\n\n```python\nfrom ds_utils.metrics.confusion_matrix import plot_confusion_matrix\n\nplot_confusion_matrix(y_test, y_pred, [0, 1, 2])\n```\n\n![multi label classification confusion matrix](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_confusion_matrix/test_plot_confusion_matrix_binary.png)\n\n### Plot Metric Growth per Labeled Instances\n\nReceives train and test sets, and plots the given metric change with an increasing number of trained instances.\n\n```python\nfrom ds_utils.metrics.learning_curves import plot_metric_growth_per_labeled_instances\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import RandomForestClassifier\n\nplot_metric_growth_per_labeled_instances(\n    x_train, y_train, x_test, y_test,\n    {\n        \"DecisionTreeClassifier\": DecisionTreeClassifier(random_state=0),\n        \"RandomForestClassifier\": RandomForestClassifier(random_state=0, n_estimators=5)\n    }\n)\n```\n\n![metric growth per labeled instances with n samples](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_learning_curves/test_plot_metric_growth_per_labeled_instances_with_n_samples.png)\n\n### Visualize Accuracy Grouped by Probability\n\nReceives test true labels and classifier probability predictions, divides and classifies the results, and finally\nplots a stacked bar chart with the results. [Original code](https://github.com/EthicalML/XAI)\n\n```python\nfrom ds_utils.metrics.probability_analysis import visualize_accuracy_grouped_by_probability\n\nvisualize_accuracy_grouped_by_probability(\n    test[\"target\"],\n    1,\n    classifier.predict_proba(test[selected_features]),\n    display_breakdown=False\n)\n```\n\nWithout breakdown:\n\n![visualize_accuracy_grouped_by_probability](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_probability_analysis/test_visualize_accuracy_grouped_by_probability_default.png)\n\nWith breakdown:\n\n![visualize_accuracy_grouped_by_probability_with_breakdown](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_probability_analysis/test_visualize_accuracy_grouped_by_probability_with_breakdown.png)\n\n### Receiver Operating Characteristic (ROC) Curve with Probabilities (Thresholds) Annotations\n\nPlot ROC curves with threshold annotations for multiple classifiers, using plotly as a backend.\n\n```python\nfrom ds_utils.metrics.curves import plot_roc_curve_with_thresholds_annotations\n\nclassifiers_names_and_scores_dict = {\n    \"Decision Tree\": tree_clf.predict_proba(X_test)[:, 1],\n    \"Random Forest\": rf_clf.predict_proba(X_test)[:, 1],\n    \"XGBoost\": xgb_clf.predict_proba(X_test)[:, 1]\n}\nfig = plot_roc_curve_with_thresholds_annotations(\n    y_true,\n    classifiers_names_and_scores_dict,\n    positive_label=1\n)\nfig.show()\n```\n\n![plot roc curve with thresholds annotations](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_curves/test_plot_roc_curve_with_thresholds_annotations_default.png)\n\n### Precision-Recall Curve with Probabilities (Thresholds) Annotations\n\nPlot Precision-Recall curves with threshold annotations for multiple classifiers, using plotly as a backend.\n\n```python\nfrom ds_utils.metrics.curves import plot_precision_recall_curve_with_thresholds_annotations\n\nclassifiers_names_and_scores_dict = {\n    \"Decision Tree\": tree_clf.predict_proba(X_test)[:, 1],\n    \"Random Forest\": rf_clf.predict_proba(X_test)[:, 1],\n    \"XGBoost\": xgb_clf.predict_proba(X_test)[:, 1]\n}\nfig = plot_precision_recall_curve_with_thresholds_annotations(\n    y_true,\n    classifiers_names_and_scores_dict,\n    positive_label=1\n)\nfig.show()\n```\n\n![plot precision recall curve with thresholds annotations](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_curves/test_plot_precision_recall_curve_with_thresholds_annotations_default.png)\n\n### Directional Metrics\n\nDirectional metrics evaluate forecasting performance, especially in time-series and financial contexts where trend direction is key.\n\n```python\nfrom ds_utils.metrics.time_series import directional_accuracy_score, directional_bias_score\nimport numpy as np\n\n# Directional accuracy — time series mode (uses previous value as baseline)\ny_true = np.array([100, 102, 98, 101, 99])\ny_pred = np.array([101, 103, 97, 102, 98])\nda = directional_accuracy_score(y_true, y_pred)\nprint(f\"Directional Accuracy: {da:.2%}\")\n\n# Directional bias — detect systematic over/under-prediction\ny_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0])\ny_pred = np.array([1.1, 2.1, 3.1, 4.1, 5.1])\nbias = directional_bias_score(y_true, y_pred)\nprint(f\"Directional Bias: {bias:.2f}\")\n```\n\nOutput:\n```\nDirectional Accuracy: 100.00%\nDirectional Bias: 1.00\n```\n\n### Regression Error Characteristic (REC) Curve\n\nPlot REC curves with AUC (Area Over the Curve) annotations for multiple regression models. REC curves plot the error tolerance on the x-axis against the accuracy (proportion of predictions within that tolerance) on the y-axis. The AOC provides a normalized metric in [0, 1] where 0 is perfect prediction.\n\n```python\nfrom ds_utils.metrics.regression import plot_rec_curve_with_annotations, regression_auc_score\nimport numpy as np\n\n# Generate dummy data\ny_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0])\npredictions = {\n    \"Good Model\": np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]),\n    \"Bad Model\": np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0]),\n}\n\n# Plot REC curves\nfig = plot_rec_curve_with_annotations(y_true, predictions)\nfig.show()\n\n# Get standalone AOC score\ngood_aoc = regression_auc_score(y_true, predictions[\"Good Model\"])\nprint(f\"Good Model AOC: {good_aoc:.4f}\")\n```\n\nOutput:\n```\nGood Model AOC: 0.1000\n```\n\n![plot rec curve with annotations](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png)\n\n\n### Plot Error Analysis Chart\n\nThis method automates the creation of an error analysis chart by computing the error type (correct, false_positive, false_negative) for each\nprediction and visualizing the distribution of predicted probabilities across these error types. It supports both binary and\nmulti-class classification using a one-vs-rest scheme against a specified positive class.\n\n```python\nimport matplotlib.pyplot as plt\nfrom sklearn.tree import DecisionTreeClassifier\nfrom ds_utils.metrics.probability_analysis import plot_error_analysis_chart\n\n# After training your classifier and generating predictions\ny_pred = clf.predict(X_test)\ny_proba = clf.predict_proba(X_test)[:, 1]  # probability of the positive class\n\n# Plot error analysis\nplot_error_analysis_chart(y_test, y_pred, y_proba, positive_class=1)\nplt.show()\n```\n\n![Plot Error Analysis Chart Binary](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_probability_analysis/test_plot_error_analysis_chart_binary.png)\n\nFor multi-class classification, pass the full 2-D probability matrix and specify the `classes` parameter:\n\n```python\ny_proba = clf.predict_proba(X_test)\n\nplot_error_analysis_chart(\n    y_test, y_pred, y_proba,\n    positive_class=1,\n    classes=clf.classes_.tolist()\n)\nplt.show()\n```\n\n![Plot Error Analysis Chart Multi-class](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_probability_analysis/test_plot_error_analysis_chart_multiclass.png)\n\n### Generate Error Analysis Report\n\nThis method provides a tabular error-analysis report that groups predictions by feature values and computes error metrics per group.\nIt's particularly useful for identifying specific feature ranges or categories where the model underperforms.\n\n```python\nimport pandas as pd\nimport numpy as np\nfrom sklearn.datasets import load_breast_cancer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.tree import DecisionTreeClassifier\nfrom ds_utils.metrics.error_analysis import generate_error_analysis_report\n\n# Load dataset and split\ndata = load_breast_cancer()\nX = pd.DataFrame(data.data, columns=data.feature_names)\n\n# Add a categorical feature for demonstration\nX[\"size_category\"] = pd.cut(\n    X[\"mean radius\"], bins=3, labels=[\"small\", \"medium\", \"large\"]\n).astype(str)\n\ny = data.target\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n\n# Train a classifier\nclf = DecisionTreeClassifier(random_state=42, max_depth=3)\nclf.fit(X_train[[\"mean radius\", \"mean texture\"]], y_train)\n\ny_pred = clf.predict(X_test[[\"mean radius\", \"mean texture\"]])\n\n# Generate error analysis report for numerical and categorical features\nreport = generate_error_analysis_report(\n    X_test, y_test, y_pred,\n    feature_columns=[\"mean radius\", \"mean texture\", \"size_category\"],\n    bins=3,\n    sort_metric=\"error_rate\",\n    ascending=False\n)\nprint(report)\n```\nThe output will be a pandas DataFrame:\n\n| feature | group            | count | error_count | error_rate | accuracy |\n|---------|------------------|-------|-------------|------------|----------|\n| age     | (34.667, 47.333] | 2     | 1           | 0.50       | 0.50     |\n| region  | South            | 2     | 1           | 0.50       | 0.50     |\n| region  | North            | 3     | 1           | 0.33       | 0.67     |\n| age     | (21.962, 34.667] | 4     | 1           | 0.25       | 0.75     |\n| age     | (47.333, 60.0]   | 2     | 0           | 0.00       | 1.00     |\n| region  | East             | 1     | 0           | 0.00       | 1.00     |\n| region  | West             | 2     | 0           | 0.00       | 1.00     |\n\n*(Note: Rows with equal error_rate may appear in any order)*\n## Preprocess\n\nThe preprocess module is organized into focused submodules:\n- **visualization** - Feature visualization and correlation plots\n- **statistics** - Statistical computations and feature analysis\n\n### Visualize Feature\n\nReceives a feature and visualizes its values on a graph:\n\n* If the feature is float, the method plots a violin distribution. You can optionally exclude outliers using the IQR fence method.\n* If the feature is datetime, the method plots a 2D heatmap showing day-of-week vs year-week patterns, making weekly and yearly trends immediately visible.\n* If the feature is object, categorical, boolean, or integer, the method plots a count plot (histogram). For high-cardinality features (\u003e10 unique values), it shows the top 10 with \"Other values\". You can customize sorting with `order` (e.g., by count or alphabetically) and toggle count labels with `show_counts`.\n\nWhen `remove_na=False` (default for `visualize_feature`), missing values are handled as follows:\n* **Float**: Missing values are dropped before plotting (violin plots require valid data).\n* **Datetime**: Missing values are dropped before plotting.\n* **Categorical/Integer/Boolean**: Missing values are counted and shown as a separate category (if present).\n\n```python\nfrom ds_utils.preprocess.visualization import visualize_feature\n\n# Basic usage\nvisualize_feature(X_train[\"feature\"])\n\n# Handle NA values (removes them before plotting)\nvisualize_feature(X_train[\"feature_with_nas\"], remove_na=True)\n\n# For float features, you can control outlier handling\nvisualize_feature(X_train[\"float_feature\"], include_outliers=True)  # Default\nvisualize_feature(X_train[\"float_feature\"], include_outliers=False, outlier_iqr_multiplier=1.5)\n\n# For datetime features, you can specify the first day of the week\nvisualize_feature(X_train[\"datetime_feature\"], first_day_of_week=\"Monday\")  # Default\nvisualize_feature(X_train[\"datetime_feature\"], first_day_of_week=\"Sunday\")\n\n# For categorical/object/boolean/int features, customize order and counts\nvisualize_feature(X_train[\"category_feature\"], show_counts=True)  # Default, shows count labels on bars\nvisualize_feature(X_train[\"category_feature\"], show_counts=False)  # Hides count labels\nvisualize_feature(X_train[\"category_feature\"], order=\"count_desc\")  # Sort by descending count\nvisualize_feature(X_train[\"category_feature\"], order=[\"High\", \"Medium\", \"Low\"])  # Explicit category order\n```\n\n| Feature Type      | Plot                                                                                                                                                                             |\n|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Float             | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_visualize_feature/test_visualize_feature_float_datetime_int_float.png)                            |\n| Integer           | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_visualize_feature/test_visualize_feature_float_datetime_int_int.png)                              |\n| Datetime          | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_visualize_feature/test_visualize_feature_float_datetime_int_datetime.png)                         |\n| Category / Object | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_visualize_feature/test_visualize_feature_object_category_more_than_10_categories_show_counts.png) |\n| Boolean           | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_visualize_feature/test_visualize_feature_bool_show_counts.png)                             |\n\n### Get Correlated Features\n\nCalculate which features are correlated above a threshold and extract a data frame with the correlations and correlation\nto the target feature.\n\n```python\nfrom ds_utils.preprocess.statistics import get_correlated_features\n\ncorrelations = get_correlated_features(correlation_matrix, features, target)\n```\n\n| level_0                | level_1                | level_0_level_1_corr | level_0_target_corr | level_1_target_corr |\n|------------------------|------------------------|----------------------|---------------------|---------------------|\n| income_category_Low    | income_category_Medium | 1.0                  | 0.1182165609358650  | 0.11821656093586504 |\n| term\\_ 36 months       | term\\_ 60 months       | 1.0                  | 0.1182165609358650  | 0.11821656093586504 |\n| interest_payments_High | interest_payments_Low  | 1.0                  | 0.1182165609358650  | 0.11821656093586504 |\n\n### Visualize Correlations\n\nCompute pairwise correlation of columns, excluding NA/null values, and visualize it with a heat map.\n[Original code](https://seaborn.pydata.org/examples/many_pairwise_correlations.html)\n\n```python\nfrom ds_utils.preprocess.visualization import visualize_correlations\n\nvisualize_correlations(correlation_matrix)\n```\n\n![visualize features](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_visualize_correlations/test_visualize_correlations_default.png)\n\n### Plot Correlation Dendrogram\n\nPlot a dendrogram of a correlation matrix. This chart hierarchically displays the most correlated variables by\nconnecting them in a tree-like structure. The closer to the right that the connection is, the more correlated the\nfeatures are. [Original code](https://github.com/EthicalML/XAI)\n\n```python\nfrom ds_utils.preprocess.visualization import plot_correlation_dendrogram\n\nplot_correlation_dendrogram(correlation_matrix)\n```\n\n![plot correlation dendrogram](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_correlation_dendrogram/test_plot_correlation_dendrogram_default.png)\n\n### Plot Features' Interaction\n\nPlots the joint distribution between two features:\n\n* If both features are either categorical, boolean, or object, the method plots the shared histogram.\n* If one feature is either categorical, boolean, or object and the other is numeric, the method plots a boxplot chart.\n* If one feature is datetime and the other is numeric or datetime, the method plots a line plot graph.\n* If one feature is datetime and the other is either categorical, boolean, or object, the method plots a violin plot (combination of boxplot and kernel density estimate).\n* If both features are numeric, the method plots a scatter graph.\n\nWhen `remove_na=False` (default), missing values are visualized:\n* For numeric/datetime plots: Missing values appear as rug plots or markers on the axes boundaries.\n* For categorical plots: Missing values are included as a separate category if present.\n\n```python\nfrom ds_utils.preprocess.visualization import plot_features_interaction\n\nplot_features_interaction(data, \"feature_1\", \"feature_2\")\n```\n\n|                 | Numeric                                                                                                                                                                              | Categorical                                                                                                                                                                           | Boolean                                                                                                                                                                                | Datetime                                                                                                                                                                           |\n|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| **Numeric**     | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_both_numeric.png)        |                                                                                                                                                                                       |                                                                                                                                                                                        |                                                                                                                                                                                    |\n| **Categorical** | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_numeric_categorical.png) | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_both_categorical.png)     |                                                                                                                                                                                        |                                                                                                                                                                                    |\n| **Boolean**     | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_numeric_boolean.png)     | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_categorical_bool.png)     | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_both_bool.png)             |                                                                                                                                                                                    |\n| **Datetime**    | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_datetime_numeric.png)    | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_datetime_categorical.png) | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_datetime_bool_default.png) | ![](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_relationship_between_features/test_plot_relationship_between_features_datetime_datetime.png) |\n\n### Plot PCA Explained Variance\n\nThis method plots the cumulative explained variance ratio of PCA components. It helps users quickly determine how many principal components are required to capture a desired proportion of variance in the data.\nHorizontal reference lines are drawn at 70% and 80% variance.\n\n```python\nfrom ds_utils.preprocess.visualization import plot_pca_explained_variance\n\n# Pass a dataframe containing only numerical features\nplot_pca_explained_variance(data_numerical, use_scaling=True)\n```\n\n![Plot PCA Explained Variance](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_preprocess/test_plot_pca_explained_variance/test_plot_pca_explained_variance_default.png)\n\n### Extract Statistics DataFrame per Label\n\nThis method calculates comprehensive statistical metrics for numerical features grouped by label values. Use this when\nyou want to:\n\n- Analyze how a numerical feature's distribution varies across different categories\n- Detect potential patterns or anomalies in feature behavior per group\n- Generate detailed statistical summaries for reporting or analysis\n- Understand the relationship between features and target variables\n\n```python\nfrom ds_utils.preprocess.statistics import extract_statistics_dataframe_per_label\n\nextract_statistics_dataframe_per_label(\n    df=df,\n    feature_name='amount',\n    label_name='category'\n)\n```\n\n| category | count | null_count | mean  | min | 1_percentile | 5_percentile | 25_percentile | median | 75_percentile | 95_percentile | 99_percentile | max   |\n|----------|-------|------------|-------|-----|--------------|--------------|---------------|--------|---------------|---------------|---------------|-------|\n| A        | 2     | 0          | 150.0 | 100 | 100.0        | 100.0        | 100.0         | 150.0  | 200.0         | 200.0         | 200.0         | 200.0 |\n| B        | 2     | 0          | 225.0 | 150 | 150.0        | 150.0        | 150.0         | 225.0  | 300.0         | 300.0         | 300.0         | 300.0 |\n| C        | 2     | 0          | 212.5 | 175 | 175.0        | 175.0        | 175.0         | 212.5  | 250.0         | 250.0         | 250.0         | 250.0 |\n\n### Compute Mutual Information\n\nThis method computes mutual information scores between features and a target label. Mutual information measures the mutual dependence between two variables - higher scores indicate stronger relationships between features and the target label. Features are automatically categorized as numerical or discrete and preprocessed accordingly.\n\nUse this method when you want to:\n- Identify which features have the strongest relationship with your target variable\n- Perform feature selection based on statistical dependence\n- Understand feature importance from an information theory perspective\n- Compare the predictive value of different types of features\n\n```python\nfrom ds_utils.preprocess.statistics import compute_mutual_information\n\n# Compute mutual information scores for all features\nmi_scores = compute_mutual_information(\n    df=df,\n    features=['feature1', 'feature2', 'feature3'],\n    label_col='target',\n    random_state=42\n)\n```\nThe result will be a DataFrame sorted by MI score (descending):\n\n| feature_name | mi_score |\n|--------------|----------|\n| feature1     | 0.245    |\n| feature3     | 0.182    |\n| feature2     | 0.091    |\n\n## Strings\n\n### Append Tags to Frame\n\nThis method extracts tags from a given field and appends them as new columns to the dataframe.\n\nConsider a dataset that looks like this:\n\n``x_train``:\n\n| article_name | article_tags |\n|--------------|--------------|\n| 1            | ds,ml,dl     |\n| 2            | ds,ml        |\n\n``x_test``:\n\n| article_name | article_tags |\n|--------------|--------------|\n| 3            | ds,ml,py     |\n\nUsing this code:\n\n```python\nimport pandas as pd\nfrom ds_utils.strings import append_tags_to_frame\n\nx_train = pd.DataFrame([{\"article_name\": \"1\", \"article_tags\": \"ds,ml,dl\"},\n                        {\"article_name\": \"2\", \"article_tags\": \"ds,ml\"}])\nx_test = pd.DataFrame([{\"article_name\": \"3\", \"article_tags\": \"ds,ml,py\"}])\n\nx_train_with_tags, x_test_with_tags = append_tags_to_frame(x_train, x_test, \"article_tags\", \"tag_\")\n```\n\nThe result will be:\n\n``x_train_with_tags``:\n\n| article_name | tag_ds | tag_ml | tag_dl |\n|--------------|--------|--------|--------|\n| 1            | 1      | 1      | 1      |\n| 2            | 1      | 1      | 0      |\n\n``x_test_with_tags``:\n\n| article_name | tag_ds | tag_ml | tag_dl |\n|--------------|--------|--------|--------|\n| 3            | 1      | 1      | 0      |\n\n### Extract Significant Terms from Subset\n\nThis method returns interesting or unusual occurrences of terms in a subset. It is based on the\n[elasticsearch significant_text aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html#_scripted).\n\n```python\nimport pandas as pd\nfrom ds_utils.strings import extract_significant_terms_from_subset\n\ncorpus = ['This is the first document.', 'This document is the second document.',\n          'And this is the third one.', 'Is this the first document?']\ndata_frame = pd.DataFrame(corpus, columns=[\"content\"])\n# Let's differentiate between the last two documents from the full corpus\nsubset_data_frame = data_frame[data_frame.index \u003e 1]\nterms = extract_significant_terms_from_subset(data_frame, subset_data_frame,\n                                              \"content\")\n```\n\nThe output for ``terms`` will be the following table:\n\n| third | one | and | this | the  | is   | first | document | second |\n|-------|-----|-----|------|------|------|-------|----------|--------|\n| 1.0   | 1.0 | 1.0 | 0.67 | 0.67 | 0.67 | 0.5   | 0.25     | 0.0    |\n\n## Transformers\n\nThe ``transformers`` package provides scikit-learn compatible wrappers for preprocessing steps that need ``get_feature_names_out`` (feature names in pipelines) and consistent output for downstream estimators.\n\n### MultiLabelBinarizerTransformer\n\nWraps ``sklearn.preprocessing.MultiLabelBinarizer`` so multi-label columns work with ``Pipeline``, ``ColumnTransformer``, and ``set_output(transform=\"pandas\")``. Pass **one iterable of labels per sample** (see the `MultiLabelBinarizer \u003chttps://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html\u003e`_ documentation—a flat list of strings is not valid input).\n\n```python\nfrom ds_utils.transformers.multi_label_binarizer import MultiLabelBinarizerTransformer\nfrom sklearn.pipeline import Pipeline\n\nX = [[\"sci-fi\", \"action\"], [\"romance\"], [\"action\", \"comedy\"]]\nmlb = MultiLabelBinarizerTransformer()\nX_transformed = mlb.fit_transform(X)\nfeature_names = mlb.get_feature_names_out()\n\npipeline = Pipeline([(\"mlb\", MultiLabelBinarizerTransformer())])\npipeline.set_output(transform=\"pandas\")\ndf_transformed = pipeline.fit_transform(X)\n```\n\nBoth `X_transformed` (as a numpy array) and `df_transformed` (as a pandas DataFrame) contain the same binarized data, using `feature_names` for columns.\n\n**Output:**\n\n| label_action | label_comedy | label_romance | label_sci-fi |\n|--------------|--------------|---------------|--------------|\n| 1.0          | 0.0          | 0.0           | 1.0          |\n| 0.0          | 0.0          | 1.0           | 0.0          |\n| 1.0          | 1.0          | 0.0           | 0.0          |\n\n### SentenceEmbeddingTransformer\n\nWraps [sentence-transformers](https://sbert.net/) models for use in sklearn pipelines. Produces dense embedding matrices from text inputs with lazy model loading, ``None``/``NaN`` handling, and ``get_feature_names_out`` support.\n\n\u003e **Note:** Requires the optional ``nlp`` extras: ``pip install data-science-utils[nlp]``\n\n```python\nfrom ds_utils.transformers.sentence_embedding import SentenceEmbeddingTransformer\n\ntexts = [\"The quick brown fox\", \"jumps over the lazy dog\", \"Hello world\"]\nembed = SentenceEmbeddingTransformer()\nembeddings = embed.fit_transform(texts)\nfeature_names = embed.get_feature_names_out()\n```\n\nBoth `embeddings` (as a numpy array) and `feature_names` describe the same embedding matrix of shape `(n_samples, embedding_dimension)` (e.g. `(3, 384)` for the default `sentence-transformers/all-MiniLM-L6-v2` model).\n\n**Output:**\n\n| dim_0    | dim_1    | dim_2    | ... | dim_383  |\n|----------|----------|----------|-----|----------|\n| -0.0123  |  0.0456  |  0.0789  | ... |  0.0012  |\n|  0.0345  | -0.0678  |  0.0901  | ... | -0.0234  |\n|  0.0567  |  0.0890  | -0.0123  | ... |  0.0456  |\n\nPipeline usage with a classifier:\n\n```python\nfrom ds_utils.transformers.sentence_embedding import SentenceEmbeddingTransformer\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.ensemble import RandomForestClassifier\n\npipeline = Pipeline([\n    ('embeddings', SentenceEmbeddingTransformer(normalize_embeddings=True)),\n    ('classifier', RandomForestClassifier()),\n])\npipeline.fit(X_train, y_train)\npredictions = pipeline.predict(X_test)\n```\n\nKey features:\n- **Lazy loading**: the model is loaded only when ``fit()`` is first called\n- **Flexible input**: accepts lists, ``pd.Series``, single-column ``pd.DataFrame``, and ``np.ndarray``\n- **Configurable**: ``batch_size``, ``normalize_embeddings``, ``precision`` (``'float32'``, ``'int8'``, ``'binary'``, etc.), ``truncate_dim``, ``prompt``/``prompt_name``\n\n\n\n## Unsupervised\n\n### Cluster Cardinality\n\nCluster cardinality is the number of examples per cluster. This method plots the number of points per cluster as a bar\nchart.\n\n```python\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom sklearn.cluster import KMeans\nfrom ds_utils.unsupervised import plot_cluster_cardinality\n\ndata = pd.read_csv(path / to / dataset)\nestimator = KMeans(n_clusters=8, random_state=42)\nestimator.fit(data)\n\nplot_cluster_cardinality(estimator.labels_)\n\nplt.show()\n```\n\n![Cluster Cardinality](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_unsupervised/test_cluster_cardinality.png)\n\n### Plot Cluster Magnitude\n\nCluster magnitude is the sum of distances from all examples to the centroid of the cluster. This method plots the\nTotal Point-to-Centroid Distance per cluster as a bar chart.\n\n```python\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom sklearn.cluster import KMeans\nfrom scipy.spatial.distance import euclidean\nfrom ds_utils.unsupervised import plot_cluster_magnitude\n\ndata = pd.read_csv(path / to / dataset)\nestimator = KMeans(n_clusters=8, random_state=42)\nestimator.fit(data)\n\nplot_cluster_magnitude(data, estimator.labels_, estimator.cluster_centers_, euclidean)\n\nplt.show()\n```\n\n![Plot Cluster Magnitude](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_unsupervised/test_plot_cluster_magnitude.png)\n\n### Magnitude vs. Cardinality\n\nHigher cluster cardinality tends to result in a higher cluster magnitude, which intuitively makes sense. Clusters\nare considered anomalous when cardinality doesn't correlate with magnitude relative to the other clusters. This\nmethod helps find anomalous clusters by plotting magnitude against cardinality as a scatter plot.\n\n```python\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom sklearn.cluster import KMeans\nfrom scipy.spatial.distance import euclidean\nfrom ds_utils.unsupervised import plot_magnitude_vs_cardinality\n\ndata = pd.read_csv(path / to / dataset)\nestimator = KMeans(n_clusters=8, random_state=42)\nestimator.fit(data)\n\nplot_magnitude_vs_cardinality(data, estimator.labels_, estimator.cluster_centers_, euclidean)\n\nplt.show()\n```\n\n![Magnitude vs. Cardinality](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_unsupervised/test_plot_magnitude_vs_cardinality.png)\n\n### Optimum Number of Clusters\n\nK-means clustering requires you to decide the number of clusters `k` beforehand. This method runs the KMeans algorithm\nand\nincreases the cluster number at each iteration. The total magnitude or sum of distances is used as the loss metric.\n\nNote: Currently, this method only works with ``sklearn.cluster.KMeans``.\n\n```python\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom scipy.spatial.distance import euclidean\nfrom ds_utils.unsupervised import plot_loss_vs_cluster_number\n\ndata = pd.read_csv(path / to / dataset)\n\nplot_loss_vs_cluster_number(data, 3, 20, euclidean)\n\nplt.show()\n```\n\n![Optimum Number of Clusters](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_unsupervised/test_plot_loss_vs_cluster_number.png)\n\n## XAI (Explainable AI)\n\n## Plot Feature Importance\n\nThis method plots feature importance as a bar chart, helping to visualize which features have the most significant\nimpact on the model's decisions.\n\n```python\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom sklearn.tree import DecisionTreeClassifier\nfrom ds_utils.xai import plot_features_importance\n\n# Load the dataset\ndata = pd.read_csv(path / to / dataset)\ntarget = data[\"target\"]\nfeatures = data.columns.tolist()\nfeatures.remove(\"target\")\n\n# Train a decision tree classifier\nclf = DecisionTreeClassifier(random_state=42)\nclf.fit(data[features], target)\n\n# Plot feature importance\nplot_features_importance(features, clf.feature_importances_)\n\nplt.show()\n```\n\n![Plot Features Importance](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_xai/test_plot_features_importance/test_plot_features_importance.png)\n\nThis visualization helps in understanding which features are most influential in the model's decision-making process,\nproviding valuable insights for feature selection and model interpretation.\n\n## Explore More\n\nExcited about what you've seen so far? There's even more to discover! Dive deeper into each module to unlock the full\npotential of DataScienceUtils:\n\n* [Metrics](https://datascienceutils.readthedocs.io/en/latest/metrics.html) - Powerful methods for calculating and\n  visualizing algorithm performance evaluation. Gain insights into how your models are performing.\n\n* [Preprocess](https://datascienceutils.readthedocs.io/en/latest/preprocess/index.html) - Essential data preprocessing\n  techniques to prepare your data for training. Now organized into visualization and statistics submodules for better code organization.\n\n* [Strings](https://datascienceutils.readthedocs.io/en/latest/strings.html) - Efficient methods for manipulating and\n  processing strings in dataframes. Handle text data with ease.\n\n* [Unsupervised](https://datascienceutils.readthedocs.io/en/latest/unsupervised.html) - Tools for calculating and\n  visualizing the performance of unsupervised models. Understand your clustering and dimensionality reduction results\n  better.\n\n* [XAI](https://datascienceutils.readthedocs.io/en/latest/xai.html) - Methods to help explain model decisions, making\n  your AI more interpretable and trustworthy.\n\nEach module is designed to streamline your data science workflow, providing you with the tools you need to preprocess\ndata, train models, evaluate performance, and interpret results. Check out the detailed documentation for each module to\nsee how DataScienceUtils can enhance your projects!\n\n## Contributing\n\nWe're thrilled that you're interested in contributing to Data Science Utils! Your contributions help make this project\nbetter for everyone. Whether you're a seasoned developer or just getting started, there's a place for you here.\n\n### How to Contribute\n\n1. **Find an area to contribute to**: Check out our [issues](https://github.com/idanmoradarthas/DataScienceUtils/issues)\n   page for open tasks, or think of a feature you'd like to add.\n\n2. **Fork the repository**: Make your own copy of the project to work on.\n\n3. **Create a branch**: Make your changes in a new git branch.\n\n4. **Make your changes**: Add your improvements or fixes. We appreciate:\n    - Bug reports and fixes\n    - Feature requests and implementations\n    - Documentation improvements\n    - Performance optimizations\n    - User experience enhancements\n\n5. **Test your changes**: Ensure your code works as expected and doesn't introduce new issues.\n\n6. **Submit a pull request**: Open a PR with a clear title and description of your changes.\n\n### Coding Guidelines\n\nWe follow the [Python Software Foundation Code of Conduct](http://www.python.org/psf/codeofconduct/) and\nthe [Matplotlib Coding Guidelines](https://matplotlib.org/stable/devel/coding_guide.html). Please adhere to\nthese guidelines in your contributions.\n\n### Getting Help\n\nIf you're new to open source or need any help, don't hesitate to ask questions in\nthe [issues](https://github.com/idanmoradarthas/DataScienceUtils/issues) section or reach out to the\nmaintainers. We're here to help!\n\n### Code Quality\n\nThis project uses [Ruff](https://github.com/astral-sh/ruff) for linting and code formatting.\nContributors are encouraged to use it to ensure code quality and consistency.\nYou can check your code by running:\n\n```bash\nruff check .\nruff format .\n```\n\nThese checks are also part of our CI pipeline.\n\n### Why Contribute?\n\n- **Improve your skills**: Gain experience working on a real-world project.\n- **Be part of a community**: Connect with other developers and data scientists.\n- **Make a difference**: Your contributions will help others in their data science journey.\n- **Get recognition**: All contributors are acknowledged in our project.\n\nRemember, no contribution is too small. Whether it's fixing a typo in documentation or adding a major feature, all\ncontributions are valued and appreciated.\n\nThank you for helping make Data Science Utils better for everyone!\n\n## Installation Guide\n\nThe preferred way to install DataScienceUtils is using the AI skills installer\nscript, because it installs the package and AI skills in one guided flow.\n\n### 1. Preferred: Install using the skills installer script\n\nDataScienceUtils ships AI skills that teach Claude Code, Cursor, GitHub Copilot,\nand Gemini CLI how to use this library correctly — including function signatures,\nimport paths, and common pitfalls.\n\n**Mac / Linux:**\n```bash\nbash \u003c(curl -sL https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/install.sh)\n```\n\n**Windows (PowerShell):**\n```powershell\nirm https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/install.ps1 | iex\n```\n\nThe installer will:\n- Detect which AI tools you have installed (Claude Code, Cursor, GitHub Copilot, Gemini CLI)\n- Ask whether to install for the current project or globally\n- Auto-detect pip or conda and install `data-science-utils`\n- Copy skill files into the correct directory for each detected tool\n\nFor a full step-by-step guide, see the\n[AI Skills documentation](https://datascienceutils.readthedocs.io/en/stable/ai_skills.html).\n\n### 2. Install from PyPI\n\nThe simplest way to install Data Science Utils and its dependencies is from PyPI using pip, Python's preferred package\ninstaller:\n\n```bash\npip install data-science-utils\n```\n\nTo upgrade Data Science Utils to the latest version, use:\n\n```bash\npip install -U data-science-utils\n```\n\n### 3. Install from Source\n\nIf you prefer to install from source, you can clone the repository and install:\n\n```bash\ngit clone https://github.com/idanmoradarthas/DataScienceUtils.git\ncd DataScienceUtils\npip install .\n```\n\nAlternatively, you can install directly from GitHub using pip:\n\n```bash\npip install git+https://github.com/idanmoradarthas/DataScienceUtils.git\n```\n\n### 4. Install using Anaconda\n\nIf you're using Anaconda, you can install using conda:\n\n```bash\nconda install idanmorad::data-science-utils\n```\n\n### Note on Dependencies\n\nData Science Utils has several dependencies, including numpy, pandas, matplotlib, plotly and scikit-learn. These will be\nautomatically installed when you install the package using the methods above.\n\nFor NLP features (``SentenceEmbeddingTransformer``), install with the optional ``nlp`` extras:\n\n```bash\npip install data-science-utils[nlp]\n```\n\n\n## Staying Updated\n\nData Science Utils is an active project that routinely publishes new releases with additional methods and improvements.\nWe recommend periodically checking for updates to access the latest features and bug fixes.\n\nIf you encounter any issues during installation, please check our\nGitHub [issues](https://github.com/idanmoradarthas/DataScienceUtils/issues) page or open a new issue for assistance.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidanmoradarthas%2Fdatascienceutils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidanmoradarthas%2Fdatascienceutils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidanmoradarthas%2Fdatascienceutils/lists"}