{"id":49326846,"url":"https://github.com/meesho/spark_calibration","last_synced_at":"2026-04-26T20:32:29.516Z","repository":{"id":197941943,"uuid":"699851545","full_name":"Meesho/spark_calibration","owner":"Meesho","description":"Spark Calibration - A python package for calibrating probabilities predicted by ML model involving large training \u0026 test datasets as spark dataframes","archived":false,"fork":false,"pushed_at":"2025-12-10T03:59:06.000Z","size":46,"stargazers_count":18,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-17T06:25:24.521Z","etag":null,"topics":["calibration","classification","pysparkml","python","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Meesho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-10-03T13:20:38.000Z","updated_at":"2025-12-10T03:52:45.000Z","dependencies_parsed_at":"2023-10-03T19:34:38.568Z","dependency_job_id":"7ce0464d-a6a5-4b90-9164-10079c293670","html_url":"https://github.com/Meesho/spark_calibration","commit_stats":{"total_commits":10,"total_committers":2,"mean_commits":5.0,"dds":"0.19999999999999996","last_synced_commit":"aca3cd252797b00da0de5d57c60cd421a9bf15a4"},"previous_names":["meesho/spark_calibration"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Meesho/spark_calibration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Meesho%2Fspark_calibration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Meesho%2Fspark_calibration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Meesho%2Fspark_calibration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Meesho%2Fspark_calibration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Meesho","download_url":"https://codeload.github.com/Meesho/spark_calibration/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Meesho%2Fspark_calibration/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32312317,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T19:15:34.056Z","status":"ssl_error","status_checked_at":"2026-04-26T19:15:15.467Z","response_time":129,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calibration","classification","pysparkml","python","spark"],"created_at":"2026-04-26T20:32:29.053Z","updated_at":"2026-04-26T20:32:29.509Z","avatar_url":"https://github.com/Meesho.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Model calibration with pyspark\n\n\u003cimg width=\"1069\" alt=\"Screenshot 2023-10-10 at 3 19 39 PM\" src=\"https://github.com/Meesho/spark_calibration/assets/102668625/d66ad0c9-3501-4f15-a73f-9266f7d3ab4c\"\u003e\n\n\nThis package provides a Betacal class which allows the user to fit/train the default beta calibration model on pyspark dataframes and predict calibrated scores\n\n\n## Setup\n\nspark-calibration package is [uploaded to PyPi](https://pypi.org/project/spark-calibration/) and can be installed with this command:\n\n```\npip install spark-calibration\n```\n\n## Usage\n\n### Training\n\ntrain_df should be a pyspark dataframe containing:\n- A column with raw model scores (default name: `score`)\n- A column with binary labels (default name: `label`)\n- (Optional) A column with sample weights (default name: `weight`)\n\nYou can specify different column names when calling `fit()`. In some tree-based models like LightGBM, the predicted scores may fall outside the [0, 1] range and can even be negative. Please apply a sigmoid function to normalize the outputs accordingly.\n\n```python\nfrom spark_calibration import Betacal\nfrom spark_calibration import display_classification_calib_metrics\nfrom spark_calibration import plot_calibration_curve\n\n# Initialize model\nbc = Betacal(parameters=\"abm\")\n\n# Load training data\ntrain_df = spark.read.parquet(\"s3://train/\")\n\n# Fit the model\nbc.fit(train_df)\n\n# Or specify custom column names\n# bc.fit(train_df, score_col=\"raw_score\", label_col=\"actual_label\")\n\n# Fit with sample weights (optional)\n# bc.fit(train_df, weight_col=\"sample_weight\")\n\n# Access model parameters\nprint(f\"Model coefficients: a={bc.a}, b={bc.b}, c={bc.c}\")\n\n# Or use get_params() method\nparams = bc.get_params()\nprint(f\"Model parameters: {params}\")\n```\n\nThe model learns three parameters:\n- a: Coefficient for log(score)\n- b: Coefficient for log(1-score) \n- c: Intercept term\n\n### Saving and Loading Models\n\nYou can save the trained model to disk and load it later:\n\n```python\n# Save model\nsave_path = bc.save(\"/path/to/save/\")\n\n# Load model\nloaded_model = Betacal.load(\"/path/to/save/\")\n```\n\n### Prediction\n\ntest_df should be a pyspark dataframe containing a column with raw model scores. By default, this column should be named `score`, but you can specify a different column name when calling `predict()`. The `predict` function adds a new column `prediction` which has the calibrated score.\n\n```python\ntest_df = spark.read.parquet(\"s3://test/\")\n\n# Using default column name 'score'\ntest_df = bc.predict(test_df)\n\n# Or specify a custom score column name\n# test_df = bc.predict(test_df, score_col=\"raw_score\")\n```\n\n### Pre \u0026 Post Calibration Classification Metrics\n\nThe test_df should have `score`, `prediction` \u0026 `label` columns. \nThe `display_classification_calib_metrics` functions displays `brier_score_loss`, `log_loss`, `area_under_PR_curve` and `area_under_ROC_curve`\n```python\ndisplay_classification_calib_metrics(test_df)\n```\n#### Output\n```\nmodel brier score loss: 0.08072683729933376\ncalibrated model brier score loss: 0.01014015353257748\ndelta: -87.44%\n\nmodel log loss: 0.3038106859864252\ncalibrated model log loss: 0.053275633947890755\ndelta: -82.46%\n\nmodel aucpr: 0.03471287564672635\ncalibrated model aucpr: 0.03471240518472563\ndelta: -0.0%\n\nmodel roc_auc: 0.7490639506966398\ncalibrated model roc_auc: 0.7490649764289607\ndelta: 0.0%\n```\n\n### Plot the Calibration Curve\n\nComputes true, predicted probabilities (pre \u0026 post calibration) using quantile binning strategy with 50 bins and plots the calibration curve\n\n```python\nplot_calibration_curve(test_df)\n```\n\u003cimg width=\"1069\" alt=\"Screenshot 2023-10-10 at 3 19 39 PM\" src=\"https://github.com/Meesho/spark_calibration/assets/102668625/d66ad0c9-3501-4f15-a73f-9266f7d3ab4c\"\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeesho%2Fspark_calibration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeesho%2Fspark_calibration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeesho%2Fspark_calibration/lists"}