{"id":20284065,"url":"https://github.com/ccao-data/report-model-benchmark","last_synced_at":"2026-05-30T21:31:38.581Z","repository":{"id":199540078,"uuid":"698417063","full_name":"ccao-data/report-model-benchmark","owner":"ccao-data","description":"Benchmark of timing for CCAO models on different hardware","archived":false,"fork":false,"pushed_at":"2023-10-06T22:41:37.000Z","size":166,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-09T23:28:58.649Z","etag":null,"topics":["benchmark","data-science","lightgbm","r","xgboost"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ccao-data.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-29T21:47:39.000Z","updated_at":"2023-10-10T20:10:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"da88f9e3-a309-42f6-a79b-77168b4f2baa","html_url":"https://github.com/ccao-data/report-model-benchmark","commit_stats":null,"previous_names":["ccao-data/report-model-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ccao-data/report-model-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Freport-model-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Freport-model-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Freport-model-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Freport-model-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ccao-data","download_url":"https://codeload.github.com/ccao-data/report-model-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Freport-model-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33711018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","data-science","lightgbm","r","xgboost"],"created_at":"2024-11-14T14:18:12.921Z","updated_at":"2026-05-30T21:31:38.560Z","avatar_url":"https://github.com/ccao-data.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"Model Benchmark\n================\n\nThis repository compares the time-to-train of two gradient-boosted\ndecision tree (GBDT) frameworks - LightGBM and XGBoost - across\ndifferent hardware and versions. The purpose of this comparison is to\nhelp the CCAO make two decisions:\n\n1.  Which GBDT framework to use for its 2024 automated valuation models\n2.  Whether or not to purchase or rent additional hardware (a GPU) in\n    order to improve model training speed\n\nBelow are the results of our tests:\n\n\u003e :warning: **NOTE:**\n\u003e\n\u003e The performance statistics presented here are only for cross-model\n\u003e comparison and do not reflect any real model results e.g. they are\n\u003e included only to show that each GBDT framework and version generates\n\u003e similar results, given the same data.\n\n## Results\n\n| Server | Type                                                                                                                         | Device | Package Version | Peak Device Utilization | Wall Time (Full Run) | Wall Time (Prediction) | Wall Time (SHAP) |      RMSE |      MAE |   MAPE |    R2 |    COD |   PRD | PRB    | MKI   |\n|:-------|:-----------------------------------------------------------------------------------------------------------------------------|:-------|:----------------|------------------------:|---------------------:|-----------------------:|-----------------:|----------:|---------:|-------:|------:|-------:|------:|:-------|:------|\n| CCAO   | LightGBM                                                                                                                     | CPU    | 3.3.5           |                    100% |               3m 55s |                  1m 5s |        2h 1m 38s | \\$130,989 | \\$74,124 | 26.97% | 0.883 | 27.634 | 1.140 | −0.225 | 0.851 |\n| CCAO   | LightGBM                                                                                                                     | CPU    | 4.1.0           |                    100% |                4m 8s |                  1m 4s |        2h 1m 48s | \\$130,989 | \\$74,124 | 26.97% | 0.883 | 27.634 | 1.140 | −0.225 | 0.851 |\n| CCAO   | XGBoost\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e1\u003c/sup\u003e\u003c/span\u003e  | CPU    | 2.0.0.1         |                    100% |               2m 41s |                    12s |           2m 14s | \\$126,389 | \\$74,129 | 26.76% | 0.885 | 27.458 | 1.130 | −0.212 | 0.866 |\n| NVIDIA | LightGBM                                                                                                                     | CPU    | 3.3.5           |                    100% |               4m 12s |                     9s |           19m 5s | \\$130,989 | \\$74,124 | 26.97% | 0.883 | 27.634 | 1.140 | −0.225 | 0.851 |\n| NVIDIA | LightGBM                                                                                                                     | CPU    | 4.1.0.99        |                    100% |               1m 37s |                    17s |          19m 44s | \\$130,989 | \\$74,124 | 26.97% | 0.883 | 27.634 | 1.140 | −0.225 | 0.851 |\n| NVIDIA | LightGBM\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e2\u003c/sup\u003e\u003c/span\u003e | GPU    | 4.1.0.99        |                     10% |               2m 33s |                    19s |          19m 39s | \\$130,541 | \\$74,282 | 27.23% | 0.884 | 27.903 | 1.143 | −0.230 | 0.847 |\n| NVIDIA | XGBoost\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e1\u003c/sup\u003e\u003c/span\u003e  | CPU    | 2.0.0.1         |                    100% |               1m 25s |                     5s |              52s | \\$126,894 | \\$74,191 | 26.83% | 0.884 | 27.552 | 1.130 | −0.210 | 0.868 |\n| NVIDIA | XGBoost\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e1\u003c/sup\u003e\u003c/span\u003e  | GPU    | 2.0.0.1         |                     92% |               1m 12s |                    13s |               5s | \\$126,952 | \\$74,124 | 26.78% | 0.885 | 27.515 | 1.130 | −0.213 | 0.867 |\n| AWS    | LightGBM                                                                                                                     | CPU    | 4.1.0.99        |                    100% |               2m 38s |                  1m 1s |       1h 45m 59s | \\$130,989 | \\$74,124 | 26.97% | 0.883 | 27.634 | 1.140 | −0.225 | 0.851 |\n| AWS    | LightGBM\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e2\u003c/sup\u003e\u003c/span\u003e | GPU    | 4.1.0.99        |                      8% |               3m 56s |                 1m 10s |       1h 47m 19s | \\$130,632 | \\$74,210 | 27.18% | 0.884 | 27.866 | 1.143 | −0.230 | 0.847 |\n| AWS    | XGBoost\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e1\u003c/sup\u003e\u003c/span\u003e  | CPU    | 2.0.0.1         |                    100% |               1m 58s |                    11s |           1m 31s | \\$125,867 | \\$73,899 | 26.79% | 0.886 | 27.510 | 1.130 | −0.212 | 0.869 |\n| AWS    | XGBoost\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e1\u003c/sup\u003e\u003c/span\u003e  | GPU    | 2.0.0.1         |                     95% |               1m 31s |                    17s |               6s | \\$126,679 | \\$74,002 | 26.80% | 0.885 | 27.523 | 1.130 | −0.210 | 0.867 |\n\n1.  Categoricals with over 50 values are hashed, otherwise one-hot\n    encoded.\n2.  Categoricals with over 50 values are hashed, otherwise natively\n    handled.\n\n## Cost estimates\n\n| Server                                                                                                                     | Type     | Device | Instance Type | Est. Time Per Run | Est. Cost Per Run | Est. Total 2024 Cost |\n|:---------------------------------------------------------------------------------------------------------------------------|:---------|:-------|:--------------|------------------:|------------------:|---------------------:|\n| CCAO                                                                                                                       | LightGBM | CPU    | \\-            |        57h 55m 5s |                \\- |                   \\- |\n| CCAO                                                                                                                       | XGBoost  | CPU    | \\-            |        9h 25m 14s |                \\- |                   \\- |\n| NVIDIA\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e1\u003c/sup\u003e\u003c/span\u003e | XGBoost  | GPU    | \\-            |        3h 54m 34s |           \\$35.00 |           \\$7,000.00 |\n| AWS\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e2\u003c/sup\u003e\u003c/span\u003e    | XGBoost  | GPU    | Normal        |        4h 54m 10s |            \\$7.96 |           \\$1,592.48 |\n| AWS\u003cspan class=\"gt_footnote_marks\" style=\"white-space:nowrap;font-style:italic;font-weight:normal;\"\u003e\u003csup\u003e2\u003c/sup\u003e\u003c/span\u003e    | XGBoost  | GPU    | Spot          |        4h 54m 10s |            \\$3.29 |             \\$658.56 |\n\n1.  Estimate assumes a fixed cost for an NVIDIA A40 of \\$7,000.\n2.  Estimates use AWS costs for `g5.4xlarge` instances created\n    ephemerally using AWS Batch + Fargate. As of 2023-10-06, costs are:\n    - Normal hourly pricing: \\$1.62\n    - Spot hourly pricing: \\$0.67\n3.  All estimates assume 200 total runs in 2024, costs per run decrease\n    for the NVIDIA option as number of runs increases.\n\n## Hardware\n\nThese tests were run on three different machines: an on-prem modeling\nserver used by the CCAO, a test server provided by NVIDIA via the\n[LaunchPad](https://www.nvidia.com/en-us/launchpad/) program, and a\nstandard AWS `g5.4xlarge` EC2 instance. The machines have the following\nspecifications:\n\n|              | CCAO                                     | NVIDIA                                 | g5.4xlarge                       |\n|--------------|------------------------------------------|----------------------------------------|----------------------------------|\n| **CPU**      | Xeon Silver 4208 CPU @ 2.10GHz, 16 cores | Xeon Gold 6354 CPU @ 3.00GHz, 16 cores | AMD EPYC 7R32 @ 3.3GHz, 16 cores |\n| **Memory**   | 128GiB                                   | 512GiB                                 | 64GiB                            |\n| **GPU**      | \\-                                       | NVIDIA A40, 48GB                       | NVIDIA A10G, 24GB                |\n| **OS**       | Ubuntu 22.04 LTS                         | Ubuntu 22.04 LTS                       | Ubuntu 22.04 LTS                 |\n| **Compiler** | gcc (11.4.0) -O3 -march=native           | gcc (11.4.0) -O3 -march=native         | gcc (11.4.0) -O3 -march=native   |\n\n## Tasks\n\nThe tasks performed for this benchmark (as shown in the results table)\nare as follows:\n\n- **Full Run** - The model is trained on the training + test set, to be\n  used for prediction on unseen data.\n- **Prediction** - The trained model is used to predict on the\n  assessment data.\n- **SHAP** - The trained model is used to predict SHAP values for the\n  first 50K rows of assessment data.\n- For performance metrics, the model is trained on a training set, then\n  predicts on a holdout test set. Performance is calculated using the\n  test set predictions.\n\n## Inputs\n\nThe benchmark uses the following inputs:\n\n- Input data from the 2023 CCAO residential valuation model:\n  - [`input/training_data.parquet`](https://ccao-data-public-us-east-1.s3.amazonaws.com/models/inputs/res/2023/training_data.parquet) -\n    424,950 rows\n  - [`input/assessment_data.parquet`](https://ccao-data-public-us-east-1.s3.amazonaws.com/models/inputs/res/2023/assessment_data.parquet) -\n    1,099,226 rows\n- (Hyper)parameters used for this benchmark can be found in\n  [`params.yaml`](./params.yaml)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccao-data%2Freport-model-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fccao-data%2Freport-model-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccao-data%2Freport-model-benchmark/lists"}