{"id":13424957,"url":"https://github.com/nikolaydubina/go-ml-benchmarks","last_synced_at":"2025-08-26T11:38:42.385Z","repository":{"id":97356370,"uuid":"337365980","full_name":"nikolaydubina/go-ml-benchmarks","owner":"nikolaydubina","description":"⏱ Benchmarks of machine learning inference for Go","archived":false,"fork":false,"pushed_at":"2024-05-09T04:53:59.000Z","size":15006,"stargazers_count":31,"open_issues_count":6,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T04:14:41.398Z","etag":null,"topics":["benchmarks","cpp","go","grpc","inference","machine-learning","python","scikit-learn","xgboost"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nikolaydubina.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"nikolaydubina"}},"created_at":"2021-02-09T10:20:46.000Z","updated_at":"2025-03-15T23:21:11.000Z","dependencies_parsed_at":"2024-06-21T15:52:13.171Z","dependency_job_id":null,"html_url":"https://github.com/nikolaydubina/go-ml-benchmarks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nikolaydubina/go-ml-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikolaydubina%2Fgo-ml-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikolaydubina%2Fgo-ml-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikolaydubina%2Fgo-ml-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikolaydubina%2Fgo-ml-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nikolaydubina","download_url":"https://codeload.github.com/nikolaydubina/go-ml-benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nikolaydubina%2Fgo-ml-benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272215295,"owners_count":24893429,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarks","cpp","go","grpc","inference","machine-learning","python","scikit-learn","xgboost"],"created_at":"2024-07-31T00:01:01.225Z","updated_at":"2025-08-26T11:38:42.369Z","avatar_url":"https://github.com/nikolaydubina.png","language":"Go","funding_links":["https://github.com/sponsors/nikolaydubina"],"categories":["Go","Benchmarks","基准","Twitter"],"sub_categories":["Other Software","其他软件","[Tools](#tools-1)"],"readme":"# Go Machine Learning Benchmarks\n\n\u003e Given a raw data in a Go service, how quickly can I get machine learning inference for it?\n\nTypically, Go is dealing with structured single sample data.\nThus, we are focusing on tabular machine learning models only, such as popular [XGBoost](https://github.com/dmlc/xgboost).\nIt is common to run Go service in a backed form and on Linux platform, thus we do not consider other deployment options.\nIn the work bellow, we compare typical implementations on how this inference task can be performed.\n\n![diagram](docs/go-ml-benchmarks.png)\n\n```\nhost: AWS EC2 t2.xlarge shared\nos: Ubuntu 20.04 LTS \ngoos: linux\ngoarch: amd64\ncpu: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz\nBenchmarkXGB_Go_GoFeatureProcessing_GoLeaves_noalloc                              491 ns/op\nBenchmarkXGB_Go_GoFeatureProcessing_GoLeaves                                      575 ns/op\nBenchmarkXGB_Go_GoFeatureProcessing_UDS_RawBytes_Python_XGB                    243056 ns/op\nBenchmarkXGB_CGo_GoFeatureProcessing_XGB                                       244941 ns/op\nBenchmarkXGB_Go_GoFeatureProcessing_UDS_gRPC_CPP_XGB                           367433 ns/op\nBenchmarkXGB_Go_GoFeatureProcessing_UDS_gRPC_Python_XGB                        785147 ns/op\nBenchmarkXGB_Go_UDS_gRPC_Python_sklearn_XGB                                  21699830 ns/op\nBenchmarkXGB_Go_HTTP_JSON_Python_Gunicorn_Flask_sklearn_XGB                  21935237 ns/op\n```\n\n### Abbreviations and Frameworks\n\n- Transport: Unix Domain Sockets (UDS), TCP, HTTP\n- Encoding: JSON, [gRPC](https://grpc.io/), raw bytes with fixed number of float64 IEEE 754\n- Preprocessing: [go-featureprocessing](https://github.com/nikolaydubina/go-featureprocessing), [sklearn](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing)\n- Model: [XGBoost](https://github.com/dmlc/xgboost), [Leaves](https://github.com/dmitryikh/leaves) (Leaves is XGBoost in native Go)\n- Web Servers: for Python used [Gunicorn](https://gunicorn.org/) + [Flask](https://flask.palletsprojects.com/en/1.1.x/)\n\n### Dataset and Model\n\nWe are using classic [Titanic dataset](https://www.kaggle.com/c/titanic).\nIt contains numerical and categorical features, which makes it a representative of typical case.\nData and notebooks to train model and preprocessor is available in /data and /notebooks.\n\n### Some numbers for reference\n\nHow fast do you need to get?\n\n```\n                   200ps - 4.6GHz single cycle time\n                1ns      - L1 cache latency\n               10ns      - L2/L3 cache SRAM latency\n               20ns      - DDR4 CAS, first byte from memory latency\n               20ns      - C++ raw hardcoded structs access\n               80ns      - C++ FlatBuffers decode/traverse/dealloc\n              150ns      - PCIe bus latency\n              171ns      - cgo call boundary, 2015\n              200ns      - HFT FPGA\n              475ns      - 2020 MLPerf winner recommendation inference time per sample\n ----------\u003e  500ns      - go-featureprocessing + leaves\n              800ns      - Go Protocol Buffers Marshal\n              837ns      - Go json-iterator/go json unmarshal\n           1µs           - Go protocol buffers unmarshal\n           3µs           - Go JSON Marshal\n           7µs           - Go JSON Unmarshal\n          10µs           - PCIe/NVLink startup time\n          17µs           - Python JSON encode/decode times\n          30µs           - UNIX domain socket; eventfd; fifo pipes\n         100µs           - Redis intrinsic latency; KDB+; HFT direct market access\n         200µs           - 1GB/s network air latency; Go garbage collector pauses interval 2018\n         230µs           - San Francisco to San Jose at speed of light\n         500µs           - NGINX/Kong added latency\n     10ms                - AWS DynamoDB; WIFI6 \"air\" latency\n     15ms                - AWS Sagemaker latency; \"Flash Boys\" 300million USD HFT drama\n     30ms                - 5G \"air\" latency\n     36ms                - San Francisco to Hong-Kong at speed of light\n    100ms                - typical roundtrip from mobile to backend\n    200ms                - AWS RDS MySQL/PostgreSQL; AWS Aurora\n 10s                     - AWS Cloudfront 1MB transfer time\n```\n\n### Profiling and Analysis\n\n**[491ns/575ns]** Leaves — we see that most of time taken in Leaves Random Forest code. Leaves code does not have mallocs. Inplace preprocessing does not have mallocs, with non-inplace version malloc happen and takes and takes half of time of preprocessing.\n![leaves](docs/profiles-readme/leaves.png)\n\n**[243µs]** UDS Raw bytes Python — we see that Python takes much longer time than preprocessing in Go, however Go is at least visible on the chart. We also note that Python spends most of the time in `libgomp.so` call, this library is in GNU OpenMP written in C which does parallel operations.\n\n![uds](docs/profiles-readme/uds.png)\n\n**[244µs]** CGo version — similarly, we see that call to `libgomp.so` is being done. It is much smaller compare to rest of o CGo code, as compared to Python version above. Over overall results are not better then? Likely this is due to performance degradation from Go to CGo. We also note that malloc is done.\n\n![cgo](docs/profiles-readme/cgo.png)\n\n**[367µs]** gRPC over UDS to C++ — we see that Go code is around 50% of C++ version. In C++ 50% of time spend on gRPC code.\nLastly, C++ also uses `libgomp.so`. We don't see on this chart, but likely Go code also spends considerable time on gRPC code.\n\n![cgo](docs/profiles-readme/grpc-cpp.png)\n\n**[785µs]** gRPC over UDS to Python wihout sklearn — we see that Go code is visible in the chart. Python spends only portion on time in `libgomp.so`.\n\n![cgo](docs/profiles-readme/grpc-python-processed.png)\n\n**[21ms]** gRPC over UDS to Python with sklearn — we see that Go code (`main.test`) is no longer visible the chart. Python spends only small fraction of time on `libgomp.so`.\n\n![cgo](docs/profiles-readme/grpc-python-sklearn.png)\n\n\n**[22ms]** REST service version with sklearn — similarly, we see that Go code (`main.test`) is no longer visible in the chart. Python spends more time in `libgomp.so` as compared to Python + gRPC + skelarn version, however it is not clear why results are worse.\n\n![cgo](docs/profiles-readme/rest.png)\n\n### Future work\n\n- [ ] go-featureprocessing - gRPCFlatBuffers - C++ - XGB\n- [ ] batch mode\n- [ ] UDS - gRPC - C++ - ONNX (sklearn + XGBoost)\n- [ ] UDS - gRPC - Python - ONNX (sklearn + XGBoost)\n- [ ] cgo ONNX (sklearn + XGBoost) (examples: [1](http://onnx.ai/sklearn-onnx/auto_examples/plot_pipeline_xgboost.html))\n- [ ] native Go ONNX (sklearn + XGBoost) — no official support, https://github.com/owulveryck/onnx-go is not complete\n- [ ] text\n- [ ] images\n- [ ] videos\n\n### Reference\n\n- [Go GC updates, 2018](https://blog.golang.org/ismmkeynote)\n- [cgo performance, GopherCon'18](https://about.sourcegraph.com/go/gophercon-2018-adventures-in-cgo-performance/)\n- [cgo performance, CockroachDB](https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/)\n- [cgo call to CPython, Datadog](https://www.datadoghq.com/blog/engineering/cgo-and-python/)\n- [cgo call to CPython, EuroPython'19](https://ep2019.europython.eu/talks/Zktoaai-golang-to-python/)\n- [HFT latency](https://en.wikipedia.org/wiki/Ultra-low_latency_direct_market_access)\n- [HFT FPGA latency](https://ieeexplore.ieee.org/document/6299067)\n- [HFT FPGA 200 nanoseconds, 2018](https://apnews.com/press-release/pr-businesswire/2edb1f8f12d64ab490ef0c180e648e24)\n- [Google TPU latency](https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html)\n- [PCIe latency](https://www.cl.cam.ac.uk/research/srg/netos/projects/pcie-bench/neugebauer2018understanding.pdf)\n- \"Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect\", 2019\n- [\"Evaluation of Inter-Process Communication Mechanisms\"](http://pages.cs.wisc.edu/~adityav/Evaluation_of_Inter_Process_Communication_Mechanisms.pdf)\n- [UNIX local IPC latencies](http://kamalmarhubi.com/blog/2015/06/10/some-early-linux-ipc-latency-data/)\n- [Cache and DRAM latency](https://en.wikipedia.org/wiki/CPU_cache)\n- [MLPerf benchmarks](https://github.com/mlcommons/inference)\n- [MLPerf benchmarks results, 2020](https://mlperf.org/inference-results-0-7)\n- [Redis latency](https://redis.io/topics/latency)\n- [Huawei WIFI6 latency](https://e.huawei.com/sg/products/enterprise-networking/wlan/wifi-6)\n- [Verizon 5G latency](https://www.verizon.com/about/our-company/5g/5g-latency)\n- [NGINX added latency](https://www.nginx.com/blog/nginx-controller-api-management-module-vs-kong-performance-comparison/)\n- [AWS Sagemaker latency](https://aws.amazon.com/blogs/machine-learning/load-test-and-optimize-an-amazon-sagemaker-endpoint-using-automatic-scaling/)\n- [AWS Aurora latency](https://aws.amazon.com/blogs/database/using-aurora-to-drive-3x-latency-improvement-for-end-users/)\n- [AWS Cloudfront transfer rates](https://media.amazonwebservices.com/FS_WP_AWS_CDN_CloudFront.pdf)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikolaydubina%2Fgo-ml-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnikolaydubina%2Fgo-ml-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikolaydubina%2Fgo-ml-benchmarks/lists"}