{"id":17036588,"url":"https://github.com/sgrondin/tdigest","last_synced_at":"2025-04-12T12:52:42.362Z","repository":{"id":66076082,"uuid":"275415713","full_name":"SGrondin/tdigest","owner":"SGrondin","description":"OCaml implementation of the T-Digest algorithm","archived":false,"fork":false,"pushed_at":"2024-02-25T16:12:22.000Z","size":107,"stargazers_count":26,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-04T18:45:10.157Z","etag":null,"topics":["ocaml","statistics","tdigest"],"latest_commit_sha":null,"homepage":"","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SGrondin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-27T16:56:09.000Z","updated_at":"2024-01-23T21:05:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"dac9f30c-bae2-4dcf-ae6f-91b826ce9aa2","html_url":"https://github.com/SGrondin/tdigest","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SGrondin%2Ftdigest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SGrondin%2Ftdigest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SGrondin%2Ftdigest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SGrondin%2Ftdigest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SGrondin","download_url":"https://codeload.github.com/SGrondin/tdigest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248571627,"owners_count":21126520,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ocaml","statistics","tdigest"],"created_at":"2024-10-14T08:51:02.074Z","updated_at":"2025-04-12T12:52:42.340Z","avatar_url":"https://github.com/SGrondin.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"Tdigest\n=======\n\nOCaml implementation of the T-Digest algorithm.\n\n```ocaml\nlet td =\n  Tdigest.create ()\n  |\u003e Tdigest.add_list [ 10.0; 11.0; 12.0; 13.0 ]\nin\n\nTdigest.percentiles td [ 0.; 0.25; 0.5; 0.75; 1. ]\n(* [ Some 10; Some 10.5; Some 11.5; Some 12.5; Some 13 ] *)\n\nTdigest.p_ranks td [ 9.; 10.; 11.; 12.; 13.; 14. ]\n(* [ Some 0; Some 0.125; Some 0.375; Some 0.625; Some 0.875; Some 1 ] *)\n```\n\nThe T-Digest is a data structure and algorithm for constructing an approximate distribution for a collection of real numbers presented as a stream.\n\nThe T-Digest can estimate percentiles or quantiles extremely accurately even at the tails, while using a fraction of the space of the original data.\n\nA median of medians is not equal to the median of the whole dataset. Percentiles are critical measures that are expensive to compute due to their requirement of having the entire **sorted** dataset present in one place. These downsides are addressed by using the T-Digest.\n\nA T-Digest is concatenable, making it a good fit for distributed systems. The internal state of a T-Digest can be exported as a binary string, and the concatenation of any number of those strings can then be imported to form a new T-Digest.\n\n```ocaml\nlet combined = Tdigest.merge [ td1; td2; td3 ] in\n```\n\nA T-Digest's state can be stored in a database `VARCHAR`/`TEXT` column and multiple such states can be merged by concatenating strings:\n```sql\n-- Combine multiple states in the database\nSELECT\n  STRING_AGG(M.tdigest_state) AS concat_state\nFROM my_table AS M\n```\n```ocaml\n(* Then load this combined state into a single T-Digest *)\nlet combined = Tdigest.of_string concat_state in\n```\n\nLinks:\n- [A simple overview of the T-Digest](https://dataorigami.net/blogs/napkin-folding/19055451-percentile-and-quantile-estimation-of-big-data-the-t-digest)\n- [A walkthrough of the algorithm by its creator](https://mapr.com/blog/better-anomaly-detection-t-digest-whiteboard-walkthrough/)\n- [The white paper](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf)\n\nThis library started off as a port of [Will Welch's JavaScript implementation](https://github.com/welch/tdigest), down to the unit tests. However some modifications have been made to adapt it to OCaml, the most important one being immutability. As such, almost every function in the `Tdigest` module return a new `Tdigest.t`, including \"reading\" ones since they may trigger intermediate computations worth caching.\n\n## Usage\n\nThe API is well documented [here](https://github.com/SGrondin/tdigest/blob/master/src/tdigest.mli).\n\n```sh\nopam install tdigest\n```\n\n### Marshal\n\nThe `Tdigest.t` type cannot be marshalled.\n\nUse the functions in `Tdigest.Marshallable` if your application requires marshalling a T-Digest data structure. Note that `Tdigest.Marshallable.t` is approximately 5 times slower than `Tdigest.t`.\n\n## Performance\n\nOn an ancient 2015 MacBook Pro, this implementation can incorporate 1,000,000 random floating points in just 770ms.\n\nExporting and importing state (`to_string`/`of_string`) is cheap.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsgrondin%2Ftdigest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsgrondin%2Ftdigest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsgrondin%2Ftdigest/lists"}