{"id":13719007,"url":"https://github.com/Martin-Nyaga/fast_statistics","last_synced_at":"2025-05-07T11:30:31.366Z","repository":{"id":45324722,"uuid":"316867365","full_name":"Martin-Nyaga/fast_statistics","owner":"Martin-Nyaga","description":"Fast computation of descriptive statistics in ruby using native code and SIMD","archived":false,"fork":false,"pushed_at":"2023-10-20T00:38:33.000Z","size":117,"stargazers_count":60,"open_issues_count":3,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-26T18:47:33.812Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Martin-Nyaga.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-11-29T03:21:02.000Z","updated_at":"2023-11-02T08:45:29.000Z","dependencies_parsed_at":"2024-01-07T21:13:30.700Z","dependency_job_id":"f3151cfb-2650-41de-97da-afdc75744729","html_url":"https://github.com/Martin-Nyaga/fast_statistics","commit_stats":{"total_commits":40,"total_committers":1,"mean_commits":40.0,"dds":0.0,"last_synced_commit":"0878ccd7b80a1ad6af1a13a4ba8ff95ddf76cc3c"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Martin-Nyaga%2Ffast_statistics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Martin-Nyaga%2Ffast_statistics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Martin-Nyaga%2Ffast_statistics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Martin-Nyaga%2Ffast_statistics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Martin-Nyaga","download_url":"https://codeload.github.com/Martin-Nyaga/fast_statistics/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224592315,"owners_count":17337073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T01:00:40.937Z","updated_at":"2024-11-14T08:31:02.653Z","avatar_url":"https://github.com/Martin-Nyaga.png","language":"Ruby","funding_links":[],"categories":["Statistics"],"sub_categories":[],"readme":"# Fast Statistics :rocket:\n![Build Status](https://travis-ci.com/Martin-Nyaga/fast_statistics.svg?branch=master)\n\nA high performance native ruby extension (written in C++) for computation of\ndescriptive statistics.\n\n## Overview\nThis gem provides fast computation of descriptive statistics (min, max, mean,\nmedian, 1st and 3rd quartiles, population standard deviation) for a multivariate\ndataset (represented as a 2D array) in ruby.\n\nIt is **~11x** faster than an optimal algorithm in hand-written ruby, and\n**~4.7x** faster than the next fastest available ruby gem or native extension\n(see [benchmarks](#benchmarks) below).\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'fast_statistics'\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install fast_statistics\n\n## Usage\n\nGiven you have some multivariate (2-dimensional) data:\n```ruby\ndata = [\n  [0.6269, 0.3783, 0.1477, 0.2374],\n  [0.4209, 0.1055, 0.8000, 0.2023],\n  [0.1124, 0.1021, 0.1936, 0.8566],\n  [0.6454, 0.5362, 0.4567, 0.8309],\n  [0.4828, 0.1572, 0.5706, 0.4085],\n  [0.5594, 0.0979, 0.4078, 0.5885],\n  [0.8659, 0.5346, 0.5566, 0.6166],\n  [0.7256, 0.5841, 0.8546, 0.3918]\n]\n```\n\nYou can compute descriptive statistics for all the inner arrays as follows:\n\n```ruby\nrequire \"fast_statistics\"\n\nFastStatistics::Array2D.new(data).descriptive_statistics\n# Result: \n#\n# [{:min=\u003e0.1477,\n#   :max=\u003e0.6269,\n#   :mean=\u003e0.347575,\n#   :median=\u003e0.30785,\n#   :q1=\u003e0.214975,\n#   :q3=\u003e0.44045,\n#   :standard_deviation=\u003e0.18100761551658537},\n#  {:min=\u003e0.1055,\n#   :max=\u003e0.8,\n#   :mean=\u003e0.38217500000000004,\n#   :median=\u003e0.3116,\n#   :q1=\u003e0.1781,\n#   :q3=\u003e0.515675,\n#   :standard_deviation=\u003e0.26691825878909076},\n#  ...,\n#  {:min=\u003e0.3918,\n#   :max=\u003e0.8546,\n#   :mean=\u003e0.639025,\n#   :median=\u003e0.6548499999999999,\n#   :q1=\u003e0.536025,\n#   :q3=\u003e0.75785,\n#   :standard_deviation=\u003e0.1718318709523935}]\n```\n\n## Benchmarks\n\nSome alternatives compared are:\n- [descriptive_statistics](https://github.com/thirtysixthspan/descriptive_statistics)\n- [ruby-native-statistics](https://github.com/corybuecker/ruby-native-statistics)\n- [Numo::NArray](https://github.com/ruby-numo/numo-narray)\n- Hand-written ruby (using the same algorithm implemented in C++ in this gem)\n\nYou can reivew the benchmark implementations at `benchmark/benchmark.rb` and run the\nbenchmark with `rake benchmark`. \n\nResults:\n```\nComparing calculated statistics with 10 values for 8 variables...\nTest passed, results are equal to 6 decimal places!\n\nBenchmarking with 100,000 values for 12 variables...\nWarming up --------------------------------------\ndescriptive_statistics   1.000  i/100ms\n           Custom ruby   1.000  i/100ms\n                narray   1.000  i/100ms\nruby_native_statistics   1.000  i/100ms\n        FastStatistics   3.000  i/100ms\nCalculating -------------------------------------\ndescriptive_statistics   0.473  (± 0.0%) i/s -      3.000  in   6.354555s\n           Custom ruby   2.518  (± 0.0%) i/s -     13.000  in   5.169084s\n                narray   4.231  (± 0.0%) i/s -     22.000  in   5.210299s\nruby_native_statistics   5.962  (± 0.0%) i/s -     30.000  in   5.041869s\n        FastStatistics   28.417  (±10.6%) i/s -    141.000  in   5.012229s\n\nComparison:\n        FastStatistics:   28.4 i/s\nruby_native_statistics:    6.0 i/s - 4.77x  (± 0.00) slower\n                narray:    4.2 i/s - 6.72x  (± 0.00) slower\n           Custom ruby:    2.5 i/s - 11.29x  (± 0.00) slower\ndescriptive_statistics:    0.5 i/s - 60.09x  (± 0.00) slower\n```\n\n## Background \u0026 Implementation \n\nThe inspiration for this gem was a use-case in an analytics ruby application,\nwhere we frequently had to compute descriptive statistics for fairly large\nmultivariate datasets. Calculations in ruby were not fast enough, so I\nfirst explored performing the computations natively in [this\nrepository](https://github.com/Martin-Nyaga/ruby-ffi-simd). The results were\npromising, so I decided to package it as a ruby gem.\n\nI've now ran this in production for some time, and I'm quite happy with it. Feel\nfree to let me know in [this discussion\nthread](https://github.com/Martin-Nyaga/fast_statistics/discussions/1) if you\nuse it, or open an issue if you run into any problems.\n\n### How is the performance achieved? \nThe following factors combined help this gem achieve high performance compared\nto available native alternatives and hand-written computations in ruby:\n\n- It is written in C++ and so can leverage the speed of native execution. \n- It minimises the number of operations by calculating the statistics in as few\n  operations as possible (1 sort + 2 loops). Most native alternatives don't\n  provide a built in way to get all these statistics at once. Instead, they only\n  provide APIs where you make single calls for individual statistics. Through\n  such an API, building this set of summary statistics typically ends up looping\n  through the data more times than is necessary.\n- This gem uses explicit 128-bit-wide SIMD intrinsics (on platforms where they\n  are available) to parallelize computations for 2 variables at the same time\n  where possible, giving an additional speed advantage while still being single\n  threaded.\n\n### Limitations of the current implementation\nThe speed gains notwithstanding, there are some limitations in the current implementation:\n- The variables in the 2D array must all have the same number of data points\n  (inner arrays must have the same length) and contain only numbers (i.e. no\n  `nil` awareness is present).\n- There is currently no API to calculate single statistics (although this may be\n  made available in the future).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at\nhttps://github.com/Martin-Nyaga/fast_statistics.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT\nLicense](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMartin-Nyaga%2Ffast_statistics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMartin-Nyaga%2Ffast_statistics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMartin-Nyaga%2Ffast_statistics/lists"}