{"id":13718718,"url":"https://github.com/ondra-m/ruby-spark","last_synced_at":"2025-04-05T08:08:49.594Z","repository":{"id":26428709,"uuid":"29879192","full_name":"ondra-m/ruby-spark","owner":"ondra-m","description":"Ruby wrapper for Apache Spark","archived":false,"fork":false,"pushed_at":"2017-08-31T10:27:59.000Z","size":650,"stargazers_count":225,"open_issues_count":23,"forks_count":29,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-04-25T19:22:00.836Z","etag":null,"topics":["distributed","rdd","ruby","ruby-spark","spark"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ondra-m.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-01-26T20:07:38.000Z","updated_at":"2024-04-21T14:14:02.000Z","dependencies_parsed_at":"2022-08-21T01:20:15.180Z","dependency_job_id":null,"html_url":"https://github.com/ondra-m/ruby-spark","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondra-m%2Fruby-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondra-m%2Fruby-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondra-m%2Fruby-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondra-m%2Fruby-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ondra-m","download_url":"https://codeload.github.com/ondra-m/ruby-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247305935,"owners_count":20917208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed","rdd","ruby","ruby-spark","spark"],"created_at":"2024-08-03T01:00:36.474Z","updated_at":"2025-04-05T08:08:49.566Z","avatar_url":"https://github.com/ondra-m.png","language":"Ruby","funding_links":[],"categories":["Distributed Computing","NLP Pipeline Subtasks"],"sub_categories":["Pipeline Generation"],"readme":"# Ruby-Spark [![Build Status](https://travis-ci.org/ondra-m/ruby-spark.svg?branch=master)](https://travis-ci.org/ondra-m/ruby-spark)\n\nApache Spark™ is a fast and general engine for large-scale data processing.\n\nThis Gem allows the use Spark functionality on Ruby.\n\n\u003e Word count in Spark's Ruby API\n\n```ruby\nfile = spark.text_file(\"hdfs://...\")\n\nfile.flat_map(:split)\n    .map(lambda{|word| [word, 1]})\n    .reduce_by_key(lambda{|a, b| a+b})\n```\n\n- [Apache Spark](http://spark.apache.org)\n- [Wiki](https://github.com/ondra-m/ruby-spark/wiki)\n- [Rubydoc](http://www.rubydoc.info/gems/ruby-spark)\n\n## Installation\n\n### Requirments\n\n- Java 7+\n- Ruby 2+\n- wget or curl\n- MRI or JRuby\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'ruby-spark'\n```\n\nAnd then execute:\n\n```\n$ bundle\n```\n\nOr install it yourself as:\n\n```\n$ gem install ruby-spark\n```\n\nRun `rake compile` if you are using gem from local filesystem.\n\n### Build Apache Spark\n\nThis command will download Spark and build extensions for this gem ([SBT](ext/spark/build.sbt) is used for compiling). For more informations check [wiki](https://github.com/ondra-m/ruby-spark/wiki/Installation). Jars will be stored at you HOME directory.\n\n```\n$ ruby-spark build\n```\n\n\n## Usage\n\nYou can use Ruby Spark via interactive shell (Pry is used)\n\n```\n$ ruby-spark shell\n```\n\nOr on existing project.\n\nIf you want configure Spark first. See [configurations](https://github.com/ondra-m/ruby-spark/wiki/Configuration) for more details.\n\n```ruby\nrequire 'ruby-spark'\n\n# Configuration\nSpark.config do\n   set_app_name \"RubySpark\"\n   set 'spark.ruby.serializer', 'oj'\n   set 'spark.ruby.serializer.batch_size', 100\nend\n\n# Start Apache Spark\nSpark.start\n\n# Context reference\nSpark.sc\n```\n\nFinally, to stop the cluster. On the shell is Spark stopped automatically when environment exit.\n\n```ruby\nSpark.stop\n```\nAfter first use, global configuration is created at **~/.ruby-spark.conf**. There can be specified properties for Spark and RubySpark.\n\n\n\n## Creating RDD (a new collection)\n\nSingle text file:\n\n```ruby\nrdd = sc.text_file(FILE, workers_num, serializer=nil)\n```\n\nAll files on directory:\n\n```ruby\nrdd = sc.whole_text_files(DIRECTORY, workers_num, serializer=nil)\n```\n\nDirect uploading structures from ruby:\n\n```ruby\nrdd = sc.parallelize([1,2,3,4,5], workers_num, serializer=nil)\nrdd = sc.parallelize(1..5, workers_num, serializer=nil)\n```\n\nThere is 2 conditions:\n1. choosen serializer must be able to serialize it\n2. data must be iterable\n\nIf you do not specified serializer -\u003e default is used (defined from spark.ruby.serializer.* options). [Check this](https://github.com/ondra-m/ruby-spark/wiki/Loading-data#custom-serializer) if you want create custom serializer.\n\n## Operations\n\nAll operations can be divided into 2 groups:\n\n- **Transformations:** append new operation to current RDD and return new\n- **Actions:** add operation and start calculations\n\nMore informations:\n\n- [Wiki page](https://github.com/ondra-m/ruby-spark/wiki/RDD)\n- [Rubydoc](http://www.rubydoc.info/github/ondra-m/ruby-spark/master/Spark/RDD)\n- [rdd.rb](https://github.com/ondra-m/ruby-spark/blob/master/lib/spark/rdd.rb)\n\nYou can also check official Spark documentation. First make sure that method is implemented here.\n\n- [Transformations](http://spark.apache.org/docs/latest/programming-guide.html#transformations)\n- [Actions](http://spark.apache.org/docs/latest/programming-guide.html#actions)\n\n#### Transformations\n\n\u003cdl\u003e          \n  \u003cdt\u003e\u003ccode\u003erdd.map(function)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn a new RDD by applying a function to all elements of this RDD.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.flat_map(function)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn a new RDD by first applying a function to all elements of this RDD, and then flattening the results.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.map_partitions(function)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn a new RDD by applying a function to each partition of this RDD.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.filter(function)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn a new RDD containing only the elements that satisfy a predicate.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.cartesian(other)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements `(a, b)` where `a` is in `self` and `b` is in `other`.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.intersection(other)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn the intersection of this RDD and another one. The output will not contain any duplicate elements, even if the input RDDs did.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.sample(with_replacement, fraction, seed)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn a sampled subset of this RDD. Operations are base on Poisson and Uniform distributions.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.group_by_key(num_partitions)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eGroup the values for each key in the RDD into a single sequence.\u003c/dd\u003e\n  \n  \u003cdt\u003e\u003ca href=\"http://www.rubydoc.info/gems/ruby-spark/Spark/RDD\" target=\"_blank\"\u003e\u003ccode\u003e...many more...\u003c/code\u003e\u003c/a\u003e\u003c/dt\u003e\n  \u003cdd\u003e\u003c/dd\u003e\n\u003c/dl\u003e\n\n\n#### Actions\n\n\u003cdl\u003e \n  \u003cdt\u003e\u003ccode\u003erdd.take(count)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eTake the first num elements of the RDD.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.reduce(function)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReduces the elements of this RDD using the specified lambda or method.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.aggregate(zero_value, seq_op, comb_op)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eAggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral “zero value”.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.histogram(buckets)\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eCompute a histogram using the provided buckets.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ccode\u003erdd.collect\u003c/code\u003e\u003c/dt\u003e\n  \u003cdd\u003eReturn an array that contains all of the elements in this RDD.\u003c/dd\u003e\n\n  \u003cdt\u003e\u003ca href=\"http://www.rubydoc.info/gems/ruby-spark/Spark/RDD\" target=\"_blank\"\u003e\u003ccode\u003e...many more...\u003c/code\u003e\u003c/a\u003e\u003c/dt\u003e\n  \u003cdd\u003e\u003c/dd\u003e\n\u003c/dl\u003e\n\n\n## Examples\n\n##### Basic methods\n\n```ruby\n# Every batch will be serialized by Marshal and will have size 10\nser = Spark::Serializer.build('batched(marshal, 10)')\n\n# Range 0..100, 2 workers, custom serializer\nrdd = Spark.sc.parallelize(0..100, 2, ser)\n\n\n# Take first 5 items\nrdd.take(5)\n# =\u003e [0, 1, 2, 3, 4]\n\n\n# Numbers reducing\nrdd.reduce(lambda{|sum, x| sum+x})\nrdd.reduce(:+)\nrdd.sum\n# =\u003e 5050\n\n\n# Reducing with zero items\nseq = lambda{|x,y| x+y}\ncom = lambda{|x,y| x*y}\nrdd.aggregate(1, seq, com)\n# 1. Every workers adds numbers\n#    =\u003e [1226, 3826]\n# 2. Results are multiplied\n#    =\u003e 4690676\n\n\n# Statistic method\nrdd.stats\n# =\u003e StatCounter: (count, mean, max, min, variance,\n#                  sample_variance, stdev, sample_stdev)\n\n\n# Compute a histogram using the provided buckets.\nrdd.histogram(2)\n# =\u003e [[0.0, 50.0, 100], [50, 51]]\n\n\n# Mapping\nrdd.map(lambda {|x| x*2}).collect\n# =\u003e [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, ...]\nrdd.map(:to_f).collect\n# =\u003e [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, ...]\n\n\n# Mapping the whole collection\nrdd.map_partitions(lambda{|part| part.reduce(:+)}).collect\n# =\u003e [1225, 3825]\n\n\n# Selecting\nrdd.filter(lambda{|x| x.even?}).collect\n# =\u003e [0, 2, 4, 6, 8, 10, 12, 14, 16, ...]\n\n\n# Sampling\nrdd.sample(true, 10).collect\n# =\u003e [3, 36, 40, 54, 58, 82, 86, 95, 98]\n\n\n# Sampling X items\nrdd.take_sample(true, 10)\n# =\u003e [53, 87, 71, 74, 18, 75, 55, 94, 46, 32]\n\n\n# Using external process\nrdd.pipe('cat', \"awk '{print $1*10}'\")\n# =\u003e [\"0\", \"10\", \"20\", \"30\", \"40\", \"50\", ...]\n```\n\n##### Words count using methods\n\n```ruby\n# Content:\n# \"first line\"\n# \"second line\"\nrdd = sc.text_file(PATH)\n\n# [\"first\", \"line\", \"second\", \"line\"]\nrdd = rdd.flat_map(lambda{|line| line.split})\n\n# [[\"first\", 1], [\"line\", 1], [\"second\", 1], [\"line\", 1]]\nrdd = rdd.map(lambda{|word| [word, 1]})\n\n# [[\"first\", 1], [\"line\", 2], [\"second\", 1]]\nrdd = rdd.reduce_by_key(lambda{|a, b| a+b})\n\n# {\"first\"=\u003e1, \"line\"=\u003e2, \"second\"=\u003e1}\nrdd.collect_as_hash\n```\n\n##### Estimating PI with a custom serializer\n\n```ruby\nslices = 3\nn = 100000 * slices\n\ndef map(_)\n  x = rand * 2 - 1\n  y = rand * 2 - 1\n\n  if x**2 + y**2 \u003c 1\n    return 1\n  else\n    return 0\n  end\nend\n\nrdd = Spark.context.parallelize(1..n, slices, serializer: 'oj')\nrdd = rdd.map(method(:map))\n\nputs 'Pi is roughly %f' % (4.0 * rdd.sum / n)\n```\n\n##### Estimating PI\n\n```ruby\nrdd = sc.parallelize([10_000], 1)\nrdd = rdd.add_library('bigdecimal/math')\nrdd = rdd.map(lambda{|x| BigMath.PI(x)})\nrdd.collect # =\u003e #\u003cBigDecimal, '0.31415926...'\u003e\n```\n\n### Mllib (Machine Learning Library)\n\nMllib functions are using Spark's Machine Learning Library. Ruby objects are serialized and deserialized in Java so you cannot use custom classes. Supported are primitive types such as string or integers.\n\nAll supported methods/models:\n\n- [Rubydoc / Mllib](http://www.rubydoc.info/github/ondra-m/ruby-spark/Spark/Mllib)\n- [Github / Mllib](https://github.com/ondra-m/ruby-spark/tree/master/lib/spark/mllib)\n\n##### Linear regression\n\n```ruby\n# Import Mllib classes into Object\n# Otherwise are accessible via Spark::Mllib::LinearRegressionWithSGD\nSpark::Mllib.import(Object)\n\n# Training data\ndata = [\n  LabeledPoint.new(0.0, [0.0]),\n  LabeledPoint.new(1.0, [1.0]),\n  LabeledPoint.new(3.0, [2.0]),\n  LabeledPoint.new(2.0, [3.0])\n]\n\n# Train a model\nlrm = LinearRegressionWithSGD.train(sc.parallelize(data), initial_weights: [1.0])\n\nlrm.predict([0.0])\n```\n\n##### K-Mean\n\n```ruby\nSpark::Mllib.import\n\n# Dense vectors\ndata = [\n  DenseVector.new([0.0,0.0]),\n  DenseVector.new([1.0,1.0]),\n  DenseVector.new([9.0,8.0]),\n  DenseVector.new([8.0,9.0])\n]\n\nmodel = KMeans.train(sc.parallelize(data), 2)\n\nmodel.predict([0.0, 0.0]) == model.predict([1.0, 1.0])\n# =\u003e true\nmodel.predict([8.0, 9.0]) == model.predict([9.0, 8.0])\n# =\u003e true\n```\n\n## Benchmarks\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondra-m%2Fruby-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fondra-m%2Fruby-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondra-m%2Fruby-spark/lists"}