{"id":16753412,"url":"https://github.com/thbar/kiba-common","last_synced_at":"2025-08-13T06:33:26.000Z","repository":{"id":19799383,"uuid":"87941359","full_name":"thbar/kiba-common","owner":"thbar","description":"Commonly used components for Kiba ETL","archived":false,"fork":false,"pushed_at":"2024-06-13T21:16:16.000Z","size":80,"stargazers_count":36,"open_issues_count":2,"forks_count":6,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-14T02:50:05.255Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://www.kiba-etl.org/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thbar.png","metadata":{"files":{"readme":"README.md","changelog":"Changes.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"thbar"}},"created_at":"2017-04-11T14:05:37.000Z","updated_at":"2024-05-02T14:39:37.000Z","dependencies_parsed_at":"2024-06-21T17:53:37.521Z","dependency_job_id":null,"html_url":"https://github.com/thbar/kiba-common","commit_stats":{"total_commits":73,"total_committers":3,"mean_commits":"24.333333333333332","dds":0.0273972602739726,"last_synced_commit":"2aaed8db4578889023cc42700f87ea47b00dd134"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thbar%2Fkiba-common","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thbar%2Fkiba-common/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thbar%2Fkiba-common/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thbar%2Fkiba-common/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thbar","download_url":"https://codeload.github.com/thbar/kiba-common/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229743822,"owners_count":18117460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T02:50:01.065Z","updated_at":"2024-12-14T19:06:22.832Z","avatar_url":"https://github.com/thbar.png","language":"Ruby","funding_links":["https://github.com/sponsors/thbar"],"categories":[],"sub_categories":[],"readme":"Kiba Common is a companion gem to [Kiba](https://github.com/thbar/kiba) and [Kiba Pro](https://github.com/thbar/kiba/blob/master/Pro-Changes.md) in which I'll share commonly used helpers.\n\n[![Gem Version](https://badge.fury.io/rb/kiba-common.svg)](https://badge.fury.io/rb/kiba-common)\n[![Build Status](https://github.com/thbar/kiba-common/actions/workflows/ci.yml/badge.svg)](https://github.com/thbar/kiba-common/actions) \n\n## Usage\n\nAdd `kiba-common` to your `Gemfile`:\n\n```ruby\ngem 'kiba-common'\n```\n\nThen see below for each module usage \u0026 require clause.\n\n## Supported Ruby versions\n\n`kiba-common` currently supports Ruby 2.5+, JRuby 9.2+ and TruffleRuby. See [test matrix](https://github.com/thbar/kiba-common/actions).\n\n\n## Available components\n\n### Kiba::Common::Sources::Enumerable\n\nA source enabling any enumerable (or rather any class implementing `each`) to generate rows for a data pipeline.\n\nUsage:\n\n```ruby\nrequire 'kiba-common/sources/enumerable'\n\n# will generate one row per number between 1 and 100\nsource Kiba::Common::Sources::Enumerable, (1..100)\n```\n\nYou can pass a callable to make sure the evaluation will occur after your [pre-processors](https://github.com/thbar/kiba/wiki/Implementing-pre-and-post-processors) (which is considered a good thing), like this:\n\n```ruby\n# will evaluate the list of files at run time, after \"pre_process\" steps are called\nsource Kiba::Common::Sources::Enumerable, -\u003e { Dir[\"input/*.json\"] }\n```\n\n### Kiba::Common::Transforms::SourceTransformAdapter\n\nLet's say you have a source (e.g. `CSVSource`), which you would like to instantiate for each input row (e.g. a list of filenames). \n\nNormally you'd have to bake file iteration right inside your source. \n\nBut since Kiba v2 introduction of `StreamingRunner`, it is possible for transforms to yield an arbitrary (potentially infinite) amount of rows.\n\nLeveraging that possibility, you can use a `SourceTransformAdapter` to dynamically instantiate the source for each of your input rows.\n\nThis allows you to mix-and-match components in a much more versatile \u0026 powerful way.\n\nRequirements: Kiba v2 with `StreamingRunner` enabled.\n\nUsage:\n\n```ruby\nsource Kiba::Common::Sources::Enumerable, -\u003e { Dir[\"input/*.csv\"] }\n\ntransform do |r|\n  # build up the args that you would normally pass to your source, e.g.\n  # source MyCSV, filename: 'file.csv'\n  # but instead, using the input as a parameter\n  [MyCSVSource, filename: r]\nend\n\n# this will instantiate one source per input row, yielding rows\n# that your source would normally yield\ntransform Kiba::Common::Transforms::SourceTransformAdapter\n```\n\nThis can be used for a wide array of scenarios, including extracting data for N third-party accounts of a same system, with an array of API keys etc.\n\n### Kiba::Common::Transforms::EnumerableExploder\n\nA transform calling `each` on input rows (assuming they are e.g. arrays of sub-rows) and yielding one output row per enumerated element.\n\nRequirements: [Kiba v2](https://github.com/thbar/kiba/releases/tag/v2.0.0) with `StreamingRunner` enabled.\n\nUsage:\n\n```ruby\nrequire 'kiba-common/transforms/enumerable_exploder'\n\ntransform Kiba::Common::Transforms::EnumerableExploder\n```\n\nThis can help if you are reading XML/JSON documents from a source and each input document contains multiple rows that you want to extract.\n\n```ruby\nsource Kiba::Common::Sources::Enumerable, -\u003e { Dir[\"input/*.xml\"] }\n\ntransform { |r| IO.binread(r) }\ntransform { |r| Nokogiri::XML(r) }\n# this will return an array of XML elements\ntransform { |r| r.search('/orders') }\n# this will explode the array (one order per output row)\ntransform Kiba::Common::Transforms::EnumerableExploder\n```\n\nSimilarly, if you have a CSV document as your input:\n\n| po_number  | buyers |\n| ------------- | ------------- |\n| 00001  | John:Mary:Sally  |\n\nand you want to reformat it to the following:\n\n| po_number | buyer |\n|-------------|---------|\n| 00001 | John |\n| 00001 | Mary |\n| 00001 | Sally |\n\nthen you can explode them again with:\n\n```ruby\nsource MyCSVSource, filename: \"input.csv\"\n\ntransform do |row|\n  row.fetch(:buyers).split(':').map do |buyer|\n    { \n      po_number: row.fetch(:po_number),\n      buyer: buyer\n    }\n  end\nend\n\ntransform Kiba::Common::Transforms::EnumerableExploder\n```\n\n### Kiba::Common::Sources::CSV\n\nA CSV source for basic needs (in particular, it doesn't yield row metadata, which are useful in more advanced scenarios).\n\nUse the `csv_options` keyword to control the input format like when using [Ruby CSV class](http://ruby-doc.org/stdlib-2.4.0/libdoc/csv/rdoc/CSV.html#method-c-new).\n\nUsage:\n\n```ruby\nrequire 'kiba-common/sources/csv'\n\n# by defaults, csv_options are empty\nsource Kiba::Common::Sources::CSV, filename: 'input.csv'\n\n# you can provide your own csv_options\nsource Kiba::Common::Sources::CSV, filename: 'input.csv',\n  csv_options: { headers: true, header_converters: :symbol }\n```\n\nNote that the emitted rows are instances of [`CSV::Row`](http://ruby-doc.org/stdlib-2.5.3/libdoc/csv/rdoc/CSV/Row.html), part `Array` and part `Hash`, which retain order of fields and allow duplicates (unlike a regular `Hash`).\n\nIf the rest of your pipeline expects `Hash` rows (like for instance `Kiba::Common::Destinations::CSV`), you'll want to transform the rows to `Hash` instances yourself using [`to_hash`](http://ruby-doc.org/stdlib-2.5.3/libdoc/csv/rdoc/CSV/Row.html#method-i-to_hash). This will \"collapse the row into a simple Hash. Be warned that this discards field order and clobbers duplicate fields.\"\n\n```ruby\ntransform { |r| r.to_hash }\n```\n\n#### Handling multiple input CSV files\n\nYou can process multiple files by chaining the various components available in Kiba Common (see `test/test_integration#test_multiple_csv_inputs` for an actual demo):\n\n```ruby\n# create one row per input filename\nsource Kiba::Common::Sources::Enumerable, -\u003e { Dir[File.join(dir, '*.csv')] }\n\n# out of that row, create configuration for a CSV source\ntransform do |r|\n  [\n    Kiba::Common::Sources::CSV,\n    filename: r,\n    csv_options: { headers: true, header_converters: :symbol }\n  ]\nend\n\n# instantiate \u0026 yield CSV rows for each configuration\ntransform Kiba::Common::Transforms::SourceTransformAdapter\n```\n\nAlternatively, you can wrap this source in your own source like this:\n\n```ruby\nclass MultipleCSVSource\n  def initialize(file_pattern:, csv_options:)\n\n  def each\n    Dir[file_pattern].each do |filename|\n      Kiba::Common::Sources::CSV.new(filename, csv_options).each do |row|\n        yield row\n      end\n    end\n  end\nend\n```\n\n### Kiba::Common::Destinations::CSV\n\nA way to dump `Hash` rows as CSV.\n\nAll rows are expected to have the exact same set of keys as the first row.\n\nThe headers will be the first row keys unless you pass an array of keys via `headers`.\n\nAll keys are mandatory (although they can have a `nil` value).\n\nUse the `csv_options` keyword to control the output format like when using [Ruby CSV class](http://ruby-doc.org/stdlib-2.4.0/libdoc/csv/rdoc/CSV.html#method-c-new).\n\nUsage:\n\n```ruby\nrequire 'kiba-common/destinations/csv'\n\n# by default, the headers will be picked from the first row:\ndestination Kiba::Common::Destinations::CSV,\n  filename: 'output.csv'\n\n# if you need a different separator:\ndestination Kiba::Common::Destinations::CSV,\n  filename: 'output.csv',\n  csv_options: { col_sep: ';' }\n  \n# to enforce a specific set of headers:\ndestination Kiba::Common::Destinations::CSV,\n  filename: 'output.csv',\n  headers: [:field, :other_field]\n```\n\n### Kiba::Common::Destinations::Lambda\n\nAt times, it can be convenient to use a block form for a destination (pretty much like Kiba's built-in \"block transform\"), especially for one-off scripts.\n\nThe Lambda destination is there for that purpose.\n\nExample use:\n\n```ruby\nrequire 'kiba-common/destinations/lambda'\n\ndestination Kiba::Common::Destinations::Lambda,\n  # called at destination instantiation time (once)\n  on_init: -\u003e { ... },\n  # called for each row\n  on_write: -\u003e (row) { ... },\n  # called after all the rows have been written\n  on_close: -\u003e { ... }\n```\n\nEach \"callback\" (e.g. `on_init`) is optional.\n\nThe callback code can refer to scope variables or instance variables you may have declared above.\n\n### Kiba::Common::DSLExtensions::Logger\n\nA simple logging facility.\n\nUsage:\n\n```ruby\nrequire 'kiba-common/dsl_extensions/logger'\nextend Kiba::Common::DSLExtensions::Logger\n\npre_process do\n  logger.info \"pre_process is running!\"\nend\n```\n\nBy default the logger will output to `STDOUT`.\n\nYou can customize that behaviour by setting the logger:\n\n```ruby\nrequire 'kiba-common/dsl_extensions/logger'\nextend Kiba::Common::DSLExtensions::Logger\n\nlogger = Logger.new(xxx)\n```\n\n### Kiba::Common::DSLExtensions::ShowMe\n\nA way to color-dump rows on the screen, useful during development when you are inspecting the data (requires either the `amazing_print` or the `awesome_print` gem - you will have to require one or the other manually yourself, in order to ensure compatibility between both gems).\n\nUsage:\n\n```ruby\nrequire 'kiba-common/dsl_extensions/show_me'\nextend Kiba::Common::DSLExtensions::ShowMe\n\nsource MySource\ntransform MyTransform\nshow_me! # will color-print the row at this step of the pipeline\n```\n\nYou can also pre-process the data to only show specific parts of the row:\n\n```ruby\nrequire 'active_support/core_ext/hash/except'\n\nshow_me! { |r| r.except(:some_noisy_field) }\n```\n\n## Contributing \u0026 Legal\n\nSee [LICENSE](LICENSE) for license.\n\n(agreement below borrowed from Sidekiq Legal)\n\nBy submitting a Pull Request, you disavow any rights or claims to any changes submitted to the Kiba Common project and assign the copyright of those changes to LoGeek SARL.\n\nIf you cannot or do not want to reassign those rights (your employment contract for your employer may not allow this), you should not submit a PR. Open an issue and someone else can do the work.\n\nThis is a legal way of saying \"If you submit a PR to us, that code becomes ours\". 99.9% of the time that's what you intend anyways; we hope it doesn't scare you away from contributing.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthbar%2Fkiba-common","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthbar%2Fkiba-common","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthbar%2Fkiba-common/lists"}