{"id":27197703,"url":"https://github.com/preaction/etl-yertl","last_synced_at":"2025-04-09T20:29:25.373Z","repository":{"id":16334577,"uuid":"19084239","full_name":"preaction/ETL-Yertl","owner":"preaction","description":"ETL With a Shell","archived":false,"fork":false,"pushed_at":"2020-07-28T02:29:36.000Z","size":812,"stargazers_count":27,"open_issues_count":52,"forks_count":4,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-06-21T16:47:04.855Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://preaction.me/yertl","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/preaction.png","metadata":{"files":{"readme":"README.mkdn","changelog":"CHANGES","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-23T20:21:09.000Z","updated_at":"2021-03-16T18:07:20.000Z","dependencies_parsed_at":"2022-07-16T21:46:10.116Z","dependency_job_id":null,"html_url":"https://github.com/preaction/ETL-Yertl","commit_stats":null,"previous_names":[],"tags_count":44,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preaction%2FETL-Yertl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preaction%2FETL-Yertl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preaction%2FETL-Yertl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preaction%2FETL-Yertl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/preaction","download_url":"https://codeload.github.com/preaction/ETL-Yertl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248106804,"owners_count":21048798,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T20:29:24.555Z","updated_at":"2025-04-09T20:29:25.346Z","avatar_url":"https://github.com/preaction.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NAME\n\nETL::Yertl - ETL with a Shell\n\n# VERSION\n\nversion 0.044\n\n# STATUS\n\n\u003ca href=\"https://travis-ci.org/preaction/ETL-Yertl\"\u003e\u003cimg src=\"https://travis-ci.org/preaction/ETL-Yertl.svg?branch=master\"\u003e\u003c/a\u003e\u003ca href=\"https://coveralls.io/r/preaction/ETL-Yertl\"\u003e\u003cimg src=\"https://coveralls.io/repos/preaction/ETL-Yertl/badge.png\" alt=\"Coverage Status\" /\u003e\u003c/a\u003e\n\n# SYNOPSIS\n\n    ### On a shell...\n    # Convert file to Yertl's format\n    $ yfrom csv file.csv \u003ework.yml\n    $ yfrom json file.json \u003ework.yml\n\n    # Convert file to output format\n    $ yto csv work.yml\n    $ yto json work.yml\n\n    # Parse HTTP logs into documents\n    $ ygrok '%{LOG.HTTP_COMMON}' httpd.log\n\n    # Read data from a database\n    $ ysql db_name 'SELECT * FROM employee'\n\n    # Write data to a database\n    $ ysql db_name 'INSERT INTO employee ( id, name ) VALUES ( $.id, $.name )'\n\n    ### In Perl...\n    use ETL::Yertl;\n\n    # Give everyone a 5% raise\n    my $xform = file( '\u003c', 'employees.yaml' )\n              | transform( sub { $_-\u003e{salary} *= 1.05 } )\n              \u003e\u003e stdout;\n    $xform-\u003erun;\n\n# DESCRIPTION\n\nETL::Yertl is an ETL ([Extract, Transform,\nLoad](https://en.wikipedia.org/wiki/Extract,_transform,_load)) for shells. It is\ndesigned to accept data from multiple formats (CSV, JSON), manipulate them\nusing simple tools, and then convert them to an output format.\n\nYertl will have tools for:\n\n- Extracting data from databases (MySQL, Postgres, MongoDB)\n- Loading data into databases\n- Extracting data from web services\n- Writing data to web services\n- Distributing data through messaging APIs (ZeroMQ)\n\n# SUBROUTINES\n\n## stdin\n\n    my $stdin = stdin( %args );\n\nGet a [ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) object for standard input. `%args` is a list\nof key/value pairs passed to [\"new\" in ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream#new). Useful keys are:\n\n- format\n\n    Specify the format that standard input is. Defaults to `yaml` or the value\n    of `YERTL_FORMAT` (see [\"get\\_default\" in ETL::Yertl::Format](https://metacpan.org/pod/ETL::Yertl::Format#get_default).\n\n## stdout\n\n    my $stdout = stdout( %args );\n\nGet a [ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) object for standard output. `%args` is a list\nof key/value pairs passed to [\"new\" in ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream#new). Useful keys are:\n\n- format\n\n    Specify the format that standard input is. Defaults to `yaml` or the value\n    of `YERTL_FORMAT` (see [\"get\\_default\" in ETL::Yertl::Format](https://metacpan.org/pod/ETL::Yertl::Format#get_default).\n\n- autoflush\n\n    Immediately write documents to standard output to improve\n    responsiveness, instead of queuing them for later writes for efficiency.\n    This defaults to `1` (on). Set it to `0` to turn autoflush off.\n\n## transform\n\n    my $xform = transform( sub { ... } );\n    my $xform = transform( 'Local::Transform::Class' =\u003e @args );\n\nCreate a new [ETL::Yertl::Transform](https://metacpan.org/pod/ETL::Yertl::Transform) object, passing in either a subref\nto transform documents, or a class to instantiate and arguments to pass\nto its constructor.\n\nThe subref will receive two arguments: The [ETL::Yertl::Transform](https://metacpan.org/pod/ETL::Yertl::Transform)\nobject and the document to transform. `$_` will also be set to the\ndocument to transform.  The subref should return the transformed\ndocument (either a new document, or the existing document after being\nmodified).\n\nIf given a transform class, it should inherit from\n[ETL::Yertl::Transform](https://metacpan.org/pod/ETL::Yertl::Transform). The class will be loaded and an object\ninstantiated using the `@args`.\n\n## file\n\n    my $stream = file( $mode, $path, %args );\n\nCreate a [ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) object for the given `$path`.\n`$mode` should be one of `\u003c` for reading or `\u003e` for writing.\n`%args` are additional arguments to pass to the\n[ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) constructor. Useful keys are:\n\n- format\n\n    Specify the format that standard input is. Defaults to `yaml` or the value\n    of `YERTL_FORMAT` (see [\"get\\_default\" in ETL::Yertl::Format](https://metacpan.org/pod/ETL::Yertl::Format#get_default).\n\n## yq\n\n    my $xform = yq( $filter );\n\nCreate a [ETL::Yertl::Transform::Yq](https://metacpan.org/pod/ETL::Yertl::Transform::Yq) object with the given filter. See\n[\"SYNTAX\" in yq](https://metacpan.org/pod/yq#SYNTAX) for full filter syntax.\n\n## loop\n\n    my $loop = loop();\n\nGet the [IO::Async::Loop](https://metacpan.org/pod/IO::Async::Loop) singleton. Use this to add other [IO::Async](https://metacpan.org/pod/IO::Async) objects\nto a larger program. This is not needed for simple Yertl streams, and is mostly\nused internally.\n\nThis is not exported by default. You can import it using `use ETL::Yertl 'loop'`.\n\n# SEE ALSO\n\n- [http://preaction.me/yertl](http://preaction.me/yertl)\n\n    The Yertl home page.\n\n## Yertl Tools\n\n- [yfrom](https://metacpan.org/pod/yfrom)\n\n    Convert incoming data (CSV, JSON) to Yertl documents.\n\n- [yto](https://metacpan.org/pod/yto)\n\n    Convert Yertl documents into another format (CSV, JSON).\n\n- [ygrok](https://metacpan.org/pod/ygrok)\n\n    Parse lines of text into Yertl documents.\n\n- [ysql](https://metacpan.org/pod/ysql)\n\n    Read/write documents from SQL databases.\n\n- [yq](https://metacpan.org/pod/yq)\n\n    A powerful mini-language for munging and filtering.\n\n## Other Tools\n\nHere are some other tools that can be used with Yertl\n\n- [recs (App::RecordStream)](https://metacpan.org/pod/App::RecordStream)\n\n    A set of tools for manipulating JSON (constrast with Yertl's YAML). For\n    interoperability, set the `YERTL_FORMAT` environment variable to\n    `\"json\"`.\n\n- [Catmandu](http://librecat.org)\n\n    A generic data processing toolkit. Convert data between multiple\n    formats, import/export into multiple databases, and manipulate data with\n    a mini-language.\n\n    This project is very much like Yertl, and more mature besides.\n\n- [jq](http://stedolan.github.io/jq/)\n\n    A filter for JSON documents. The inspiration for [yq](https://metacpan.org/pod/yq). For\n    interoperability, set the `YERTL_FORMAT` environment variable to\n    `\"json\"`.\n\n- [jt](https://metacpan.org/pod/App::jt)\n\n    JSON Transformer. Allows multiple ways of manipulating JSON, including\n    [JSONPath](http://goessner.net/articles/JsonPath/). For interoperability,\n    set the `YERTL_FORMAT` environment variable to `\"json\"`.\n\n- [pv (Pipe Viewer)](http://www.ivarch.com/programs/pv.shtml)\n\n    This tool helps examine how fast data is flowing through a shell\n    pipeline. If the size of the data is known, it can even provide\n    a progress bar and an ETA.\n\n- [netcat (nc)](http://netcat.sourceforge.net)\n\n    Netcat allows simple streaming over a network. Using Netcat you can\n    start a Yertl pipeline on one machine and finish it on another machine.\n    For example, you could generate metrics on each client machine, and then\n    write them to a central machine to insert into a database on that\n    machine.\n\n    Netcat does not come with any security, so be careful (use firewalls).\n\n- [socat](http://www.dest-unreach.org/socat/doc/socat.html)\n\n    Socat is a multi-purpose relay. It is similar to Netcat but with many\n    more features such as SSL and client verification. Socat has security,\n    so you can use this like Netcat in cases where you must accept data from\n    the Internet.\n\n- [parallel (GNU Parallel)](https://www.gnu.org/software/parallel/)\n\n    GNU Parallel is a shell tool for executing jobs in parallel on one or more\n    computers. Parallel is very similar to `xargs`, except it will execute\n    the commands on other computers.\n\n- [distribution](https://github.com/wizzat/distribution)\n\n    This tool creates charts. Pipe into it from `yq` to create simple\n    charts from your data.\n\n# AUTHOR\n\nDoug Bell \u003cpreaction@cpan.org\u003e\n\n# CONTRIBUTORS\n\n- James E Keenan \u003cjkeenan@cpan.org\u003e\n- Luke Triantafyllidis \u003cltriant@cpan.org\u003e\n\n# COPYRIGHT AND LICENSE\n\nThis software is copyright (c) 2018 by Doug Bell.\n\nThis is free software; you can redistribute it and/or modify it under\nthe same terms as the Perl 5 programming language system itself.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpreaction%2Fetl-yertl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpreaction%2Fetl-yertl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpreaction%2Fetl-yertl/lists"}