https://github.com/preaction/etl-yertl

ETL With a Shell
https://github.com/preaction/etl-yertl
Last synced: about 1 year ago
JSON representation
ETL With a Shell
Host: GitHub
URL: https://github.com/preaction/etl-yertl
Owner: preaction
License: other
Created: 2014-04-23T20:21:09.000Z (about 12 years ago)
Default Branch: master
Last Pushed: 2020-07-28T02:29:36.000Z (almost 6 years ago)
Last Synced: 2024-06-21T16:47:04.855Z (about 2 years ago)
Language: Perl
Homepage: http://preaction.me/yertl
Size: 793 KB
Stars: 27
Watchers: 6
Forks: 4
Open Issues: 52
Metadata Files:
- Readme: README.mkdn
- Changelog: CHANGES
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project

README

          # NAME

ETL::Yertl - ETL with a Shell

# VERSION

version 0.044

# STATUS



# SYNOPSIS

    ### On a shell...

    # Convert file to Yertl's format

    $ yfrom csv file.csv >work.yml

    $ yfrom json file.json >work.yml

    # Convert file to output format

    $ yto csv work.yml

    $ yto json work.yml

    # Parse HTTP logs into documents

    $ ygrok '%{LOG.HTTP_COMMON}' httpd.log

    # Read data from a database

    $ ysql db_name 'SELECT * FROM employee'

    # Write data to a database

    $ ysql db_name 'INSERT INTO employee ( id, name ) VALUES ( $.id, $.name )'

    ### In Perl...

    use ETL::Yertl;

    # Give everyone a 5% raise

    my $xform = file( '<', 'employees.yaml' )

              | transform( sub { $_->{salary} *= 1.05 } )

              >> stdout;

    $xform->run;

# DESCRIPTION

ETL::Yertl is an ETL ([Extract, Transform,

Load](https://en.wikipedia.org/wiki/Extract,_transform,_load)) for shells. It is

designed to accept data from multiple formats (CSV, JSON), manipulate them

using simple tools, and then convert them to an output format.

Yertl will have tools for:

- Extracting data from databases (MySQL, Postgres, MongoDB)

- Loading data into databases

- Extracting data from web services

- Writing data to web services

- Distributing data through messaging APIs (ZeroMQ)

# SUBROUTINES

## stdin

    my $stdin = stdin( %args );

Get a [ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) object for standard input. `%args` is a list

of key/value pairs passed to ["new" in ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream#new). Useful keys are:

- format

    Specify the format that standard input is. Defaults to `yaml` or the value

    of `YERTL_FORMAT` (see ["get\_default" in ETL::Yertl::Format](https://metacpan.org/pod/ETL::Yertl::Format#get_default).

## stdout

    my $stdout = stdout( %args );

Get a [ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) object for standard output. `%args` is a list

of key/value pairs passed to ["new" in ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream#new). Useful keys are:

- format

    Specify the format that standard input is. Defaults to `yaml` or the value

    of `YERTL_FORMAT` (see ["get\_default" in ETL::Yertl::Format](https://metacpan.org/pod/ETL::Yertl::Format#get_default).

- autoflush

    Immediately write documents to standard output to improve

    responsiveness, instead of queuing them for later writes for efficiency.

    This defaults to `1` (on). Set it to `0` to turn autoflush off.

## transform

    my $xform = transform( sub { ... } );

    my $xform = transform( 'Local::Transform::Class' => @args );

Create a new [ETL::Yertl::Transform](https://metacpan.org/pod/ETL::Yertl::Transform) object, passing in either a subref

to transform documents, or a class to instantiate and arguments to pass

to its constructor.

The subref will receive two arguments: The [ETL::Yertl::Transform](https://metacpan.org/pod/ETL::Yertl::Transform)

object and the document to transform. `$_` will also be set to the

document to transform.  The subref should return the transformed

document (either a new document, or the existing document after being

modified).

If given a transform class, it should inherit from

[ETL::Yertl::Transform](https://metacpan.org/pod/ETL::Yertl::Transform). The class will be loaded and an object

instantiated using the `@args`.

## file

    my $stream = file( $mode, $path, %args );

Create a [ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) object for the given `$path`.

`$mode` should be one of `<` for reading or `>` for writing.

`%args` are additional arguments to pass to the

[ETL::Yertl::FormatStream](https://metacpan.org/pod/ETL::Yertl::FormatStream) constructor. Useful keys are:

- format

    Specify the format that standard input is. Defaults to `yaml` or the value

    of `YERTL_FORMAT` (see ["get\_default" in ETL::Yertl::Format](https://metacpan.org/pod/ETL::Yertl::Format#get_default).

## yq

    my $xform = yq( $filter );

Create a [ETL::Yertl::Transform::Yq](https://metacpan.org/pod/ETL::Yertl::Transform::Yq) object with the given filter. See

["SYNTAX" in yq](https://metacpan.org/pod/yq#SYNTAX) for full filter syntax.

## loop

    my $loop = loop();

Get the [IO::Async::Loop](https://metacpan.org/pod/IO::Async::Loop) singleton. Use this to add other [IO::Async](https://metacpan.org/pod/IO::Async) objects

to a larger program. This is not needed for simple Yertl streams, and is mostly

used internally.

This is not exported by default. You can import it using `use ETL::Yertl 'loop'`.

# SEE ALSO

- [http://preaction.me/yertl](http://preaction.me/yertl)

    The Yertl home page.

## Yertl Tools

- [yfrom](https://metacpan.org/pod/yfrom)

    Convert incoming data (CSV, JSON) to Yertl documents.

- [yto](https://metacpan.org/pod/yto)

    Convert Yertl documents into another format (CSV, JSON).

- [ygrok](https://metacpan.org/pod/ygrok)

    Parse lines of text into Yertl documents.

- [ysql](https://metacpan.org/pod/ysql)

    Read/write documents from SQL databases.

- [yq](https://metacpan.org/pod/yq)

    A powerful mini-language for munging and filtering.

## Other Tools

Here are some other tools that can be used with Yertl

- [recs (App::RecordStream)](https://metacpan.org/pod/App::RecordStream)

    A set of tools for manipulating JSON (constrast with Yertl's YAML). For

    interoperability, set the `YERTL_FORMAT` environment variable to

    `"json"`.

- [Catmandu](http://librecat.org)

    A generic data processing toolkit. Convert data between multiple

    formats, import/export into multiple databases, and manipulate data with

    a mini-language.

    This project is very much like Yertl, and more mature besides.

- [jq](http://stedolan.github.io/jq/)

    A filter for JSON documents. The inspiration for [yq](https://metacpan.org/pod/yq). For

    interoperability, set the `YERTL_FORMAT` environment variable to

    `"json"`.

- [jt](https://metacpan.org/pod/App::jt)

    JSON Transformer. Allows multiple ways of manipulating JSON, including

    [JSONPath](http://goessner.net/articles/JsonPath/). For interoperability,

    set the `YERTL_FORMAT` environment variable to `"json"`.

- [pv (Pipe Viewer)](http://www.ivarch.com/programs/pv.shtml)

    This tool helps examine how fast data is flowing through a shell

    pipeline. If the size of the data is known, it can even provide

    a progress bar and an ETA.

- [netcat (nc)](http://netcat.sourceforge.net)

    Netcat allows simple streaming over a network. Using Netcat you can

    start a Yertl pipeline on one machine and finish it on another machine.

    For example, you could generate metrics on each client machine, and then

    write them to a central machine to insert into a database on that

    machine.

    Netcat does not come with any security, so be careful (use firewalls).

- [socat](http://www.dest-unreach.org/socat/doc/socat.html)

    Socat is a multi-purpose relay. It is similar to Netcat but with many

    more features such as SSL and client verification. Socat has security,

    so you can use this like Netcat in cases where you must accept data from

    the Internet.

- [parallel (GNU Parallel)](https://www.gnu.org/software/parallel/)

    GNU Parallel is a shell tool for executing jobs in parallel on one or more

    computers. Parallel is very similar to `xargs`, except it will execute

    the commands on other computers.

- [distribution](https://github.com/wizzat/distribution)

    This tool creates charts. Pipe into it from `yq` to create simple

    charts from your data.

# AUTHOR

Doug Bell 

# CONTRIBUTORS

- James E Keenan 

- Luke Triantafyllidis 

# COPYRIGHT AND LICENSE

This software is copyright (c) 2018 by Doug Bell.

This is free software; you can redistribute it and/or modify it under

the same terms as the Perl 5 programming language system itself.
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/preaction/etl-yertl

Awesome Lists containing this project

README
