{"id":15178563,"url":"https://github.com/chubbard/gratum","last_synced_at":"2025-06-11T14:08:04.078Z","repository":{"id":34294869,"uuid":"140464704","full_name":"chubbard/gratum","owner":"chubbard","description":"A simplified ETL engine for groovy.","archived":false,"fork":false,"pushed_at":"2025-04-09T17:16:12.000Z","size":1080,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-09T17:44:19.560Z","etag":null,"topics":["etl","etl-framework","etl-pipeline","groovy"],"latest_commit_sha":null,"homepage":null,"language":"Groovy","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chubbard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-10T17:12:10.000Z","updated_at":"2025-01-23T04:04:03.000Z","dependencies_parsed_at":"2023-12-12T01:30:01.978Z","dependency_job_id":"f39b65f3-71f5-418e-8503-6eb7ac8ad25a","html_url":"https://github.com/chubbard/gratum","commit_stats":{"total_commits":254,"total_committers":3,"mean_commits":84.66666666666667,"dds":0.08661417322834641,"last_synced_commit":"3ac5e01c4c75ce5adf100972ab896a52204da636"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chubbard%2Fgratum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chubbard%2Fgratum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chubbard%2Fgratum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chubbard%2Fgratum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chubbard","download_url":"https://codeload.github.com/chubbard/gratum/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250111133,"owners_count":21376596,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","etl-framework","etl-pipeline","groovy"],"created_at":"2024-09-27T15:05:31.576Z","updated_at":"2025-06-11T14:08:04.067Z","avatar_url":"https://github.com/chubbard.png","language":"Groovy","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n# Gratum ETL\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.chubbard/gratum/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.github.chubbard/gratum)\n[![Build Status](https://travis-ci.com/chubbard/gratum.svg?branch=master)](https://travis-ci.com/chubbard/gratum)\n\nA lightweight standalone ETL engine for groovy.  Gratum is groovy + datum.\nAnd gratum just happens to mean thank you or pleasant in Latin.  We built \ngratum with a couple of beliefs about data transformations.\n\n1. Groovy is a great language that easy to get things done quickly.\n2. Given a groovy shell and a good ETL DSL we can perform ETL is any environment.\n3. Functional programming and pipeline architecture can handle any transformation.\n\n## Installation\n\nFor Gradle:\n\n     compile group: 'com.github.chubbard', name: 'gratum', version: '1.1.15'\n\nFor Maven:\n\n      \u003cdependency\u003e\n        \u003cgroupId\u003ecom.github.chubbard\u003c/groupId\u003e\n        \u003cartifactId\u003egratum\u003c/artifactId\u003e\n        \u003cversion\u003e1.1.15\u003c/version\u003e\n      \u003c/dependency\u003e\n      \n## Oh Shell Yeah!\n\nGratum is meant to be usable in the groovy shell for getting things done quickly.  \nWe love awk and sed for quickly manipulating files, but we aren't always working on \nunix like systems.  And writing bash scripts to preserve everything we're doing is cumbersome.\nThe groovy shell provides us a portable environment to do similar manipulations.\n\nMake sure you have your `~/.groovy/grapeConfig.xml` setup first.  If you need a little refresher\nhere you go:\n\n    \u003civysettings\u003e\n        \u003csettings defaultResolver=\"downloadGrapes\"/\u003e\n        \u003cresolvers\u003e\n            \u003cchain name=\"downloadGrapes\"\u003e\n                \u003cibiblio name=\"local\" root=\"file:${user.home}/.m2/repository/\" m2compatible=\"true\"/\u003e    \t\n                \u003cibiblio name=\"ibiblio\" m2compatible=\"true\"/\u003e\n                \u003cibiblio name=\"java.net2\" root=\"http://download.java.net/maven/2/\" m2compatible=\"true\"/\u003e\n                \u003c!-- uncomment if you want SNAPSHOTS\n                \u003cibiblio name=\"maven.snapshot\" root=\"https://oss.sonatype.org/content/repositories/snapshots/\" m2compatible=\"true\"/\u003e\n                --\u003e\n            \u003c/chain\u003e\n        \u003c/resolvers\u003e\n    \u003c/ivysettings\u003e \n  \nHere is how to get gratum started up in your shell:\n\n    groovy:000\u003e import groovy.grape.Grape\n    groovy:000\u003e Grape.grab(group: 'com.github.chubbard', module:'gratum')\n    groovy:000\u003e import gratum.etl.*\n    groovy:000\u003e import static gratum.source.CsvSource.*\n    groovy:000\u003e csv('sample.csv').into().go\n\nBut, to make it easier to get started you'll want to add the following to your \n`$HOME/.groovy/groovysh.rc` file:\n\n    groovy.grape.Grape.grab(group: 'com.github.chubbard', module: 'gratum')\n    import gratum.etl.*\n    import gratum.source.*\n    import static gratum.source.CsvSource.*\n    import static gratum.source.CollectionSource.*\n    import static gratum.source.HttpSource.*\n    import static gratum.source.ZipSource.*\n\n## Examples\n\n#### Filter by a column on a single value.\n\n    from([\n            [ name: 'Chuck', gender: 'Male'],\n            [ name: 'Jane', gender: 'Female'],\n            [ name: 'Rob', gender: 'Male'],\n            [ name: 'Charlie', gender: 'Female'],\n            [ name: 'Sue', gender: 'Male'],\n            [ name: 'Jenny', gender: 'Female']\n        ]).\n        filter([gender: 'Female']).\n        go()\n\nThat should produce the following:\n\n    ----\n    Rejections by category\n    IGNORE_ROW: 3\n    \n    ----\n    ==\u003e Collection(6)\n    loaded 3\n    rejected 3\n    took 3 ms\n\n#### Filtering by multiple columns\n\n    from([\n            [ name: 'Chuck', gender: 'Male', city: 'London'],\n            [ name: 'Jane', gender: 'Female', city: 'London'],\n            [ name: 'Rob', gender: 'Male', city: 'Manchester'],\n            [ name: 'Charlie', gender: 'Female', city: 'Liverpool'],\n            [ name: 'Sue', gender: 'Male', city: 'Oxford'],\n            [ name: 'Jenny', gender: 'Female', city: 'Oxford']\n        ]).\n        filter([gender: 'Female', city: 'Oxford']).\n        go()\n\nThat should produce the following:\n\n    ----\n    Rejections by category\n    IGNORE_ROW: 5\n    \n    ----\n    ==\u003e Collection(6)\n    loaded 1\n    rejected 5\n    took 3 ms\n\n#### Filtering using a Collection\n\n    from([\n            [ name: 'Chuck', gender: 'Male'],\n            [ name: 'Jane', gender: 'Female'],\n            [ name: 'Rob', gender: 'Male'],\n            [ name: 'Charlie', gender: 'Female'],\n            [ name: 'Sue', gender: 'Male'],\n            [ name: 'Jenny', gender: 'Female']\n        ]).\n        filter([name: ['Chuck', 'Jane', 'Rob']]).\n        go()\n\nThat should produce the following:\n\n    ----\n    Rejections by category\n    IGNORE_ROW: 3\n    \n    ----\n    ==\u003e Collection(6)\n    loaded 3\n    rejected 3\n    took 3 ms\n\n#### Filter using a Regular Expression\n\n    from([\n            [ name: 'Chuck', gender: 'Male'],\n            [ name: 'Jane', gender: 'Female'],\n            [ name: 'Rob', gender: 'Male'],\n            [ name: 'Charlie', gender: 'Female'],\n            [ name: 'Sue', gender: 'Male'],\n            [ name: 'Jenny', gender: 'Female']\n        ]).\n        filter([name: ~/Ch.*/]).\n        go()\n\nThat should produce the following:\n\n    ----\n    Rejections by category\n    IGNORE_ROW: 4\n    \n    ----\n    ==\u003e Collection(6)\n    loaded 2\n    rejected 4\n    took 3 ms\n    \n#### Filter using a closure\n\n    csv('bank_balance.csv').into().asDouble('amount').filter { Map row -\u003e row.amount \u003e 500 }.go\n\nIn the above we load a file named bank_balance.csv then we convert the column `amount`\nto a double value.  Then we filter on `amount` for all rows containing an `amount`\ngreater than 500.\n\n#### Filter using closures and maps\n\nYou can mix closures with the map version of the `filter` method.  Here is an example:\n\n    from([\n            [ name: 'Chuck', gender: 'Male', age: 33],\n            [ name: 'Jane', gender: 'Female', age: 24],\n            [ name: 'Rob', gender: 'Male', age: 28],\n            [ name: 'Charlie', gender: 'Female', age: 35],\n            [ name: 'Sue', gender: 'Male', age: 50],\n            [ name: 'Jenny', gender: 'Female', age: 42]\n        ]).\n        filter([gender: 'Male', age: { v -\u003e v \u003e 30 }]).\n        go()\n\nThat should produce the following:\n\n    ----\n    Rejections by category\n    IGNORE_ROW: 4\n    \n    ----\n    ==\u003e Collection(6)\n    loaded 2\n    rejected 4\n    took 3 ms\n \n\n#### Sort by column\n\n    from([\n            [ name: 'Chuck', gender: 'Male', age: 40],\n            [ name: 'Jane', gender: 'Female', age: 25],\n            [ name: 'Rob', gender: 'Male', age: 33],\n            [ name: 'Charlie', gender: 'Female', age: 55],\n            [ name: 'Sue', gender: 'Male', age: 65],\n            [ name: 'Jenny', gender: 'Female', age: 43]\n        ]).\n       filter { Map row -\u003e row.age \u003e 50 }.\n       sort('age').\n       printRow().\n       go()        \n\n## Digging Deeper\n\n### Baby Steps\n\nSo how do Pipelines work?  A pipeline is a series of ordered steps.  Every\nPipeline has a Source.  A source feeds a Pipeline.  In the example below\nthe `from` method creates the source, attaches it to a Pipeline, and returns\nthe Pipeline.  The [`CollectionSource.from`](https://chubbard.github.io/gratum/groovydoc/gratum/source/CollectionSource.html)\nmethod takes in a collection as the source for the row data.\n\n    from([\n            [ name: 'Chuck', gender: 'Male'],\n            [ name: 'Jane', gender: 'Female'],\n            [ name: 'Rob', gender: 'Male'],\n            [ name: 'Charlie', gender: 'Female'],\n            [ name: 'Sue', gender: 'Male'],\n            [ name: 'Jenny', gender: 'Female']\n        ]).\n       addStep { Map row -\u003e\n           println( row )\n           return row\n       }.\n       go()\n       \nTo add a step to the pipeline use the `addStep` method.  The `addStep` method\ntakes a closure where the only parameter is a Map object.  That represents a \nsingle row or line we're processing.  Every closure added with the step must \nreturn a value.  Provided the return value is non-null it will be used \nas the parameter to the next step on the Pipeline.  This let's any step act as\na transformation function.  \n\nIf a step returns a null then it rejects this row, and no more steps are \nprocessed afterwards.  The Pipeline goes to the next row and starts over\nfrom the first step.\n\n### Rejections\n\nA Pipeline does not have to process every row.  This could be on purpose\nor because the data does not match expectations, a script error, etc.  Below\nis the example for how to reject a row:\n\n    from([\n            [ name: 'Chuck', gender: 'Male', age: 40],\n            [ name: 'Jane', gender: 'Female', age: 25],\n            [ name: 'Rob', gender: 'Male', age: 33],\n            [ name: 'Charlie', gender: 'Female', age: 55],\n            [ name: 'Sue', gender: 'Male', age: 65],\n            [ name: 'Jenny', gender: 'Female', age: 43]\n        ]).\n       addStep { Map row -\u003e\n          return row.gender == 'Female' ? row : null \n       }.\n       go()\n      \nThe example above shows how a step can reject a row by returning null, but\nthere is a better way.  \n\nWhat is often most important is the reason for why a rejections occurred.  In\nthe examples above we rejected by returning null, but that doesn't tell us\nwhy something was rejected.  Using the [`reject`](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#reject(java.util.Map,java.lang.String,%20gratum.etl.RejectionCategory))\nmethod we can specify more detail about why a row was rejected.\n\n    from([\n            [ name: 'Chuck', gender: 'Male', age: 40],\n            [ name: 'Jane', gender: 'Female', age: 25],\n            [ name: 'Rob', gender: 'Male', age: 33],\n            [ name: 'Charlie', gender: 'Female', age: 55],\n            [ name: 'Sue', gender: 'Male', age: 65],\n            [ name: 'Jenny', gender: 'Female', age: 43]\n        ]).\n       addStep { Map row -\u003e\n          return row.gender == 'Female' ? row : reject(row, \"Rejected gender ${row.gender}\", RejectionCategory.IGNORE_ROW) \n       }.\n       go()\n\nThis allows you to provide a [`RejectionCategory`](https://chubbard.github.io/gratum/groovydoc/gratum/etl/RejectionCategory.html) \nand a free form description.  Using categories is a great way to quickly group types of \nrejections into more easily managed sets.\n\n### What's in a name?\n\nKnowing why something was rejected is good, but where it was rejected is important for \npinpointing which step produced the rejection.  All steps have a name to \nidentify where in the pipeline a rejection occurs.\n\n    from([\n            [ name: 'Chuck', gender: 'Male', age: 40],\n            [ name: 'Jane', gender: 'Female', age: 25],\n            [ name: 'Rob', gender: 'Male', age: 33],\n            [ name: 'Charlie', gender: 'Female', age: 55],\n            [ name: 'Sue', gender: 'Male', age: 65],\n            [ name: 'Jenny', gender: 'Female', age: 43]\n        ]).\n       addStep('Filter by Gender = Female') { Map row -\u003e\n          return row.gender == 'Female' ? row : reject(row, \"Rejected gender ${row.gender}\", RejectionCategory.IGNORE_ROW) \n       }.\n       go()\n\nThe step name is provided as the first argument to the `addStep` method.  When a rejection is returned from \na step.  The step's name is added to the Rejection.\n\n### Processing Rejections\n\nDepending on your task you may or may not need to process rejections, but when you need to you can override the default\nrejection mechanism to process individual rejections using `onRejection` method.  Here is an example:\n\n    from([\n            [ name: 'Chuck', gender: 'Male', age: 40],\n            [ name: 'Jane', gender: 'Female', age: 25],\n            [ name: 'Rob', gender: 'Male', age: 33],\n            [ name: 'Charlie', gender: 'Female', age: 55],\n            [ name: 'Sue', gender: 'Male', age: 65],\n            [ name: 'Jenny', gender: 'Female', age: 43]\n        ]).\n        filter( gender: 'Male' ).\n        onRejection { Pipeline rejections -\u003e\n            rejections.save( 'rejections.csv', '|')\n        }.\n        go()\n\nBy default, rejections that result in a SCRIPT_ERROR are written to the console.  By invoking `onRejection` method you\noverride that behavior.\n\nIn this example of non-Male rows will be rejected by the `filter` method.  Then the `onRejection` registers\na closure, but instead of the normal Map argument it takes another Pipeline.  That pipeline is the rejections\npipeline, and we can add steps and call operations on it just like any normal pipeline.  But what travels over\nthat Pipeline are all the rejected rows from the parent Pipeline.  In this example, we used the `save` \nmethod to write out a pipe-delimited text file (ie a CSV).\n\nWhat's different about the rows that travel through the rejection pipeline is that they have 3 addition columns:\n\n1. **rejectionCategory**\n2. **rejectionReason**\n3. **rejectionStep**\n4. **rejectionException** (if the **rejectionCategory** is a SCRIPT_ERROR)\n\nThe rest of the columns are the original columns from the row object.\n\n### Let's go Already!\n\nIn all the examples our Pipeline chains have ended with a method `go`.  The `go` method is important\nas it starts the processing of the `Source`.  The `go` method executes the processing of the chain, but\nit also returns an instance of `LoadStaistic` which gives us statistics about the processing of the \n`Source`.  For example, it contains total number of rows that loaded, rows that were rejected, total\namount of time used while processing the `Source`, step timings, or rejections by categories.\n\n    LoadStatistic stats = from([\n            [ name: 'Chuck', gender: 'Male', age: 40],\n            [ name: 'Jane', gender: 'Female', age: 25],\n            [ name: 'Rob', gender: 'Male', age: 33],\n            [ name: 'Charlie', gender: 'Female', age: 55],\n            [ name: 'Sue', gender: 'Male', age: 65],\n            [ name: 'Jenny', gender: 'Female', age: 43]\n        ]).\n        filter( gender: 'Male' ).\n        go()\n    println( stats ) \n\nThis yields the following:\n\n    ----\n    Rejections by category\n    IGNORE_ROW: 3\n    \n    ----\n    ==\u003e Collection(6) \n    loaded 3 \n    rejected 3 \n    took 29 ms\n\nThe upper section is the rejections by category.  As discussed above rejections have categories so we group\nall rejections by their common categories and list the counts there.  The next section shows total rows\nthat loaded, the total rejections, and the total time it took to process the source.  Above those stats is the \nname of the pipeline.\n\n## Operations\n\nIt's much easier to use the existing operation methods that are included in the Pipeline.  For example,\nwe used an `addStep` to add a step that filtered out rows.  This is so common there is already an operation\nthat does this for you called `filter`.  There are plenty of existing methods to perform common operations.\n\n## Concurrency\n\nGratum out of the box is single threaded, and it uses the existing thread that invokes `go()` to process the `Pipeline`.\nHowever, there are times when you want to process data across many threads to speed things up on modern CPU architecture.\nA `Pipeline` is single threaded.  A single thread traverses a single instance of\na pipeline when it is processing data.  So each thread has its own instance of a `Pipeline` that's even if it appears\nthat only one `Pipeline` has been created.  `Pipeline` instances are not shared between threads.\n\nConcurrency in Gratum is performed using the `LocalConcurrentContext` object.  This object provides the operators to\ndefine what portions of a `Pipeline` are concurrent and which parts are single threaded.  `LocalConcurrentContext` works\non a spread-collect style concurrency.  Rows traversing a `Pipeline` are spread across multiple threads for \nprocessing, and the output of those threads are collected back onto a collector Pipeline for further processing.\n\nLet's look at an example:\n\n```groovy\n   csv(\"titanic.csv\", \",\")\n    .apply( new LocalConcurrentContext(4, 200)\n        .spread { pipeline -\u003e\n            pipeline.filter(gender: \"male\")\n        }.collect { pipeline -\u003e\n            pipeline.save(\"male_passengers.csv\", \",\")\n        }.connect()\n    )\n    .go()\n```\nIn this example, we are reading the CSV file `titanic.csv` and each row in the CSV is being spread across 4 workers.\n(1 worker equates to a thread).  Each worker pulls rows from a queue (i.e. queue of size 200) that the producer thread (i.e. \nthe thread invoking `go()`) will place a row onto it.  When the queue fills up the producer thread is paused and waits\nfor the queue to drain.\n\nThe `spread` operator defines the worker's pipeline.  The spread closure is invoked multiple times to configure a Pipeline \nfor each worker.  This way each worker receives the same instructions, but also separate instances.  You can treat things\nwithin this closure as belonging to that worker thread (this is important later).\n\nThe `collect` operator defines the pipeline used to gather results from the worker's pipelines.  This runs on a single \nthread, but it is not the same as the original thread.  All rows that were outputted from each worker's pipeline will\nbe passed to the collect pipeline for further processing.  Using the `collect` operator is optional.\n\nFinally, the `connect` operator returns the closure used to configure the original pipeline and set up the spread-collect \npattern.  This is required.\n\nIn the example, above titantic.csv is read out by the producer thread, each row is sent to 1 of 4 workers, the workers filter \nout any rows that are `gender == \"male\"` and returns only the male rows.  The collect pipeline receives the filtered rows \nand writes those to `male_passengers.csv`.  This is a trivial example that wouldn't really benefit from threading, but\nit does illustrate the patterns involved.\n\n### Safe Concurrency\n\nConcurrency brings with it issues of safety when dealing with reading and writing access to shared resources.  This often\nbrings up locking, mutexes, latches, etc.  Gratum does not dictate any locking your program should use.  If you must\nsynchronize access that is an exercise for you to figure out.  But, there are some ways to safely access worker local \nstate that doesn't involve synchronization.\n\nClosures are great because they can access the outer scope to read and modify values.  These operations can function like\ninstance variables on objects giving a closure  state that persists between function calls.  When using concurrency\nin Gratum you must be careful about which variables you modify and read from within your closures as workers could access\nsomething that is shared creating bugs.\n\nHere is an example:\n\n```groovy\n\n// this would be visible and shared across all threads (producer, workers, collector).\n// Variables at this scope would need synchronization to prevent race conditions and\n// concurrency errors.  Be very very careful accessing variables defined at this scope.\nMap\u003cString,String\u003e values = [:]\n\ncsv(\"images.csv\", \",\")\n    .asDouble(\"price\")\n    .apply( new LocalConcurrentContext(4, 200)\n            .spread { pipeline -\u003e\n                // these local variables are visible to operations on the \n                // pipeline used by the worker.  They are not shared between\n                // workers so it's safe to use these.\n                Base64 base64 = Base64.newInstance()\n                Decoder decoder = base64.getDecoder()\n                \n                pipeline.addField(\"image\") { row -\u003e\n                    // this method uses a single instance of Base64 decoder across all rows played down its pipeline.\n                    // this reduces the memory used and increases performance because it's not constructing an instance\n                    // for every row it encounters!\n                    decoder.decode( row[\"imageBase64\"] )\n                    row\n                }\n                .save( new Sink() {\n                    int imagesProcessed = 0\n                    \n                    String getName() { return \"png\" }\n                    \n                    void attach(Pipeline pipeline) {\n                        pipeline.addStep(\"Save PNG\") { row-\u003e\n                            File output = new File(row.filename as String)\n                            output.withOutputStream { stream -\u003e\n                                stream.write( row.image as byte[] )\n                            }\n                            imagesProcessed++\n                        }\n                    }\n                    \n                    Map\u003cString,String\u003e getResult() {\n                        return [processed: imagesProcessed]\n                    }\n                })\n            }\n            .collect { pipeline -\u003e\n                // this is local to the collector thread so variables defined within this closure\n                // are safe to use without synchronization\n                int total = 0\n                pipeline.addStep(\"Total\") { row -\u003e\n                    total += row.processed\n                }\n                .after {\n                    println(\"Processed ${total} images\")\n                }\n            }\n            .connect()\n    )\n    .go()\n```\n\nIn the above example, it shows how variables defined in different closures are shared among threads or not shared (and thus\nsafe to use in your operators).  Defining variables within the `spread` and `collect` closures are safe to use within their\nrespective pipeline's operators.  It's not safe to use variables defined in the method where the pipeline is first \nconstructed from multiple threads (ie the `spread` and `collect` closures).  The producer thread may use them safely (i.e.\nany operator defined above the `apply` method using the `LocalConncurrentContext`).\n\n## Examples\n\nUse `OkHttpSource` to fetch JSON from a URL and iterate over an array within that JSON document.\n```\nimport static gratum.source.OkHttpSource.*\n\nhttp(\"http://api.open-notify.org/astros.json\")\n    .into()\n    .inject(\"\") { row -\u003e\n        row.json?.people\n    }\n    .printRow(\"name\",\"craft\")\n    .go()\n```\nAnother `OkHttpSource` to fetch JSON data from a URL and iterate over the data contained in the JSON document.\n```\nimport groovy.json.JsonSlurper\n\nimport static gratum.source.OkHttpSource.*\n\nhttp( \"https://www.freeforexapi.com/api/live\" ) {\n        query( pairs: [\"EURGBP\",\"USDJPY\"].join(\",\") )\n        header(\"Accept\", \"application/json\")\n    }\n    .into()\n    .addField(\"json\") { row -\u003e\n        // this has to be here because this service returns a Content-Type of text/html \n        // instead of the correct mime-type of application/json so we manually parse it\n        new JsonSlurper().parse(row.body.charStream())\n    }\n    .filter(\"Only proceed when markets are open\") { row -\u003e row.json.rates }\n    .inject(\"Flatten JSON and inject rates\") { row -\u003e\n        row.json?.rates?.collect { currency, obj -\u003e [ currency: currency, rate: obj.rate, timestamp: obj.timestamp ] }\n    }\n    .printRow(\"currency\", \"rate\", \"timestamp\")\n    .go()\n```\n\n## API Docs\n\n[groovydoc](https://chubbard.github.io/gratum/groovydoc)\n\n[javadoc](https://chubbard.github.io/gratum/javadoc)\n\n### Basics\n\n[addStep](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#addStep(java.lang.String,%20Closure%3CMap%3E))\n\n#### Column Manipulation\n\n[addField](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#addField(java.lang.String,%20groovy.lang.Closure))\n\n[renameFields](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#renameFields(java.util.Map))\n\n[setField](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#setField(java.lang.String,%20java.lang.Object))\n\n[clip](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#clip(java.lang.String))\n\n[defaultValues](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#defaultValues(java.util.Map))\n\n[defaultsBy](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#defaultsBy(java.util.Map))\n\n#### Data Types\n\n[asInt](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#asInt(java.lang.String))\n\n[asDouble](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#asDouble(java.lang.String))\n\n[asBoolean](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#asBoolean(java.lang.String))\n\n[asDate](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#asDate(java.lang.String,%20java.lang.String))\n\n#### Filtering\n\n[filter(Map)](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#filter(java.util.Map))\n\n[filter(Closure)](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#filter(groovy.lang.Closure))\n\n[groupBy](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#groupBy(java.lang.String))\n\n### Data Manipulation\n\n[sort](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#sort(java.lang.String))\n\n[sort](http://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#sort(Tuple2%3CString,%20SortOrder%3E))\n\n[sort](http://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#sort(java.lang.String,%20Comparator%3CMap%3CString,%20Object%3E%3E))\n\n[trim](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#trim())\n\n[unique](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#unique(java.lang.String))\n\n[limit](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#limit(long,boolean))\n\n[replaceAll](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#replaceAll(java.lang.String,%20java.util.regex.Pattern,%20java.lang.String))\n\n[replaceValues](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#replaceValues(java.lang.String,%20Map%3CString,%20String%3E))\n\n### Branching\n\n[branch](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#branch(Closure%3CVoid%3E))\n\n[branch](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#branch(Map%3CString,%20Object%3E,%20Closure%3CVoid%3E))\n\n[onRejection](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#onRejection(Closure%3CVoid%3E))\n\n#### Pipeline Manipulation\n\n[concat](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#concat(gratum.etl.Pipeline))\n\n[exchange](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#exchange(Closure%3CPipeline%3E))\n\n[inject(Closure)](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#inject(groovy.lang.Closure))\n\n[inject(String, Closure)](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#inject(java.lang.String,%20groovy.lang.Closure))\n\n[intersect](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#intersect(gratum.etl.Pipeline,%20def))\n\n[join](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#join(gratum.etl.Pipeline,%20def,%20boolean))\n\n[fillDownBy](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#fillDownBy(Closure%3CBoolean%3E))\n\n[configure](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#configure(Closure%3CPipeline%3E))\n\n[apply](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#apply(Closure%3CVoid%3E))\n\n[reduce](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#reduce(java.lang.String,%20Map%3CString,%20Object%3E,%20Closure%3CMap%3CString,%20Object%3E%3E))\n\n[flattenWindow](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#flattenWindow(java.lang.String,%20Closure%3CList%3CMap%3CString,%20Object%3E%3E%3E))\n\n### Output\n\n[save](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#save(java.lang.String,%20java.lang.String,%20List%3CString%3E))\n\n[save](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#save(Sink%3CMap%3CString,%20Object%3E%3E))\n\n[printRow](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#printRow(java.lang.String))\n\n[encryptPgp](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#encryptPgp(java.lang.String,%20groovy.lang.Closure))\n\n[decryptPgp](https://chubbard.github.io/gratum/groovydoc/gratum/etl/Pipeline.html#decryptPgp(java.lang.String,%20groovy.lang.Closure))\n\n## Sources\n\nOperations are great, but you need data for those operations to work on.  Sources are how that data is \npassed into the Pipeline.  These are the Sources you can use to provide data.\n\n[csv](https://chubbard.github.io/gratum/groovydoc/gratum/source/CsvSource.html)\n\n[file](https://chubbard.github.io/gratum/groovydoc/gratum/source/FileSystemSource.html)\n\n[xls](https://chubbard.github.io/gratum/groovydoc/gratum/source/XlsSource.html)\n\n[xlsx](https://chubbard.github.io/gratum/groovydoc/gratum/source/XlsxSource.html)\n\n[json](https://chubbard.github.io/gratum/groovydoc/gratum/source/JsonSource.html)\n\n[jsonl](https://chubbard.github.io/gratum/groovydoc/gratum/source/JsonSource.html)\n\n[collection](https://chubbard.github.io/gratum/groovydoc/gratum/source/CollectionSource.html)\n\n[zip](https://chubbard.github.io/gratum/groovydoc/gratum/source/ZipSource.html)\n\n[http/https](https://chubbard.github.io/gratum/groovydoc/gratum/source/OkHttpSource.html)\n\n[http/https](https://chubbard.github.io/gratum/groovydoc/gratum/source/HttpSource.html) - (deprecated)\n\n[jdbc](https://chubbard.github.io/gratum/groovydoc/gratum/source/JdbcSource.html)\n\n[concat](https://chubbard.github.io/gratum/groovydoc/gratum/source/ConcatSource.html)\n\n[ssh](https://chubbard.github.io/gratum/groovydoc/gratum/source/SshSource.html)\n\n[archived](https://chubbard.github.io/gratum/groovydoc/gratum/source/ArchivedSource.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchubbard%2Fgratum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchubbard%2Fgratum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchubbard%2Fgratum/lists"}