{"id":13518331,"url":"https://github.com/bpolaszek/bentools-etl","last_synced_at":"2025-04-06T08:14:05.711Z","repository":{"id":52206502,"uuid":"51530658","full_name":"bpolaszek/bentools-etl","owner":"bpolaszek","description":"PHP ETL (Extract / Transform / Load) library with SOLID principles + almost no dependency.","archived":false,"fork":false,"pushed_at":"2023-12-28T17:16:03.000Z","size":341,"stargazers_count":120,"open_issues_count":0,"forks_count":4,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-27T11:12:05.045Z","etag":null,"topics":["callable","etl","export","extract","extractor","import","input","invoke","load","loader","loop","output","pattern","php","transform","transformer"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bpolaszek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-11T17:06:37.000Z","updated_at":"2024-06-21T01:05:22.335Z","dependencies_parsed_at":"2023-11-10T12:45:51.081Z","dependency_job_id":"bc074f64-23b8-40dc-bf5a-d51ca66c49eb","html_url":"https://github.com/bpolaszek/bentools-etl","commit_stats":{"total_commits":90,"total_committers":5,"mean_commits":18.0,"dds":"0.25555555555555554","last_synced_commit":"ab170a6be4c5f898c6e4a8e44d23fa184b7d5e0e"},"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bpolaszek%2Fbentools-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bpolaszek%2Fbentools-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bpolaszek%2Fbentools-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bpolaszek%2Fbentools-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bpolaszek","download_url":"https://codeload.github.com/bpolaszek/bentools-etl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247451665,"owners_count":20940944,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["callable","etl","export","extract","extractor","import","input","invoke","load","loader","loop","output","pattern","php","transform","transformer"],"created_at":"2024-08-01T05:01:43.663Z","updated_at":"2025-04-06T08:14:05.605Z","avatar_url":"https://github.com/bpolaszek.png","language":"PHP","readme":"[![Latest Stable Version](https://poser.pugx.org/bentools/etl/v/stable)](https://packagist.org/packages/bentools/etl)\n[![License](https://poser.pugx.org/bentools/etl/license)](https://packagist.org/packages/bentools/etl)\n[![CI Workflow](https://github.com/bpolaszek/bentools-etl/actions/workflows/ci.yml/badge.svg)](https://github.com/bpolaszek/bentools-etl/actions/workflows/ci.yml)\n[![Coverage](https://codecov.io/gh/bpolaszek/bentools-etl/branch/master/graph/badge.svg?token=L5ulTaymbt)](https://codecov.io/gh/bpolaszek/bentools-etl)\n[![Total Downloads](https://poser.pugx.org/bentools/etl/downloads)](https://packagist.org/packages/bentools/etl)\n\nOkay, so you heard about the [Extract / Transform / Load](https://en.wikipedia.org/wiki/Extract,_transform,_load) pattern,\nand you're looking for a PHP library to do the stuff. Alright, let's go!\n\n`bentools/etl` is a versatile PHP library for implementing the Extract, Transform, Load (ETL) pattern, designed to streamline data processing tasks.\n\nTable of Contents\n-----------------\n\n- [Concepts](#concepts)\n- [Installation](#installation)\n- [Getting started](#usage)\n  - [The EtlState object](doc/getting-started.md#the-etlstate-object)\n  - [Skipping items](doc/getting-started.md#skipping-items)\n  - [Stopping the workflow](doc/getting-started.md#stopping-the-workflow)\n  - [Using events](doc/getting-started.md#using-events)\n  - [Flush frequency and early flushes](doc/getting-started.md#flush-frequency-and-early-flushes)\n- [Advanced Usage](doc/advanced_usage.md)\n    - [Creating your own Extractor / Transformers / Loaders](doc/advanced_usage.md#creating-your-own-extractor--transformers--loaders)\n    - [Difference between yield and return in transformers](doc/advanced_usage.md#difference-between-yield-and-return-in-transformers)\n    - [Next tick](doc/advanced_usage.md#next-tick)\n    - [Chaining extractors / transformers / loaders](doc/advanced_usage.md#chaining-extractors--transformers--loaders)\n    - [Reading from STDIN / Writing to STDOUT](doc/advanced_usage.md#reading-from-stdin--writing-to-stdout)\n    - [Instantiators](doc/advanced_usage.md#instantiators)\n    - [Using ReactPHP](doc/advanced_usage.md#using-reactphp-experimental)\n- [Recipes](doc/recipes.md)\n- [Contributing](#contribute)\n- [License](#license)\n\nConcepts\n--------\n\nLet's cover the basic concepts:\n- **Extract**: you have a source of data (a database, a CSV file, whatever) - an **extractor** is able to read that data and provide an iterator of items\n- **Transform**: apply transformation to each item. A **transformer** may generate 0, 1 or several items to **load** (for example, 1 item may generate multiple SQL queries)\n- **Load**: load transformed item to the destination. For example, **extracted items** have been **transformed** to SQL queries, and your **loader** will run those queries against your database.\n\nInstallation\n------------\n\n```bash\ncomposer require bentools/etl\n```\n\n\u003e [!WARNING]\n\u003e Current version (4.0) is a complete redesign and introduces significant BC (backward compatibility) breaks.\n\u003e Avoid upgrading from `^2.0` or `^3.0` unless you're fully aware of the changes.\n\nUsage\n-----\n\nNow let's have a look on how simple it is:\n\n```php\nuse BenTools\\ETL\\EtlExecutor;\n\n// Given\n$singers = ['Bob Marley', 'Amy Winehouse'];\n\n// Transform each singer's name to uppercase and process the array\n$etl = (new EtlExecutor())\n    -\u003etransformWith(fn (string $name) =\u003e strtoupper($name));\n\n// When\n$report = $etl-\u003eprocess($singers);\n\n// Then\nvar_dump($report-\u003eoutput); // [\"BOB MARLEY\", \"AMY WINEHOUSE\"]\n```\n\nOK, that wasn't really hard, here we basically don't have to _extract_ anything (we can already iterate on `$singers`),\nand we're not _loading_ anywhere, except into PHP's memory. \n\nYou may ask, \"why don't you just `array_map('strtoupper', $singers)` ?\" and you're totally right.\n\nBut sometimes, extracting, transforming and / or loading get a little more complex. \nYou may want to extract from a file, a crawled content on the web, perform one to many transformations, maybe skip some items,\nor reuse some extraction, transformation or loading logic.\n\nHere's another example of what you can do:\n\n```php\nuse BenTools\\ETL\\EventDispatcher\\Event\\TransformEvent;\nuse BenTools\\ETL\\Loader\\JSONLoader;\n\nuse function BenTools\\ETL\\extractFrom;\n\n$executor = extractFrom(function () {\n    yield ['firstName' =\u003e 'Barack', 'lastName' =\u003e 'Obama'];\n    yield ['firstName' =\u003e 'Donald', 'lastName' =\u003e 'Trump'];\n    yield ['firstName' =\u003e 'Joe', 'lastName' =\u003e 'Biden'];\n})\n    -\u003etransformWith(fn (array $item) =\u003e implode(' ', array_values($item)))\n    -\u003eloadInto(new JSONLoader())\n    -\u003eonTransform(function (TransformEvent $event) {\n        if ('Donald Trump' === $event-\u003etransformResult-\u003evalue) {\n            $event-\u003estate-\u003eskip();\n        }\n    });\n\n$report = $executor-\u003eprocess();\n\ndump($report-\u003eoutput); // string '[\"Barack Obama\", \"Joe Biden\"]'\n```\n\nOr: \n\n```php\n$report = $executor-\u003eprocess(destination: 'file:///tmp/presidents.json');\nvar_dump($report-\u003eoutput); // string 'file:///tmp/presidents.json' - content has been written here\n```\n\nYou get the point. Now you're up to write your own workflows! \n\nContinue reading the [Getting Started Guide](doc/getting-started.md).\n\nContribute\n----------\n\nContributions are welcome! Don't hesitate to suggest recipes.\n\nThis library is 100% covered with [Pest](https://pestphp.com) tests.\n\nPlease ensure to run tests using the command below and maintain code coverage before submitting PRs.\n\n```bash\ncomposer ci:check\n```\n\nLicense\n-------\n\nMIT.\n","funding_links":[],"categories":["PHP"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbpolaszek%2Fbentools-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbpolaszek%2Fbentools-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbpolaszek%2Fbentools-etl/lists"}