{"id":26530061,"url":"https://github.com/derafu/etl","last_synced_at":"2026-05-10T15:34:38.207Z","repository":{"id":283108948,"uuid":"948152110","full_name":"derafu/etl","owner":"derafu","description":"From spreadsheets to databases seamlessly","archived":false,"fork":false,"pushed_at":"2026-02-28T15:38:07.000Z","size":240,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-28T18:51:44.350Z","etag":null,"topics":["database","doctrine","etl","php","seed","spreadsheet","sql"],"latest_commit_sha":null,"homepage":"https://www.derafu.dev/docs/data/etl","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derafu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-13T20:50:03.000Z","updated_at":"2026-02-28T15:38:29.000Z","dependencies_parsed_at":"2025-07-12T02:22:14.372Z","dependency_job_id":"3173106b-455b-4fb7-bb89-ca4848caba1b","html_url":"https://github.com/derafu/etl","commit_stats":null,"previous_names":["derafu/seed","derafu/etl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/derafu/etl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derafu%2Fetl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derafu%2Fetl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derafu%2Fetl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derafu%2Fetl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derafu","download_url":"https://codeload.github.com/derafu/etl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derafu%2Fetl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32861582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"ssl_error","status_checked_at":"2026-05-10T13:40:02.145Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","doctrine","etl","php","seed","spreadsheet","sql"],"created_at":"2025-03-21T17:29:20.674Z","updated_at":"2026-05-10T15:34:38.161Z","avatar_url":"https://github.com/derafu.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Derafu: ETL - From Spreadsheets to Databases Seamlessly\n\n![GitHub last commit](https://img.shields.io/github/last-commit/derafu/etl/main)\n![CI Workflow](https://github.com/derafu/etl/actions/workflows/ci.yml/badge.svg?branch=main\u0026event=push)\n![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/derafu/etl)\n![GitHub Issues](https://img.shields.io/github/issues-raw/derafu/etl)\n![Total Downloads](https://poser.pugx.org/derafu/etl/downloads)\n![Monthly Downloads](https://poser.pugx.org/derafu/etl/d/monthly)\n\nA PHP package that transforms spreadsheet data into database structures and content with minimal effort.\n\n## Overview\n\nDerafu ETL provides a streamlined solution for converting data between spreadsheets and databases. With a clean, fluent API, it simplifies complex data integration tasks through a pipeline architecture.\n\n```php\n$pipeline = new Pipeline();\n$result = $pipeline\n    -\u003eextract('data.xlsx')      // Extract data from a spreadsheet.\n    -\u003etransform($rules)         // Apply transformations rules (optional).\n    -\u003eload('database.sqlite')   // Load into a database.\n    -\u003eexecute()\n;\n```\n\n## Key Features\n\n{.list-unstyled}\n- 📤 **Extract** data from various sources (XLSX, ODS, CSV, databases).\n- 🔄 **Transform** data with customizable rules.\n- 📥 **Load** data into different target systems.\n- 🔁 **Bidirectional** conversion between spreadsheets and databases.\n- 🏗️ **Schema management** with automatic table creation and structure updates.\n- 📊 **Data visualization** capabilities with schema export to Markdown, D2, and more.\n- 🧩 **Extensible** architecture for custom source and target systems.\n\n## Installation\n\nInstall via Composer:\n\n```bash\ncomposer require derafu/etl\n```\n\n## Quick Start\n\n### Command Line\n\nThe quickest way to use Derafu ETL is through the command line:\n\n```bash\nphp app/console.php derafu:etl data.xlsx database.sqlite\n```\n\nThis extracts data from `data.xlsx` and loads it into a new SQLite database on `database.sqlite`.\n\n#### Example\n\nRun the example used in tests with:\n\n```shell\nphp app/console.php derafu:etl tests/fixtures/spreadsheet-data.xlsx\n```\n\nThis will create a `spreadsheet-data.sqlite` in the current directory.\n\n\n### PHP Code\n\n```php\nuse Derafu\\ETL\\Pipeline\\Pipeline;\n\n$pipeline = new Pipeline();\n$result = $pipeline\n    -\u003eextract('data.xlsx')  // Load data from a XLSX.\n    -\u003etransform()           // This will use default transformations.\n    -\u003eload([                // You can specify the configuration for Doctrine.\n        'doctrine' =\u003e [\n            'driver' =\u003e 'pdo_sqlite',\n            'path' =\u003e 'database.sqlite',\n        ]\n    ])\n    -\u003eexecute();            // This is will run the process.\n\necho \"Rows loaded: \" . $result-\u003erowsLoaded();\n```\n\n## Understanding ETL Pipelines\n\nAn ETL pipeline consists of three main steps:\n\n1. **Extract**: Read data from a source (e.g., spreadsheet).\n2. **Transform**: Apply rules and transformations to the data.\n3. **Load**: Write the transformed data to a target (e.g., database).\n\nDerafu ETL provides a clean interface for each step while handling the complex details behind the scenes.\n\n## More than just move data to a target\n\n### Export Database Schema to Markdown\n\n```php\nuse Derafu\\ETL\\Database\\DatabaseManager;\nuse Derafu\\ETL\\Schema\\Target\\MarkdownSchemaTarget;\n\n$manager = new DatabaseManager();\n$database = $manager-\u003econnect('database.sqlite');\n\n$target = new MarkdownSchemaTarget();\n$markdown = $target-\u003eapplySchema($database-\u003eschema());\n\nfile_put_contents('schema.md', $markdown);\n```\n\n### Generate Database Diagram\n\n```php\nuse Derafu\\ETL\\Database\\DatabaseManager;\nuse Derafu\\ETL\\Schema\\Target\\D2SchemaTarget;\n\n$manager = new DatabaseManager();\n$database = $manager-\u003econnect('database.sqlite');\n\n$target = new D2SchemaTarget();\n$d2 = $target-\u003eapplySchema($database-\u003eschema());\n\nfile_put_contents('schema.d2', $d2);\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.\n\n## License\n\nThis package is open-sourced software licensed under the [MIT license](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderafu%2Fetl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderafu%2Fetl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderafu%2Fetl/lists"}