{"id":20038794,"url":"https://github.com/defra/ffc-pay-etl-framework","last_synced_at":"2025-06-27T23:03:46.077Z","repository":{"id":250115853,"uuid":"832542511","full_name":"DEFRA/ffc-pay-etl-framework","owner":"DEFRA","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-21T13:00:26.000Z","size":1939,"stargazers_count":2,"open_issues_count":11,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-21T14:22:45.283Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DEFRA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-23T08:29:40.000Z","updated_at":"2025-04-20T13:29:56.000Z","dependencies_parsed_at":"2024-08-27T09:25:48.861Z","dependency_job_id":"0a208889-6845-4bfa-94db-108b45e750e8","html_url":"https://github.com/DEFRA/ffc-pay-etl-framework","commit_stats":null,"previous_names":["defra/ffc-pay-etl-framework"],"tags_count":39,"template":false,"template_full_name":null,"purl":"pkg:github/DEFRA/ffc-pay-etl-framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fffc-pay-etl-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fffc-pay-etl-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fffc-pay-etl-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fffc-pay-etl-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DEFRA","download_url":"https://codeload.github.com/DEFRA/ffc-pay-etl-framework/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DEFRA%2Fffc-pay-etl-framework/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262347471,"owners_count":23296893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T10:33:00.295Z","updated_at":"2025-06-27T23:03:46.058Z","avatar_url":"https://github.com/DEFRA.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 🚀 Features\n\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n\n[![All Contributors](https://img.shields.io/badge/all_contributors-3-orange.svg?style=flat-square)](#contributors-)\n\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n- 💾 CSV Source and Destination\n- 👮 Validators\n- 🤖 Transformers\n- 🚨 Error Checking\n- 🐘 Write directly to Postgres\n- 👨‍⚕️ Intellisense\n\n## 📦 Install\n\n```bash\nnpm install --save-dev ffc-pay-etl-framework\n```\n\n## 🪄 Usage\n\n```js\n// ESM\nimport {\n  Etl,\n  Loaders,\n  Validators,\n  Transformers,\n  Destinations,\n} from \"ffc-pay-etl-framework\";\n\n// CJS\nconst {\n  Etl,\n  Loaders,\n  Validators,\n  Transformers,\n  Destinations,\n} = require(\"ffc-pay-etl-framework\");\n\nlet csvFile = `${process.cwd()}/test/fixtures/SoilType.csv`;\nconst spinner = new pkg.Spinner().start(\"Running ETL Pipeline\");\n\nconst etl = new Etl.Etl();\n\netl\n  .loader(Loaders.CSVLoader({ path: csvFile, columns: columns }))\n  .transform(\n    Transformers.FakerTransformer({\n      columns: [\n        {\n          name: \"Dist Name\",\n          faker: \"location.city\",\n        },\n      ],\n    })\n  )\n  .destination(\n    Destinations.CSVFileDestination({\n      fileName: \"SoilType_Output.csv\",\n      headers: true,\n      includeErrors: false,\n      quotationMarks: true,\n    })\n  )\n  .pump()\n  .on(\"finish\", () =\u003e {\n    //Update spinner\n    spinner.succeed(\"ETL Pipeline - succeeded\");\n  })\n  .on(\"result\", (data) =\u003e {\n    console.log(data); // emits the last row with error information\n  });\n```\n\n## 📢 Shout outs\n\n[Faker JS](https://fakerjs.dev/guide/)\n\n[Validator](https://github.com/validatorjs/validator.js)\n\n[Kaggle](https://www.kaggle.com/datasets/anushkahedaoo/farming-factors)\n\n## ✨ Contributing\n\nPlease make sure to read the [Contributing Guide](https://github.com/DEFRA/ffc-pay-etl-framework/blob/next/CONTRIBUTING.md) before making a pull request.\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/suityou01\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/7422644?v=4?s=100\" width=\"100px;\" alt=\"Charlie Benger-Stevenson\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eCharlie Benger-Stevenson\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/DEFRA/ffc-pay-etl-framework/commits?author=suityou01\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/GodsonLeigh\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/139965284?v=4?s=100\" width=\"100px;\" alt=\"Leigh Godson\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eLeigh Godson\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/DEFRA/ffc-pay-etl-framework/commits?author=GodsonLeigh\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/samplackett\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/60177449?v=4?s=100\" width=\"100px;\" alt=\"Sam Plackett\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSam Plackett\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/DEFRA/ffc-pay-etl-framework/commits?author=samplackett\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e    \u003c/tr\u003e\n  \u003c/tbody\u003e\n  \u003ctfoot\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" size=\"13px\" colspan=\"7\"\u003e\n        \u003cimg src=\"https://raw.githubusercontent.com/all-contributors/all-contributors-cli/1b8533af435da9854653492b1327a23a4dbd0a10/assets/logo-small.svg\"\u003e\n          \u003ca href=\"https://all-contributors.js.org/docs/en/bot/usage\"\u003eAdd your contributions\u003c/a\u003e\n        \u003c/img\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tfoot\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fffc-pay-etl-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefra%2Fffc-pay-etl-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefra%2Fffc-pay-etl-framework/lists"}