https://github.com/defra/ffc-pay-etl-framework

https://github.com/defra/ffc-pay-etl-framework

Last synced: 6 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/defra/ffc-pay-etl-framework
• Owner: DEFRA
• License: other
• Created: 2024-07-23T08:29:40.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-10-16T04:21:12.000Z (about 1 month ago)
• Last Synced: 2024-10-16T19:22:54.579Z (about 1 month ago)
• Language: JavaScript
• Size: 1.19 MB
• Stars: 1
• Watchers: 8
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE.md
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
## 🚀 Features

[![All Contributors](https://img.shields.io/badge/all_contributors-3-orange.svg?style=flat-square)](#contributors-)

- 💾 CSV Source and Destination

- 👮 Validators

- 🤖 Transformers

- 🚨 Error Checking

- 🐘 Write directly to Postgres

- 👨‍⚕️ Intellisense

## 📦 Install

```bash

npm install --save-dev ffc-pay-etl-framework

```

## 🪄 Usage

```js

// ESM

import {

  Etl,

  Loaders,

  Validators,

  Transformers,

  Destinations,

} from "ffc-pay-etl-framework";

// CJS

const {

  Etl,

  Loaders,

  Validators,

  Transformers,

  Destinations,

} = require("ffc-pay-etl-framework");

let csvFile = `${process.cwd()}/test/fixtures/SoilType.csv`;

const spinner = new pkg.Spinner().start("Running ETL Pipeline");

const etl = new Etl.Etl();

etl

  .loader(Loaders.CSVLoader({ path: csvFile, columns: columns }))

  .transform(

    Transformers.FakerTransformer({

      columns: [

        {

          name: "Dist Name",

          faker: "location.city",

        },

      ],

    })

  )

  .destination(

    Destinations.CSVFileDestination({

      fileName: "SoilType_Output.csv",

      headers: true,

      includeErrors: false,

      quotationMarks: true,

    })

  )

  .pump()

  .on("finish", () => {

    //Update spinner

    spinner.succeed("ETL Pipeline - succeeded");

  })

  .on("result", (data) => {

    console.log(data); // emits the last row with error information

  });

```

## 📢 Shout outs

[Faker JS](https://fakerjs.dev/guide/)

[Validator](https://github.com/validatorjs/validator.js)

[Kaggle](https://www.kaggle.com/datasets/anushkahedaoo/farming-factors)

## ✨ Contributing

Please make sure to read the [Contributing Guide](https://github.com/DEFRA/ffc-pay-etl-framework/blob/next/CONTRIBUTING.md) before making a pull request.

## Contributors ✨

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

  

    

      
Charlie Benger-Stevenson
💻

      
Leigh Godson
💻

      
Sam Plackett
💻    

  

  

    

      

        

          Add your contributions

        

      

    

  

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!