Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/andrehrferreira/cmmv-normalizer
Schema-based data normalization module for CMMV
https://github.com/andrehrferreira/cmmv-normalizer
cmmv csv json normalizer schema xml yaml
Last synced: 4 days ago
JSON representation
Schema-based data normalization module for CMMV
- Host: GitHub
- URL: https://github.com/andrehrferreira/cmmv-normalizer
- Owner: cmmvio
- License: mit
- Created: 2024-11-27T15:32:42.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2025-01-24T02:01:35.000Z (15 days ago)
- Last Synced: 2025-01-24T02:13:36.008Z (15 days ago)
- Topics: cmmv, csv, json, normalizer, schema, xml, yaml
- Language: TypeScript
- Homepage:
- Size: 598 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
Contract-Model-Model-View (CMMV)
Building scalable and modular applications using contracts.
## Description
The `@cmmv/normalizer` module provides tools for parsing, transforming, and normalizing data from large files in various formats, such as **JSON**, **XML**, and **YAML**. Built for **performance** and **scalability**, it uses **stream-based parsing** to handle large datasets with minimal memory usage. The module integrates seamlessly with CMMV models and contracts, ensuring data consistency and validation.
Additionally, `@cmmv/normalizer` includes **tokenization** capabilities for handling sensitive data, leveraging `@cmmv/encryptor` for AES256 encryption and elliptic curve key generation.
## Features
- **Stream-Based Parsing:** Efficiently processes large files with low memory consumption.
- **Multi-Format Support:** JSON, XML, YAML, and customizable parsers.
- **Data Normalization:** Transform and validate data using configurable schemas.
- **Integration:** Works seamlessly with CMMV contracts and models.
- **Tokenization:** Secure sensitive data with AES256 and elliptic curves.
- **Customizable Pipelines:** Create custom transformers and validations.
## Installation
Install the `@cmmv/normalizer` package via npm:
```bash
$ pnpm add @cmmv/normalizer
```
## Quick Start
Below is an example of parsing and normalizing a JSON file with a custom schema:
```typescript
import {
JSONParser, AbstractParserSchema,
ToLowerCase, Tokenizer, ToDate, ToObjectId
} from '@cmmv/normalizer';
import { ECKeys } from '@cmmv/encryptor';
const keys = ECKeys.generateKeys();
const tokenizer = Tokenizer({
publicKey: ECKeys.getPublicKey(keys),
});
class CustomerSchema extends AbstractParserSchema {
public field = {
id: {
to: 'id',
transform: [ToObjectId],
},
name: { to: 'name' },
email: {
to: 'email',
validation: /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/,
transform: [ToLowerCase],
},
phoneNumber: {
to: 'phone',
transform: [tokenizer],
},
registrationDate: {
to: 'createdAt',
transform: [ToDate],
},
};
}
new JSONParser({
input: './sample/large-customers.json',
schema: CustomerSchema,
})
.pipe(data => console.log('Processed Data:', data))
.once('end', () => console.log('Parsing Complete'))
.once('error', error => console.error('Parsing Error:', error))
.start();
```
## Supported File Formats
The ``@cmmv/normalizer`` module includes built-in parsers for the following formats:
| Format | Parser | Example File |
|--------|-------------|-------------------|
| JSON | JSONParser | large-data.json |
| XML | XMLParser | large-data.xml |
| YAML | YAMLParser | large-data.yaml |
For custom formats, you can create your own parser by extending the ``AbstractParser``.
## API Reference
### AbstractParser
All parsers extend the AbstractParser class, providing a consistent interface for processing data:
| Method | Description |
|--------------------|-------------------------------------------------------------------------------------------|
| `pipe(fn: Function)` | Adds a function to the processing pipeline for transforming or handling parsed data. |
| `start()` | Starts the parser and begins processing the input file. |
| `processData(data: any)` | Processes individual data items through the schema and transformation pipeline. |
| `finalize()` | Finalizes the parsing operation and emits an end event. |
| `error(err: Error)` | Emits an error event for handling parsing errors. |
### Built-in Normalizers
The module provides several built-in normalizers for transforming data fields:
| Normalizer | Description |
|---------------|---------------------------------------------------------|
| `ToBoolean` | Converts a value to a boolean. |
| `ToDate` | Converts a value to a JavaScript Date object. |
| `ToFixed` | Formats a number to a fixed number of decimal places. |
| `ToFloat` | Converts a value to a floating-point number. |
| `ToInt` | Converts a value to an integer. |
| `Tokenizer` | Encrypts sensitive data with AES256 and elliptic curves.|
| `ToLowerCase` | Converts a string to lowercase. |
| `ToObjectId` | Converts a value to a MongoDB ObjectId. |
| `ToString` | Converts a value to a string. |
| `ToUpperCase` | Converts a string to uppercase. |
## Custom Parser
You can create a custom parser by extending ``AbstractParser``. Here’s an example for a CSV parser:
```typescript
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as readline from 'node:readline';
import { AbstractParser, ParserOptions } from '@cmmv/normalizer';
export class CSVParser extends AbstractParser {
constructor(options?: ParserOptions) {
super();
this.options = options;
}
public override start() {
const inputPath = path.resolve(this.options.input);
if (fs.existsSync(inputPath)) {
const readStream = fs.createReadStream(inputPath);
const rl = readline.createInterface({
input: readStream,
crlfDelay: Infinity, // Handle all newlines (Windows, Unix)
});
let headers: string[] | null = null;
rl.on('line', (line) => {
if (!headers) {
// First line contains headers
headers = line.split(',');
return;
}
// Map CSV columns to header fields
const values = line.split(',');
const record = headers.reduce((acc, header, index) => {
acc[header.trim()] = values[index]?.trim();
return acc;
}, {});
// Process each record
this.processData.call(this, record);
});
rl.on('close', this.finalize.bind(this));
rl.on('error', (error) => this.error.bind(this, error));
} else {
console.error(`File not found: ${inputPath}`);
}
}
}
```
The @cmmv/normalizer module provides a powerful and extensible framework for parsing and normalizing large datasets in various formats. Its tight integration with CMMV contracts and models ensures consistency and scalability, making it an essential tool for building robust Node.js applications.