{"id":24637846,"url":"https://github.com/cmmvio/cmmv-normalizer","last_synced_at":"2026-02-13T01:40:44.374Z","repository":{"id":265246512,"uuid":"895116048","full_name":"cmmvio/cmmv-normalizer","owner":"cmmvio","description":"Schema-based data normalization module for CMMV","archived":false,"fork":false,"pushed_at":"2025-02-11T18:52:43.000Z","size":625,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-12T13:08:19.866Z","etag":null,"topics":["cmmv","csv","json","normalizer","schema","xml","yaml"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmmvio.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-27T15:32:42.000Z","updated_at":"2025-02-11T18:52:47.000Z","dependencies_parsed_at":"2025-03-29T13:44:29.332Z","dependency_job_id":null,"html_url":"https://github.com/cmmvio/cmmv-normalizer","commit_stats":null,"previous_names":["andrehrferreira/cmmv-normalizer","cmmvio/cmmv-normalizer"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/cmmvio/cmmv-normalizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmmvio%2Fcmmv-normalizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmmvio%2Fcmmv-normalizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmmvio%2Fcmmv-normalizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmmvio%2Fcmmv-normalizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmmvio","download_url":"https://codeload.github.com/cmmvio/cmmv-normalizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmmvio%2Fcmmv-normalizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29392035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T00:53:09.511Z","status":"ssl_error","status_checked_at":"2026-02-13T00:53:09.126Z","response_time":55,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cmmv","csv","json","normalizer","schema","xml","yaml"],"created_at":"2025-01-25T10:12:05.243Z","updated_at":"2026-02-13T01:40:44.344Z","avatar_url":"https://github.com/cmmvio.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\r\n  \u003ca href=\"https://cmmv.io/\" target=\"blank\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/cmmvio/docs.cmmv.io/main/public/assets/logo_CMMV2_icon.png\" width=\"300\" alt=\"CMMV Logo\" /\u003e\u003c/a\u003e\r\n\u003c/p\u003e\r\n\u003cp align=\"center\"\u003eContract-Model-Model-View (CMMV) \u003cbr/\u003e Building scalable and modular applications using contracts.\u003c/p\u003e\r\n\u003cp align=\"center\"\u003e\r\n    \u003ca href=\"https://www.npmjs.com/package/@cmmv/normalizer\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/@cmmv/normalizer.svg\" alt=\"NPM Version\" /\u003e\u003c/a\u003e\r\n    \u003ca href=\"https://github.com/cmmvio/cmmv-server/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/npm/l/@cmmv/core.svg\" alt=\"Package License\" /\u003e\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003ca href=\"https://cmmv.io\"\u003eDocumentation\u003c/a\u003e \u0026bull;\r\n  \u003ca href=\"https://github.com/cmmvio/cmmv-normalizer/issues\"\u003eReport Issue\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n## Description\r\n\r\nThe `@cmmv/normalizer` module provides tools for parsing, transforming, and normalizing data from large files in various formats, such as **JSON**, **XML**, and **YAML**. Built for **performance** and **scalability**, it uses **stream-based parsing** to handle large datasets with minimal memory usage. The module integrates seamlessly with CMMV models and contracts, ensuring data consistency and validation.\r\n\r\nAdditionally, `@cmmv/normalizer` includes **tokenization** capabilities for handling sensitive data, leveraging `@cmmv/encryptor` for AES256 encryption and elliptic curve key generation.\r\n\r\n## Features\r\n\r\n- **Stream-Based Parsing:** Efficiently processes large files with low memory consumption.\r\n- **Multi-Format Support:** JSON, XML, YAML, and customizable parsers.\r\n- **Data Normalization:** Transform and validate data using configurable schemas.\r\n- **Integration:** Works seamlessly with CMMV contracts and models.\r\n- **Tokenization:** Secure sensitive data with AES256 and elliptic curves.\r\n- **Customizable Pipelines:** Create custom transformers and validations.\r\n\r\n## Installation\r\n\r\nInstall the `@cmmv/normalizer` package via npm:\r\n\r\n```bash\r\n$ pnpm add @cmmv/normalizer\r\n```\r\n\r\n## Quick Start\r\n\r\nBelow is an example of parsing and normalizing a JSON file with a custom schema:\r\n\r\n```typescript\r\nimport { \r\n    JSONParser, AbstractParserSchema, \r\n    ToLowerCase, Tokenizer, ToDate, ToObjectId \r\n} from '@cmmv/normalizer';\r\n\r\nimport { ECKeys } from '@cmmv/encryptor';\r\n\r\nconst keys = ECKeys.generateKeys();\r\n\r\nconst tokenizer = Tokenizer({\r\n    publicKey: ECKeys.getPublicKey(keys),\r\n});\r\n\r\nclass CustomerSchema extends AbstractParserSchema {\r\n    public field = {\r\n        id: {\r\n            to: 'id',\r\n            transform: [ToObjectId],\r\n        },\r\n        name: { to: 'name' },\r\n        email: {\r\n            to: 'email',\r\n            validation: /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$/,\r\n            transform: [ToLowerCase],\r\n        },\r\n        phoneNumber: {\r\n            to: 'phone',\r\n            transform: [tokenizer],\r\n        },\r\n        registrationDate: {\r\n            to: 'createdAt',\r\n            transform: [ToDate],\r\n        },\r\n    };\r\n}\r\n\r\nnew JSONParser({\r\n    input: './sample/large-customers.json',\r\n    schema: CustomerSchema,\r\n})\r\n.pipe(data =\u003e console.log('Processed Data:', data))\r\n.once('end', () =\u003e console.log('Parsing Complete'))\r\n.once('error', error =\u003e console.error('Parsing Error:', error))\r\n.start();\r\n```\r\n\r\n## Supported File Formats\r\n\r\nThe ``@cmmv/normalizer`` module includes built-in parsers for the following formats:\r\n\r\n| Format | Parser      | Example File      |\r\n|--------|-------------|-------------------|\r\n| JSON   | JSONParser  | large-data.json   |\r\n| XML    | XMLParser   | large-data.xml    |\r\n| YAML   | YAMLParser  | large-data.yaml   |\r\n\r\nFor custom formats, you can create your own parser by extending the ``AbstractParser``.\r\n\r\n## API Reference\r\n\r\n### AbstractParser\r\n\r\nAll parsers extend the AbstractParser class, providing a consistent interface for processing data:\r\n\r\n| Method            | Description                                                                               |\r\n|--------------------|-------------------------------------------------------------------------------------------|\r\n| `pipe(fn: Function)` | Adds a function to the processing pipeline for transforming or handling parsed data.      |\r\n| `start()`          | Starts the parser and begins processing the input file.                                   |\r\n| `processData(data: any)` | Processes individual data items through the schema and transformation pipeline.         |\r\n| `finalize()`       | Finalizes the parsing operation and emits an end event.                                   |\r\n| `error(err: Error)` | Emits an error event for handling parsing errors.                                          |\r\n\r\n### Built-in Normalizers\r\n\r\nThe module provides several built-in normalizers for transforming data fields:\r\n\r\n| Normalizer    | Description                                             |\r\n|---------------|---------------------------------------------------------|\r\n| `ToBoolean`   | Converts a value to a boolean.                          |\r\n| `ToDate`      | Converts a value to a JavaScript Date object.           |\r\n| `ToFixed`     | Formats a number to a fixed number of decimal places.   |\r\n| `ToFloat`     | Converts a value to a floating-point number.            |\r\n| `ToInt`       | Converts a value to an integer.                         |\r\n| `Tokenizer`   | Encrypts sensitive data with AES256 and elliptic curves.|\r\n| `ToLowerCase` | Converts a string to lowercase.                         |\r\n| `ToObjectId`  | Converts a value to a MongoDB ObjectId.                 |\r\n| `ToString`    | Converts a value to a string.                           |\r\n| `ToUpperCase` | Converts a string to uppercase.                         |\r\n\r\n## Custom Parser\r\n\r\nYou can create a custom parser by extending ``AbstractParser``. Here’s an example for a CSV parser:\r\n\r\n```typescript\r\nimport * as fs from 'node:fs';\r\nimport * as path from 'node:path';\r\nimport * as readline from 'node:readline';\r\nimport { AbstractParser, ParserOptions } from '@cmmv/normalizer';\r\n\r\nexport class CSVParser extends AbstractParser {\r\n    constructor(options?: ParserOptions) {\r\n        super();\r\n        this.options = options;\r\n    }\r\n\r\n    public override start() {\r\n        const inputPath = path.resolve(this.options.input);\r\n\r\n        if (fs.existsSync(inputPath)) {\r\n            const readStream = fs.createReadStream(inputPath);\r\n\r\n            const rl = readline.createInterface({\r\n                input: readStream,\r\n                crlfDelay: Infinity, // Handle all newlines (Windows, Unix)\r\n            });\r\n\r\n            let headers: string[] | null = null;\r\n\r\n            rl.on('line', (line) =\u003e {\r\n                if (!headers) {\r\n                    // First line contains headers\r\n                    headers = line.split(',');\r\n                    return;\r\n                }\r\n\r\n                // Map CSV columns to header fields\r\n                const values = line.split(',');\r\n                const record = headers.reduce((acc, header, index) =\u003e {\r\n                    acc[header.trim()] = values[index]?.trim();\r\n                    return acc;\r\n                }, {});\r\n\r\n                // Process each record\r\n                this.processData.call(this, record);\r\n            });\r\n\r\n            rl.on('close', this.finalize.bind(this));\r\n            rl.on('error', (error) =\u003e this.error.bind(this, error));\r\n        } else {\r\n            console.error(`File not found: ${inputPath}`);\r\n        }\r\n    }\r\n}\r\n```\r\n\r\nThe @cmmv/normalizer module provides a powerful and extensible framework for parsing and normalizing large datasets in various formats. Its tight integration with CMMV contracts and models ensures consistency and scalability, making it an essential tool for building robust Node.js applications.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmmvio%2Fcmmv-normalizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmmvio%2Fcmmv-normalizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmmvio%2Fcmmv-normalizer/lists"}