{"id":19166007,"url":"https://github.com/manojc/xtreamer","last_synced_at":"2025-10-08T19:25:25.428Z","repository":{"id":57402391,"uuid":"137394998","full_name":"manojc/xtreamer","owner":"manojc","description":"Reads XML files using streams.","archived":false,"fork":false,"pushed_at":"2020-08-14T09:47:52.000Z","size":234,"stargazers_count":5,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-16T15:44:01.395Z","etag":null,"topics":["buffer","nodejs","npm","stream","typescript","xml","xml-parser","xml-parsing","xmltojson"],"latest_commit_sha":null,"homepage":"https://runkit.com/manojc/xtreamer-playground","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manojc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-14T18:33:13.000Z","updated_at":"2025-02-01T01:32:47.000Z","dependencies_parsed_at":"2022-09-17T06:52:19.397Z","dependency_job_id":null,"html_url":"https://github.com/manojc/xtreamer","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/manojc/xtreamer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojc%2Fxtreamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojc%2Fxtreamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojc%2Fxtreamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojc%2Fxtreamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manojc","download_url":"https://codeload.github.com/manojc/xtreamer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojc%2Fxtreamer/sbom","scorecard":{"id":615467,"data":{"date":"2025-08-11","repo":{"name":"github.com/manojc/xtreamer","commit":"5af12081c68d9dccc1cc3eb3b5b3ce69a58cca81"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"12 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-v88g-cgmw-v5xw","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-fjxv-7rqg-78g4","Warn: Project is vulnerable to: GHSA-4q6p-r6v2-jvc5","Warn: Project is vulnerable to: GHSA-896r-f27r-55mw","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-vh95-rmgr-6w4m","Warn: Project is vulnerable to: GHSA-xvch-5gv4-984h","Warn: Project is vulnerable to: GHSA-g6ww-v8xp-vmwg","Warn: Project is vulnerable to: GHSA-hrpp-h998-j3pp","Warn: Project is vulnerable to: GHSA-p8p7-x288-28g6","Warn: Project is vulnerable to: GHSA-72xf-g2v4-qvf3"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-21T03:49:37.784Z","repository_id":57402391,"created_at":"2025-08-21T03:49:37.784Z","updated_at":"2025-08-21T03:49:37.784Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000704,"owners_count":26082805,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buffer","nodejs","npm","stream","typescript","xml","xml-parser","xml-parsing","xmltojson"],"created_at":"2024-11-09T09:30:37.153Z","updated_at":"2025-10-08T19:25:25.400Z","avatar_url":"https://github.com/manojc.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Xtreamer\n\n[![npm version][npm-image]][npm-url]\n[![node][node-version-image]][node-version-url]\n[![Build Status][travis-image]][travis-url]\n[![License][license-image]][license-url]\n[![npm][downloads-image]][downloads-url]\n[![Hackage-Deps][dependency-image]][dependency-url]\n[![David][dev-dependency-image]][dev-dependency-url]\n\n\n[travis-url]: https://travis-ci.com/manojc/xtreamer\n[travis-image]: https://travis-ci.com/manojc/xtreamer.svg?branch=master\n\n[npm-url]: https://www.npmjs.com/package/xtreamer\n[npm-image]: https://img.shields.io/npm/v/xtreamer.svg\n\n[license-url]: https://opensource.org/licenses/MIT\n[license-image]: https://img.shields.io/npm/l/make-coverage-badge.svg\n\n[downloads-url]: https://www.npmjs.com/package/xtreamer\n[downloads-image]: https://img.shields.io/npm/dm/xtreamer.svg\n\n[node-version-url]: https://github.com/manojc/xtreamer\n[node-version-image]: https://img.shields.io/badge/node-%3E=6.0-blue.svg?style=flat-square\n\n[dependency-url]: https://www.npmjs.com/package/xtreamer\n[dependency-image]: https://img.shields.io/hackage-deps/v/lens.svg\n\n[dev-dependency-url]: https://www.npmjs.com/package/xtreamer\n[dev-dependency-image]: https://img.shields.io/david/dev/manojc/xtreamer.svg\n\n\n\n- [Xtreamer](#xtreamer)\n  - [Background](#background)\n  - [Why Xtreamer ?](#why-xtreamer)\n  - [Install Package](#install-package)\n  - [APIs](#apis)\n  - [Options](#options)\n  - [Usage](#usage)\n  - [Demo](#demo)\n  - [Test](#test)\n\n## V1.1.0\n\n- Support for JSON transformer option ([demo](https://github.com/manojc/xtreamer/tree/master/demo))\n\n## V1.0.0\n\n- Handling to avoid repeating nodes in comments \u0026 cdata.\n- Minor bug fixes and exception handling.\n- Unit tests to cover corner case scenarios.\n- No breaking changes.\n\n## Background\n\nWhile working on a NodeJs project, I came across a problem of processing huge XML files. Node process can store a limited amout of data after which it throws `out of memory` exception. Thus files more than a specified limit was a pain for the application.\n\nThat is where I found a couple of `npm packages` viz. [xml-flow](https://www.npmjs.com/package/xml-flow) \u0026 [xml-stream](https://www.npmjs.com/package/xml-stream). These packages process the XML files in chunks using streams (which is exactly what I was looking for).\n\nHowever, I came across few drawbacks regarding these packages explained in [this stackoverflow question](https://stackoverflow.com/questions/52129764/xml-flow-npm-package-unexpected-xml-parsing-behaviour). And thus I started working on this package which does exactly what these 2 packages do but it emits xml instead of processed JSON and gives a flexibility of choosing most compatible `xml to json npm packages`.\n\n## Why Xtreamer ?\n\n- One of the key points is `xtreamer` has `stream` as its only dependency.\n\n- As this package uses streams, the memory consumption will always be in control.\n\n- `xtreamer` itself is an extension of a `Transform Stream` so it can be piped to any input `Readable Stream` and `xtreamer` output can be piped to a `Writable Stream`.\n\n- Apart from above points, `xtreamer` provides XML nodes in response which enables it to get hooked up with any `XML-JSON` parsing npm packages as per requirement.\n\n## Install Package\n\n```shell\nnpm i xtreamer --save\n```\n\n## APIs\n\n`xtreamer` extends [Transform Steram Class](https://nodejs.org/api/stream.html#stream_duplex_and_transform_streams) \u0026 provides an additional custom event `xmldata` which emits xml node.\n\n### xtreamer( node : `string`, options?: `object` ) : [stream.Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform)\n\n`xtreamer` function initialises the transform stream. It accepts following 2 arguments -\n\n- `node`: `string`\n- `options`: `object` (coming soon)\n\nThis function return a transform stream which can be triggered by piping it with any readable stream.\n\n### xmldata event\n\nApart from [default steam events](https://nodejs.org/api/stream.html#stream_event_close), `streamer` emits `xmldata` event to emit individual xml nodes.\n\n\u003e From version `0.1.3` onwards, `xtreamer` also supports the conventional `data` event to emit individual xml nodes.\n\n```javascript\nconst xtreamer = require(\"xtreamer\");\n\nconst xtreamerTransform = xtreamer(\"XmlNode\", options);\n\n// listening to `xmldata` event here\nxtreamerTransform.on(\"xmldata\", (data) =\u003e { });\n\n// OR\n\n// `data` event also supported from version 0.1.3\nxtreamerTransform.on(\"data\", (data) =\u003e { });\n```\n\n## Options\n\n### max_xml_size ( default - 10000000 )\n\n`max_xml_size` is maximum the number of characters allowed to hold in memory. \n\n`xtreamer` raises an `error` event in case in memory xml string exceed specified limit. This ensures that the node process doesn't get terminated because of excess in memory data collection.\n\nDefault value of this option restricts the amount of data held in memory to approximately 10Mb. Following snippet shows how to override default value -\n\n```javascript\nconst xtreamer = require(\"xtreamer\");\n\n// overriding `max_xml_size` value here\nconst options = { max_xml_size: 30000000 };\n\n// passing `options` object as second parameter\nconst xtreamerTransform = xtreamer(\"XmlNode\", options);\n```\n\nTypically this value is not needed to override as in most of the cases, size of xml in a single xml node will not exceed 10Mb.\n\n### transformer ( function(xmlString): jsonObj )\n\n\u003e This feature is introduced in version 1.1.0\n\nThe `transformer` option allows to transform the xml node output to desired JSON strucure by hooking it up with the streaming pipeline. **It becomes a very useful feature where JSON parser can be dynamically injected in `xtreamer`.**\n\nThis function is supposed to accept an xml string as a parameter and should return converted valid JSON object in response. **The transformer function can also return a promise** which will be resolved by `xtreamer` before emitting `data` event.\n\nNote that the converted JSON is internally stringified before sending it back to in data event handler. So it is advised that the transformer function should always return valid JSON object in response.\n\nIn case transformer function encounters an error, `xtreamer` emits `error` event and stops the xml conversion process.\n\n## Usage\n\nFollowing code snippet uses `request` NPM package as input readable stream -\n\n```javascript\nconst request = require(\"request\");\nconst xtreamer = require(\"xtreamer\");\n\nconst sampleNode = \"SampleNode\";\nconst sampleUrl = \"http://sample-xml.com/sample.xml\";\nlet count = 0;\n\n// input readable stream with event handlers\nconst readStream = request.get(sampleUrl);\n\n// xtreamer transform stream with custom event handler\nconst xtreamerTransform = xtreamer(sampleNode)\n    .on(\"data\", () =\u003e ++count % 100 || console.log(count))\n    .on(\"end\", () =\u003e console.log(count))\n    .on(\"error\", (error) =\u003e console.error(error));\n\n// input | transform\nreadStream.pipe(xtreamerTransform);\n```\n\nAs `streamer` is a transform stream, one can also pipe the stream with other streams -\n\n```javascript\nconst { Writable } = require(\"stream\");\nconst request = require(\"request\");\nconst xtreamer = require('xtreamer');\n\nclass XtreamerClient extends Writable {\n    _write(chunk, encoding, next) {\n        // do stuff\n        next();\n    }\n}\n\nconst sampleNode = \"SampleNode\";\nconst sampleUrl = \"http://sample-xml.com/sample.xml\";\n\n// input readable stream\nconst readStream = request.get(sampleUrl);\n\n// xtreamer transform stream with custom event handler\nconst xtreamerTransform = xtreamer(sampleNode)\n    .on(\"error\", (error) =\u003e console.error(error));\n\n// input | transform | write\nreadStream.pipe(xtreamerTransform).pipe(new XtreamerClient());\n```\n\nCheck the [demo](https://github.com/manojc/xtreamer/tree/master/demo) for more examples which includes -\n\n- demo.js - emits xml nodes\n\n- transformer-demo.js - emits stringified JSON \u0026 uses `xml-js` package within `transformer function`\n\n- transformer-promise-demo.js - emits stringified JSON \u0026 uses `xml-js` package within `transformer function` that return a promise.\n\n## Demo\n\n```shell\nnpm i \u0026\u0026 npm start\n```\n\n## Test\n\n```shell\nnpm i \u0026\u0026 npm run test\n```\n\n---\n\n# Author\n\n- Author - Manoj Chalode (chalodem@gmail.com)\n\n- Copyright - github.com/manojc\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanojc%2Fxtreamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanojc%2Fxtreamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanojc%2Fxtreamer/lists"}