{"id":15673622,"url":"https://github.com/lmammino/capnp-stream","last_synced_at":"2025-05-06T22:52:34.671Z","repository":{"id":138309942,"uuid":"118563169","full_name":"lmammino/capnp-stream","owner":"lmammino","description":"A Node.js readable stream for Cap’n Proto encoded binary input","archived":false,"fork":false,"pushed_at":"2018-08-19T15:48:25.000Z","size":1883,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T03:41:29.135Z","etag":null,"topics":["binary","capn-proto","capnp","capnproto","encoding","library","node","nodejs","stream"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmammino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-23T05:39:29.000Z","updated_at":"2018-08-19T15:45:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"f8a1650b-6290-49bd-8853-0bc91c153c82","html_url":"https://github.com/lmammino/capnp-stream","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fcapnp-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fcapnp-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fcapnp-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fcapnp-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmammino","download_url":"https://codeload.github.com/lmammino/capnp-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252782470,"owners_count":21803382,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","capn-proto","capnp","capnproto","encoding","library","node","nodejs","stream"],"created_at":"2024-10-03T15:41:34.170Z","updated_at":"2025-05-06T22:52:34.647Z","avatar_url":"https://github.com/lmammino.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# capnp-stream\n\n[![npm version](https://badge.fury.io/js/capnp-stream.svg)](http://badge.fury.io/js/capnp-stream)\n[![CircleCI](https://circleci.com/gh/lmammino/capnp-stream.svg?style=shield)](https://circleci.com/gh/lmammino/capnp-stream)\n[![codecov.io](https://codecov.io/gh/lmammino/capnp-stream/coverage.svg?branch=master)](https://codecov.io/gh/lmammino/capnp-stream)\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n\nA Node.js A readable and writebale stream for [Cap’n Proto](https://capnproto.org/).\n\n\n## Install\n\nWith NPM:\n\n```bash\nnpm install --save capnp-stream\n```\n\n**Note**: capnp binaries need to be available in your system. Check out the official\n[Cap’n Proto install guide](https://capnproto.org/install.html) for OS-specific instructions.\n\n\n## Usage\n\n`capnp-stream` allows to consume Cap’n Proto files as a readable stream, that emits\nobjects.\n\n\n### Example usages:\n\nParse a Cap’n Proto encoded file\n\n```javascript\nconst { createReadStream } = require('fs')\nconst { join } = require('path')\nconst capnp = require('capnp')\nconst { ParseStream } = require('capnp-stream')\n\n// import capnp schema\nconst schema = capnp.import(join(__dirname, 'person.capnp'))\n\n// initialize stream for a given schema\nconst capnpStream = new ParseStream(schema.Person)\n\n// print as a JSON every object in the stream\ncapnpStream.on('data', (d) =\u003e {\n  console.log(JSON.stringify(d, null, 2))\n})\n\n// pipe the data file to the capnp-stream instance\ncreateReadStream('someCapnpDataFile')\n  .pipe(capnpStream)\n```\n\nEncode a incoming objects stream to a Cap’n Proto file:\n\n```javascript\nconst { createWriteStream } = require('fs')\nconst { join } = require('path')\nconst capnp = require('capnp')\nconst { SerializeStream } = require('capnp-stream')\n\n// import capnp schema\nconst schema = capnp.import(join(__dirname, 'person.capnp'))\n\n// initialize stream for a given schema\nconst capnpStream = new SerializeStream(schema.Person)\n\nsomeObjectsStream // a stream that emits JavaScript objects that conform the capnp Schema\n  .pipe(capnpStream) // encodes the javascript objects to a capnp stream\n  .pipe(createWriteStream(join(__dirname, 'somefile.dat'))) // save the encoded capnp data to a file\n```\n\n\n## `skip` and `emitEvery` options (sharding)\n\nSometimes you don't want to consume (emit) all the objects in the capnp stream. For instance, this is common when you are processing the stream in parallel multiple times with multiple processes and you want to easily implement a sharding mechanism.\n\nIn these situations you can take advantage of the `skip` and `emitEvery` options that you can pass at construction time:\n\n```javascript\nconst capnp = require('capnp')\nconst { SerializeStream } = require('capnp-stream')\n\n// import capnp schema\nconst schema = capnp.import(join(__dirname, 'person.capnp'))\n\n// initialize stream for a given schema\nconst capnpStream = new SerializeStream(schema.Person, {\n  skip: process.env.SKIP,\n  emitEvery: process.env.EMIT_EVERY\n})\n```\n\nIn this example, we pass through these options from environment variables. If you set your environment variables to be `SKIP=5` and `EMIT_EVERY=2` you will get the following objects emitted (0 based count):\n\n  - person #5\n  - person #7\n  - person #9\n  - person #11\n  - ...\n\n\n## Contributing\n\nEveryone is very welcome to contribute to this project.\nYou can contribute just by submitting bugs or suggesting improvements by\n[opening an issue on GitHub](https://github.com/lmammino/capnp-stream/issues).\n\n\n## License\n\nLicensed under [MIT License](LICENSE). © Luciano Mammino.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fcapnp-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmammino%2Fcapnp-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fcapnp-stream/lists"}