{"id":28712241,"url":"https://github.com/joyent/node-vstream","last_synced_at":"2025-06-14T23:01:53.062Z","repository":{"id":19371511,"uuid":"22611803","full_name":"TritonDataCenter/node-vstream","owner":"TritonDataCenter","description":"instrumented streams","archived":false,"fork":false,"pushed_at":"2021-11-02T20:28:33.000Z","size":24,"stargazers_count":59,"open_issues_count":6,"forks_count":4,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-06-10T08:03:54.847Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TritonDataCenter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-04T16:37:28.000Z","updated_at":"2024-11-15T13:22:45.000Z","dependencies_parsed_at":"2022-09-14T12:10:43.280Z","dependency_job_id":null,"html_url":"https://github.com/TritonDataCenter/node-vstream","commit_stats":null,"previous_names":["joyent/node-vstream"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/TritonDataCenter/node-vstream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TritonDataCenter%2Fnode-vstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TritonDataCenter%2Fnode-vstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TritonDataCenter%2Fnode-vstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TritonDataCenter%2Fnode-vstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TritonDataCenter","download_url":"https://codeload.github.com/TritonDataCenter/node-vstream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TritonDataCenter%2Fnode-vstream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259896112,"owners_count":22928325,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-14T23:01:23.863Z","updated_at":"2025-06-14T23:01:53.030Z","avatar_url":"https://github.com/TritonDataCenter.png","language":"JavaScript","readme":"# node-vstream: instrumented streams\n\nWhen working with Node streams, particularly object-mode streams, it's often\nhelpful to be able to inspect a pipeline.  This module instruments objects to\nprovide:\n\n**For all objects**:\n\n* a name: used in debug output and especially useful for data pipelines\n* custom counters: create your own counters for events of interest like\n  errors, requests handled, messages sent, or the like\n* custom warnings: Error objects that are counted, forwarded to subscribers, but\n  otherwise ignored\n\n**For all streams**:\n\n* forward and back pointers: vstream watches `pipe` events and records upstreams\n  and downstreams.  You can walk to the head of a pipeline and iterate\n  downstream streams.\n* debug methods to dump stream state, including high watermarks and data\n  buffered on the upstream and downstream sides\n\n**For object-mode transform streams**:\n\n* automatic counters for inputs processed and outputs emitted\n* provenance: history information for objects passed through the pipeline (so\n  you can report errors with precise information, like \"object X from line N\")\n\n## Usage\n\n### Instrumenting a simple pipeline\n\nYou can instrument one or more regular Node streams by calling\n`vstream.wrapStream` on them.  wrapStream returns the same stream, but attaches\na few new functions (and private properties).  Here's a simple pipeline that\nreads data from \"/usr/bin/more\", sends it through a pass-through stream, and\nthen writes it to \"/dev/null\":\n\n```javascript\nvar instream, passthru, outstream;\ninstream = vstream.wrapStream(fs.createReadStream('/usr/bin/more'), 'source');\npassthru = vstream.wrapStream(new stream.PassThrough(), 'passthru');\noutstream = vstream.wrapStream(fs.createWriteStream('/dev/null'), 'devnull');\n\ninstream.pipe(passthru);\npassthru.pipe(outstream);\n```\n\nAs an example, we'll attach a listener to each `'data'` event that dumps the\ndebug information from each stream in the pipeline.  The debug information\nincludes the stream's name, what kind of stream it is (readable, writable, or\nduplex), the amount of data buffered, and the high watermark.  We'll also dump\nthis when the pipeline finishes (when the last stream emits `'finish'`).\n\n```javascript\ninstream.on('data', report);\noutstream.on('finish', report);\n\nfunction report()\n{\n\tvar head = outstream.vsHead();\n\tassert.ok(head == instream);\n\thead.vsWalk(function (stream) { stream.vsDumpDebug(process.stdout); });\n\tconsole.error('-----');\n}\n```\n\nOn my system, this prints:\n\n```\nsource               (readable, rbuf: 0/65536)\npassthru             (duplex, wbuf: 0/16384, rbuf: 0/16384)\ndevnull              (writable, wbuf: 65536/16384)\n-----\nsource               (readable, rbuf: 0/65536)\npassthru             (duplex, wbuf: 0/16384, rbuf: 65536/16384)\ndevnull              (writable, wbuf: 65536/16384)\n-----\nsource               (readable, rbuf: 0/65536)\npassthru             (duplex, wbuf: 0/16384, rbuf: 6640/16384)\ndevnull              (writable, wbuf: 65536/16384)\n-----\nsource               (readable, rbuf: 0/65536)\npassthru             (duplex, wbuf: 0/16384, rbuf: 0/16384)\ndevnull              (writable, wbuf: 0/16384)\n-----\n```\n\n\"source\", \"passthrough\", and \"devnull\" are the names we gave these streams,\nwhich are readable, duplex, and writable, respectively.  Notice that the default\nhigh watermark for a readable file stream (\"source\") is 64K, or 65536 bytes.\nThe default high watermarks for the PassThrough and the writable file streams\nare 16K, or 16384 bytes.\n\nSeeing this while writing this example, I picked /usr/bin/more because it's\njust over 128K.  On my system, this causes Node to read two full 64K chunks,\nplus one smaller chunk.  This makes for an interesting example:\n\n* When we get the first `data` event (the first printout above), the data has\n  already been written to the PassThrough, which wrote it downstream to the\n  /dev/null stream.  The /dev/null stream has it all buffered, since it hasn't\n  had a chance to do I/O yet.  The /dev/null stream buffered all 64K, not just\n  16K, presumably because it came in as one chunk.  (Remember that the high\n  watermark is a guideline, but it's possible to buffer more bytes than that --\n  as we see here.)\n* When we get the second `data` event (the second printout above), the data is\n  still buffered on the /dev/null stream and has backed up to the PassThrough.\n  For whatever reason, the /dev/null stream isn't keeping up with the source\n  read stream, and we can see the buffering here.  This is flow control\n  (backpressure) in action.\n* At this point, if the /dev/null stream were totally blocked, we'd expect the\n  PassThrough stream to buffer at least 16K on the *write* side as well, then\n  the source would back up on its readable side, and then we'd stop reading from\n  the original file.\n* This doesn't happen, because by the time we get the third `data` event, the\n  /dev/null stream must have written some data out, because the PassThrough has\n  fewer bytes buffered.\n* When the pipeline has finished (the last printout), there are no bytes\n  buffered anywhere.\n\nThe example's pretty trivial, but you can learn a lot about Node streams by\nunderstanding what's going on at each stage.  Of course, the real point of\n`vstream` isn't to demonstrate this, but to help debug situations where things\n*aren't* going as expected.  The debug information can help you understand where\ndata has buffered up way more than you wanted (usually a memory leak) or where\nthe pipeline's plugged up.\n\n\n### Instrumenting a transform stream\n\nWhen you wrap Transform streams, vstream modifies `_transform`, `_flush`, and\n`push` to keep track of which outputs were generated by which inputs.  You can\nuse this to generate useful error messages when you encounter bad input.  Here's\na little program that reads /etc/passwd and emits a warning when it finds the\n\"nobody\" user:\n\n```javascript\nvar instream, linestream, mystream;\nvar user = 'nobody';\n\n/*\n * Read the contents of /etc/passwd and chunk it into lines.\n */\ninstream = vstream.wrapStream(fs.createReadStream('/etc/passwd'), 'source');\nlinestream = vstream.wrapTransform(new lstream());\n\n/*\n * Pipe the lines into a stream that looks for the \"nobody\" user and emits a\n * warning, which will include the line number.\n */\nmystream = new stream.Transform({ 'objectMode': true });\nmystream._transform = function myTransform(line, _, callback) {\n\tif (line.substr(0, user.length + 1) == user + ':') {\n\t\tthis.push(user);\n\t\tthis.vsWarn(new Error('found \"' + user + '\"'), 'nfoundusers');\n\t}\n\n\tcallback();\n};\nmystream = vstream.wrapTransform(mystream, 'UserSearcher');\n\ninstream.pipe(linestream);\nlinestream.pipe(mystream);\n\nmystream.on('warn', function (context, kind, error) {\n\tconsole.log('kind:    %s', kind);\n\tconsole.log('error:   %s', error.message);\n\tconsole.log('context: %s', context.label());\n\n\tmystream.vsHead().vsWalk(function (s) {\n\t\ts.vsDumpDebug(process.stdout);\n\t});\n});\n```\n\nWhen I run this on OS X Mavericks, I get:\n\n```\nkind:    nfoundusers\nerror:   found \"nobody\"\ncontext: UserSearcher input 11 from LineStream input 1: value 'nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false'\nsource               (readable, rbuf: 0/65536)\nLineStream           (duplex, wbuf: 1/16384, rbuf: 0/16384)\n    ninputs:         1\n    noutputs:        11\nUserSearcher         (duplex, wbuf: 1/16384, rbuf: 1/16384)\n    nfoundusers:     1\n    ninputs:         11\n    noutputs:        1\n```\n\nThe context for each warning keeps track of the history through all\nvstream-wrapped Transform streams.  vstream also bumps a counter for each\nwarning, which is why \"nfoundusers\" is 1.\n\n## PipelineStream\n\nvstream also provides a PipelineStream class, which takes an array of streams\nthat should be piped into each other (A piped to B piped to C) and wraps them\ninto a single stream P.  Writes to P are directed to A, and reads from P read\nfrom C.  This is mainly useful when you want to expose a single stream that's\nlogically made up of a couple of existing streams.\n\n\n## Design notes\n\nA key design principle for this module is that it should be possible to\nusefully instrument streams that were not written to support this module.  Some\nfeatures (like bumping custom counters or emitting warnings) may require adding\ncode, but basic features should work without special support, and instrumenting\na stream should not break existing functionality.  That's why this module is\nimplemented (somewhat regrettably) by attaching properties to existing objects\nrather than requiring users to inherit from a custom class.\n\nPublic properties attached to instrumented objects use camel case and start with\na \"vs\" prefix to avoid colliding with other properties a user might be using.\nSimilarly, private properties use snake case with a \"vs\\_\" prefix.\n\n\n## Caveats\n\n### Memory usage\n\nvstream tracks pipes to add upstream and downstream references, but it does not\ntrack unpipes to remove these references.  That's because streams are unpiped\nwhen the end-of-stream is reached, but this is often when it's most useful to\ndebug the pipeline, so it's useful to keep these references around.  As a\nresult, if you keep a reference to any stream in the pipeline, you'll likely\nhave references to the whole pipeline.  If you implement a stream that makes\nheavy use of pipe and unpipe, you may end up referencing more memory than you'd\nexpect.  In all cases, once all references to all streams in the pipeline have\nbeen removed, all of these objects can be garbage collected.\n\n### API stability\n\nThe stream debugging information relies on accessing the internal Node state of\nthe stream, which is not guaranteed to remain stable across Node releases.  It's\npossible that Node upgrades could break this behavior.  If you notice incorrect\noutput or unexpected crashes, please file an issue.\n\n\n## Contributions\n\nContributions welcome, and should be 'make prepush' clean.  The prepush checks\nuse [javascriptlint](https://github.com/davepacheco/javascriptlint) and\n[jsstyle](https://github.com/davepacheco/jsstyle).\n","funding_links":[],"categories":["Repository","Packages","包","Debugging / Profiling","目录"],"sub_categories":["Debugging","Debugging / Profiling","调试 / 分析","调试/分析","调试"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoyent%2Fnode-vstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoyent%2Fnode-vstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoyent%2Fnode-vstream/lists"}