{"id":21483312,"url":"https://github.com/schveiguy/iopipe","last_synced_at":"2026-01-03T20:08:11.906Z","repository":{"id":146896406,"uuid":"50388813","full_name":"schveiguy/iopipe","owner":"schveiguy","description":"D language library for modular io","archived":false,"fork":false,"pushed_at":"2024-05-24T19:42:18.000Z","size":413,"stargazers_count":77,"open_issues_count":19,"forks_count":6,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-14T19:16:37.022Z","etag":null,"topics":["dlang"],"latest_commit_sha":null,"homepage":"","language":"D","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/schveiguy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-01-25T23:35:07.000Z","updated_at":"2024-09-18T20:28:31.000Z","dependencies_parsed_at":"2023-10-20T22:15:06.396Z","dependency_job_id":null,"html_url":"https://github.com/schveiguy/iopipe","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schveiguy%2Fiopipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schveiguy%2Fiopipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schveiguy%2Fiopipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schveiguy%2Fiopipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/schveiguy","download_url":"https://codeload.github.com/schveiguy/iopipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244006303,"owners_count":20382444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dlang"],"created_at":"2024-11-23T12:46:13.325Z","updated_at":"2026-01-03T20:08:11.877Z","avatar_url":"https://github.com/schveiguy.png","language":"D","funding_links":[],"categories":[],"sub_categories":[],"readme":"# iopipe [![Build Status](https://travis-ci.org/schveiguy/iopipe.svg?branch=master)](https://travis-ci.org/schveiguy/iopipe)\nD language library for modular io\n\nAPI documentation: http://schveiguy.github.io/iopipe\n\niopipe is an input/output library for the D programming language that strives to be as close as\npossible to the unix shell \"pipe\" io specification.\n\n## Unix pipes model\nIn a unix shell, one \"pipes\" a command to another command using the pipe character `|`. For example\n```\n# find -name 'hello*' | grep world\n./blah/hello_world\n```\n\nSuch a command pipes the output of the `find` command to the input of the `grep` command\n\nIn D, a very elegant set of constructs, called ranges, can use this same type of mechanism\nto \"wrap\" one range with other ranges, similarly to\nbuilding a pipeline of i/o with the unix shell.\n\n```D\nforeach(a; someArray.retro.map!(a =\u003e a * 3).filter!(a =\u003e a % 100 != 0))\n{\n   // a will consist of multiples of 3 from someArray, in reverse order,\n   // but that are not also multiples of 100\n   ...\n}\n```\n\nThe nice thing about this is that the pipeline is compiler-generated code, evaluated lazily. This\nmeans, no new arrays are created, and the elements are generated on-demand when asked for. In addition,\nsince the pipeline is created at compile-time, it can all be optimized into the most efficient code possible.\n\niopipe attempts the same thing with buffered stream data.\n\nFor example:\n```D\nimport iopipe.textpipe;\nimport iopipe.zip;\nimport iopipe.bufpipe;\nimport std.io;\nimport std.typecons;\n\n// open a zipfile, decompress it, detect the text encoding inside, and process\n// lines that contain \"foo\"\nvoid main(string[] args)\n{\n    File(args[1])               // open a file\n         .refCounted            // File can't be copied\n         .bufd                  // buffer it\n         .unzip                 // decompress it\n         .runEncoded!((input) { // detect the text encoding and process it.\n           import std.algorithm: filter, canFind;\n           import std.stdio: writeln;\n           foreach(line; input.byLineRange!false.filter!(a =\u003e canFind(a, \"foo\")))\n               writeln(\"this line contains foo: \", line);\n         });\n}\n```\n\n## Basic Iopipe\n\nA basic iopipe has 3 primitive functions which can be called. These are the\nfunctions by which you can manipulate and process data. Typically, one is given\nan iopipe which may need adjustments or reinterpretation, and it is simply a\nmatter of wrapping that iopipe in converters or processors that effect the\ndesired type of data, format of data, or rate of data.\n\nA string of wrapped iopipes is called a *chain*. Most iopipes will use the\nmember name `chain` to denote the source iopipe from which data is retrieved.\n\n### `SomeRange window`\n\nThis property gives a view into the data of the iopipe. The range should be a\nrandom access, non-infinite range. For purposes of the iopipe library, narrow\ncharacter arrays are considered random access ranges of that code unit type.\n\nWrapping iopipes may return a subset of the wrapped window, or return the\nwindow mapped into a different type.\n\nNOTE: there are some assumptions in functions of the iopipe library that\nall windows are arrays. Although most things *should* work with non-array\nwindows, it has not been thoroughly tested. Please file any issues if you have\na use case for a non-array window!\n\n### `void release(size_t elements)`\n\nRelease the given number of elements from the beginning of the window. The\niopipe is *required* to release the specified data, such that the data no\nlonger appears in the window (how it accomplishes this may still hold the data\nin the buffer somehow). To release more elements than are in the window results\nin undefined behavior.\n\nPrevious calls or accesses to `window` should be discarded. Release is allowed\nto change the data returned by `window`.\n\n### `size_t extend(size_t elements)`\n\nExtend the current window's end by the given number of elements. If the specified\nnumber of elements is 0, then the iopipe should extend the optimal number of\nelements if it can. This should attempt to extend *at least* one element.\n\nReturns the number of elements extended. If no data can be extended, the return\nvalue is 0, and it is considered the end of the stream in *most* cases. In some\ncases, you may receive a 0 when an upstream valve is holding back some of the\ndata (this is defined by whomever implemented the valve). You are allowed to\nattempt to extend an iopipe even when a previous call to extend returned 0.\n\n## More Concepts\n\n### Sources\n\nMost iopipes start with a source. A source is a type that provides a `read`\nmember, accepting a buffer that is filled in with data from a data stream, and\nreturns how many elements were read. A `BufferManager` is used to manage the\nallocation of the data, and turn the buffered data into a proper iopipe. The\niopipe library provides two types of Buffers that can be managed -- an\n`AllocatedBuffer` type that uses an `Allocator` from std.experimental to manage\nthe allocations for the buffer, and a `RingBuffer` type which is a very fast\nversion of a Circular Buffer. Note that the RingBuffer type is posix-only, but\nWindows support will be added later.\n\nAs of this release, iopipe provides 2 basic sources, a `NullDev` which provides\nuninitialized data, and a `ZeroDev` which provides zeroed data.\n\n### Sinks\n\nA Sink can take a buffer of data and write it to an external location (such as\na file or array). A sink is simply a type that defines a `write` member,\naccepting the buffer to be written, and returning how many elements were\nwritten.\n\nThe `outputPipe` function is the only iopipe wrapper that accepts a Sink. It\nactually is simply another iopipe, and can be wrapped further for more\nprocessing if necessary. It writes to the sink as data is extended BEFORE\nproviding the data further down the chain.\n\n### IODev\n\nThe `IODev` class has been deprecated, and is now an alias to Martin Nowak's\n[std.io](https://github.com/MartinNowak/io) library. The dependency on std.io\nis optional, and if not included, the `IODev` and related functions are only\npresent if your project depends on std.io.\n\nHowever, even when depending on std.io, instead of using `openDev` or `IODev`,\nit is preferred to use std.io to open streams and then build iopipes on top of\nthose.\n\nNote a few things:\n\n1. Because of the reliance on std.io and a quirk in the comipler that was fixed\n   recently, this arrangement only builds on DMD 2.080.1 and later. If you need\n   support for earlier compilers, use 0.0.4 of iopipe or earlier. If you use\n   0.1.0 or later of iopipe and an earlier compiler, it will not link if you\n   use `IODev`.\n2. A feature of `IODev` that is not in std.io is the ability to use a `FILE *`\n   or file descriptor and not close it when the class is destroyed.\n3. std.io more sensibly uses non-copyable structs instead of classes for\n   lifetime management. Because iopipe generally copies things around even\n   though it's only going to use one copy eventually, you may need to wrap the\n   IOs in ref counting or a class (both are supported by std.io).\n\n### Valves\n\nA Valve is a control point along the iopipe chain. The concept is that you can\nuse a valve to access some nested wrapped piece of the iopipe to change\nparameters or effect certain behavior. The output system relies completely on\nvalves to work properly. The closest such item is accessed by using the `valve`\nproperty of the iopipe (which must return by reference). If no more valves\nexist, then this will not compile.\n\nAll wrapping iopipes that do not define a new valve must provide access to the\nnext upstream valve if it exists. This is essential to writing a proper iopipe\nwrapper. There is a convenience mixin to allow this to happen automatically.\n\n### Rebuffering\n\nA concept not formalized in any type, but nonetheless important for iopipe,\nwhen it is difficult or impossible to translate data in-place, it becomes\nnecessary to copy the data from its source format to a new buffer. This is done\nby transforming the iopipe into a Source, and then using a new BufferManager to\nbuffer the data.\n\nNote that a wrapping type can be both an iopipe AND a source, giving\nflexibility whenever possible.\n\nFor examples of how this is done, see `iopipe.zip` and `iopipe.bufpipe.iosrc`.\n\n### Construction\n\nWhen constructing an iopipe, each wrapping function is passed a *copy* of the\nprevious chain. This means that all iopipes must be copyable *at least* before\nprocessing begins. It is expected to treat each result of a wrapper uniquely,\nand not make copies of half constructed data.\n\nIn some cases, it's necessary to have a single instance of an iopipe's\ninternals. This is to either adhere to some low-level library requirements or\nto properly release resources. In this case, `std.typecons.RefCounted` is used\nto fill this task.\n\n## Examples\n\nTake a look at the example programs in the examples subdirectory.\n* byline - A program that can read any text file of UTF8, UTF16, or UTF32\n  encoding, and output the line lengths to the standard output stream (this\n  uses `std.stdio.writeln` to do this for now).\n* convert - Takes the standard input of any text encoding, and a parameter of\n  the type of encoding to output,\n  and converts the input to the output, adding a BOM if necessary.\n* zip - Compress the standard input to the standard output.\n* unzip - Decompress the standard input to the standard output.\n* search - Print lines that match given search terms, with some context lines\n  surrounding the line.\n  \n## Building\n\niopipe is built with [dub](http://code.dlang.org). To build the examples, use\nthe dub package command line:\n\n`dub build :examplename`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschveiguy%2Fiopipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fschveiguy%2Fiopipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschveiguy%2Fiopipe/lists"}