{"id":28068596,"url":"https://github.com/nahratzah/objpipe","last_synced_at":"2025-05-12T17:35:01.449Z","repository":{"id":48744759,"uuid":"127082498","full_name":"nahratzah/objpipe","owner":"nahratzah","description":"Collection streaming utilities.","archived":false,"fork":false,"pushed_at":"2020-04-04T01:00:29.000Z","size":164,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-29T13:41:28.722Z","etag":null,"topics":["cplusplus","cplusplus-17","mapreduce","pushpull","streaming"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nahratzah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-28T04:12:02.000Z","updated_at":"2022-07-23T09:54:05.000Z","dependencies_parsed_at":"2022-09-19T09:41:39.415Z","dependency_job_id":null,"html_url":"https://github.com/nahratzah/objpipe","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nahratzah%2Fobjpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nahratzah%2Fobjpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nahratzah%2Fobjpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nahratzah%2Fobjpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nahratzah","download_url":"https://codeload.github.com/nahratzah/objpipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253788502,"owners_count":21964535,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cplusplus","cplusplus-17","mapreduce","pushpull","streaming"],"created_at":"2025-05-12T17:34:49.851Z","updated_at":"2025-05-12T17:35:01.439Z","avatar_url":"https://github.com/nahratzah.png","language":"C++","readme":"[![Build Status](https://travis-ci.org/nahratzah/objpipe.svg?branch=master)](https://travis-ci.org/nahratzah/objpipe)\n\n# objpipe\n\nObjpipe is a library for iterating over collections, performing transformations,\nand doing all that in a type agnostic manner.\n\n## Requirements\n\nC++17, although you'll probably get by with anything higher than C++14 (clang-4.0.0 with -std=c++1z works fine for me).\n\n## Usage\n\nAn objpipe has three components:\n1. A source of data.\n2. A series of transformations on the data.\n3. Accessing the data, for example via iteration, reduction, or pushing it onward.\n\nObjpipe allows interface boundaries between each of those.\n\nObjpipe provides all of:\n1. A range concept (like\n   [Boost iterator](http://www.boost.org/doc/libs/1_66_0/libs/iterator/doc/index.html),\n   [Boost range](http://www.boost.org/doc/libs/1_66_0/libs/range/doc/html/index.html),\n   and [Range-v3](https://github.com/ericniebler/range-v3))\n   and associated ability to transform and iterate this range.\n2. A push interface (like\n   [Reactive Extensions for C++](https://github.com/Reactive-Extensions/RxCpp)).\n3. An interface to erase underlying implementation details.\n4. The ability to select between the using the range concept, or push interface,\n   *at the call site*, instead of at the implementation point.\n\n## Rationale\n\nI have a lot of files that model collections of data.\nAnd there are different file formats.\nAnd I need to iterate over all this data.\n\nA collection interface wouldn't work,\nbecause it would pin down the iterator type.\nIn the example below, the iterator type necessitates all files\nto contain a vector:\n\n    class FileInterface {\n     public:\n      // The interface now imposes a specific collection on all\n      // implementations, in this case std::vector\u003cint\u003e.\n      virtual std::vector\u003cint\u003e::iterator begin() const = 0;\n      virtual std::vector\u003cint\u003e::iterator end() const = 0;\n    };\n\nThe standard way of dealing with this, is to use a callback function.\nHowever, this makes it very hard to do composition, as each callback\nneeds to wrap the composing callback, leading to hard to understand code.\n\n    class FileInterface {\n     public:\n      virtual void visit(std::function\u003cvoid(int)\u003e callback) = 0;\n    };\n\n    class OnlyEvenNumbersFile\n    : public FileInterface\n    {\n     public:\n      // This function becomes rather hard to understand, if there\n      // is a more complex set of operations going on!\n      virtual void visit(std::function\u003cvoid(int)\u003e callback) override {\n        file-\u003evisit(\n            [callback](int x) {\n              if (x % 2 == 0)\n                callback(x);\n            });\n      }\n\n     private:\n      std::shared_ptr\u003cFileInterface\u003e file;\n    };\n\nInstead of this, I wanted to decouple the interface and implementation,\nwhile still keeping the code readable.\nObjpipe does this, by allowing functional (read-only) access to data,\nabstracting away the implementation and life time considerations.\n\nThis way, each implementation can deal with data in the way that's best\nsuited for it.\nWhile composition can be achieved by applying transformations on the objpipe.\n\n    class FileInterface {\n     public:\n      virtual objpipe::reader\u003cint\u003e getData() const = 0;\n    };\n\n    class VectorFile\n    : public std::enable_shared_from_this\u003cVectorFile\u003e,\n      public FileInterface\n    {\n     public:\n      // Not a file, just a collection with data.\n      virtual objpipe::reader\u003cint\u003e getData() const override {\n        return objpipe::of(this-\u003eshared_from_this())\n            .deref()\n            .transform(\u0026VectorFile::data)\n            .iterate();\n      }\n\n     private:\n      std::vector\u003cint\u003e data;\n    };\n\n    class StreamFile\n    : public FileInterface\n    {\n     public:\n      // Read directly from a file.\n      virtual objpipe::reader\u003cint\u003e getData() const override {\n        std::fstream file(fileName, std::ios_base::in);\n\n        // objpipe::new_array iterates over the range at construction time.\n        // It's possible to delay this until the objpipe is evaluated,\n        // but since file is not iterable by itself, this would require some\n        // code, which is beyond the scope of this example.\n        return objpipe::new_array(\n            std::istream_iterator\u003cint\u003e(file)\n            std::istream_iterator\u003cint\u003e());\n      }\n\n     private:\n      std::string fileName;\n    };\n\n    class ManyFiles\n    : public FileInterface\n    {\n     public:\n      // Composition pattern, that concatenates contents from multiple files.\n      virtual objpipe::reader\u003cint\u003e getData() const override {\n        objpipe::of(files) // Copy, since passed by lvalue reference.\n            .iterate()\n            .transform(\n                [](const std::unique_ptr\u003cFileInterface\u003e\u0026 file_ptr) {\n                  return file_ptr-\u003egetData();\n                })\n            .iterate(); // Unpack the collections from getData().\n      }\n\n     private:\n      std::vector\u003cstd::unique_ptr\u003cFileInterface\u003e\u003e files;\n    };\n\nAs can be seen, this allows for a wide variety of implementations, all of\nwhich are hidden behind a common interface.\n\nIn addition, the caller of the ``FileInterface::getData()`` method is free\nto decide how to handle the data:\n1. they could iterate over it\n2. they could copy it into a new collection\n3. they could decide to go for a callback after all (but now, the choice is not imposed by the interface)\n4. etc.\n\n### Example: simple transformation\n\nThis example computes the squares of 1, 2, and 3.\n\n    #include \u003cobjpipe/of.h\u003e\n\n    objpipe::of(1, 2, 3) // Create a source with three numbers.\n        .transform([](int x) { return x * x; }) // Compute squares.\n        .to_vector(); // returns std::vector\u003cint\u003e({ 1, 4, 9 }).\n\n### Example: traversing interface boundaries\n\nThis example shows how the source can be abstracted away.\n\nThe ``objpipe::reader\u003cT\u003e`` allows us to abstract away a specific implementation\nbehind its interface.\nWithout conversion to ``objpipe::reader\u003cT\u003e``, the entire template of operations\nwould be visible.\n\n    #include \u003cvector\u003e\n    #include \u003ccstdint\u003e\n    #include \u003cobjpipe/reader.h\u003e\n    #include \u003cobjpipe/array.h\u003e\n    #include \u003cobjpipe/callback.h\u003e\n\n    class MyIntf {\n     public:\n      virtual auto makeData() -\u003e objpipe::reader\u003cint\u003e = 0;\n    };\n\n    class MyVectorImpl\n    : public MyIntf\n    {\n     public:\n      virtual auto makeData() -\u003e objpipe::reader\u003cint\u003e override {\n        return objpipe::new_array(data.begin(), data.end());\n      }\n\n     private:\n      std::vector\u003cint\u003e data;\n    };\n\n    class MyCallbackImpl\n    : public MyIntf {\n     public:\n      virtual auto makeData() -\u003e objpipe::reader\u003cint\u003e override {\n        return objpipe::new_callback\u003cint\u003e(\n            [](auto\u0026 cb) {\n              for (int i = 0; i \u003c 42; ++i)\n                cb(i);\n            });\n      }\n    };\n\n    std::uintmax_t countEvenNumbers(MyIntf\u0026 src) {\n      return src.makeData()\n          .filter([](int x) { return x % 2 == 0; })\n          .count();\n    }\n\n### Example: map reduce\n\nVarious ways of computing the maximum value.\n\n    #include \u003cfuture\u003e\n    #include \u003coptional\u003e\n    #include \u003cobjpipe/of.h\u003e\n    #include \u003cobjpipe/push_policies.h\u003e\n\n    // Compute immediately.\n    std::optional\u003cint\u003e max = objpipe::of(3, 1, 2)\n        .max();\n\n    // Use asynchronous reduction.\n    std::future\u003cstd::optional\u003cint\u003e\u003e max_future = objpipe::of(3, 1, 2)\n        .async()\n        .max();\n\n    // Use parallel reduction.\n    std::future\u003cstd::optional\u003cint\u003e\u003e max_future = objpipe::of(3, 1, 2)\n        .async(objpipe::multithread_push())\n        .max();\n\nAnother map reduce, that collects all elements into a set.\n\n    #include \u003cset\u003e\n    #include \u003cutility\u003e\n\n    // Generic reduction.\n    std::future\u003cstd::shared_ptr\u003cstd::set\u003cint\u003e\u003e\u003e int_set = objpipe::of(3, 1, 2)\n        .async(objpipe::multithread_unordered_push())\n        .reduce(\n            []() -\u003e { // Factory for initial state of reducer.\n              return std::set\u003cint\u003e();\n            },\n            [](std::set\u003cint\u003e\u0026 state, int\u0026\u0026 element) { // Value acceptor functor.\n              state.insert(std::move(element));\n            },\n            [](std::set\u003cint\u003e\u0026 state, std::set\u003cint\u003e\u0026\u0026 to_add) { // Merge operation across states.\n              state.merge(std::move(to_add));\n            },\n            [](std::set\u003cint\u003e\u0026\u0026 state) { // Extract result from reduction.\n              return std::make_shared\u003cstd::set\u003cint\u003e\u003e(std::move(state));\n            });\n\n## Linking\n\nThis is a header-only library.\n\nTo use it, you can simple supply ``-I/path/to/objpipe/include`` to the\ncompiler.\nThe callback requires that boost/context be linked in and\nthat boost/coroutines2 is on the include path.\n\nIf you're using CMake, you can instead import the library using (for instance)\na git submodule:\n\n    add_subdirectory(path/to/objpipe)\n    target_link_libraries(my_target objpipe)\n\nIf objpipe is properly installed, the following should work:\n\n    find_package(objpipe 0.0 REQUIRED)\n    target_link_libraries(my_target objpipe)\n\n(objpipe is an interface library, so it does not have an actual library,\nbut does contain options to link correctly (thread-safe) and get the correct\nincludes via the target.)\nIn both cases, the include directories are set correctly.\n\n## Related work\n\n1. [Boost iterator](http://www.boost.org/doc/libs/1_66_0/libs/iterator/doc/index.html)\n   provides an alternative for transforming and filtering.\n   Contrary to objpipe,\n   it also allows for the iterator to modify the collection it iterates over (under certain conditions)\n   and usually maintains the iterator category.\n2. [Boost range](http://www.boost.org/doc/libs/1_66_0/libs/range/doc/html/index.html)\n   provides a range interface, similar to the pull interface of objpipe.\n   Similar to boost iterator (mentioned above) and Range-v3 (mentioned below) it preserves the iterator category.\n3. [Reactive Extensions for C++](https://github.com/Reactive-Extensions/RxCpp)\n   implements transformations and iteration.\n   Like objpipe's push interface, it is based on streaming values.\n   In addition, its observables allow for multiple subscribers, while objpipe always uses a single subscriber.\n   Unlike objpipe, it does not have a pull based interface.\n4. [Range-v3](https://github.com/ericniebler/range-v3)\n   conceptualizes sequences, like objpipe does.\n   Range-v3 ranges do not own their data, while objpipe does own its data (with some explicit exceptions).\n   Unlike objpipe, it does not seem to have an abstraction of a range, like objpipe::reader provides.\n5. A combination of\n   [STL algorithm](http://en.cppreference.com/w/cpp/header/algorithm) calls, such as\n   [std::transform](http://en.cppreference.com/w/cpp/algorithm/transform),\n   [std::copy\\_if](http://en.cppreference.com/w/cpp/algorithm/copy),\n   can be used to achieve the same transformation effects, but would lose the streaming property.\n6. Range-based-for iteration (``for (auto item : collection)``) with a code block can achieve the same effect.\n7. [Java streams](https://docs.oracle.com/javase/8/docs/api/java/util/stream/package-summary.html#package.description)\n   does the same thing as the above, but for Java, instead of C++.\n   In particular, it's (parallelizable) reduction operations inspired the way of reducing that is used in objpipe.\n\n## Additional Documentation\n\nDocumentation can be generated by the ``objpipe-doc`` target.\n\nOr alternatively, can be [browsed online](https://www.stack.nl/~ariane/objpipe/).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnahratzah%2Fobjpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnahratzah%2Fobjpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnahratzah%2Fobjpipe/lists"}