{"id":25738710,"url":"https://github.com/iterait/hipipe","last_synced_at":"2025-05-08T04:56:05.274Z","repository":{"id":33113025,"uuid":"150424803","full_name":"iterait/hipipe","owner":"iterait","description":"Super fast C++17 data transformation pipeline (with Python interface).","archived":false,"fork":false,"pushed_at":"2023-08-22T05:43:42.000Z","size":3166,"stargazers_count":17,"open_issues_count":4,"forks_count":0,"subscribers_count":7,"default_branch":"dev","last_synced_at":"2025-05-08T04:55:42.370Z","etag":null,"topics":["hpc","stream"],"latest_commit_sha":null,"homepage":"https://hipipe.org/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iterait.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-26T12:41:29.000Z","updated_at":"2023-12-05T09:24:47.000Z","dependencies_parsed_at":"2025-02-26T07:32:50.342Z","dependency_job_id":"be4bd931-52a6-49b4-a41d-efdaefae3b08","html_url":"https://github.com/iterait/hipipe","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterait%2Fhipipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterait%2Fhipipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterait%2Fhipipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iterait%2Fhipipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iterait","download_url":"https://codeload.github.com/iterait/hipipe/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253002854,"owners_count":21838640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","stream"],"created_at":"2025-02-26T07:32:43.937Z","updated_at":"2025-05-08T04:56:05.252Z","avatar_url":"https://github.com/iterait.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HiPipe\n[![CircleCI](https://circleci.com/gh/iterait/hipipe/tree/dev.svg?style=shield)](https://circleci.com/gh/iterait/hipipe/tree/dev)\n[![MIT license](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](LICENSE)\n[![Development Status](https://img.shields.io/badge/status-Regular-brightgreen.svg?style=flat)]()\n[![Master Developer](https://img.shields.io/badge/master-Filip%20Matzner-lightgrey.svg?style=flat)]()\n\n\n__HiPipe__ is a C++ library for efficient data processing. Its main purpose is to simplify\nand accelerate data preparation for deep learning models, but it is generic enough to be used\nin many other areas.\n\n__HiPipe__ lets the programmer build intuitive data streams that transform,\ncombine and filter the data that pass through. Those streams are compiled,\nbatched, and asynchronous, therefore maximizing the utilization of the provided\nhardware.\n\n- [Documentation and API reference](https://hipipe.org/).\n- [Installation guide](https://hipipe.org/installation.html).\n\n## Example\n\n```c++\nstd::vector\u003cstd::string\u003e logins = {\"marry\", \"ted\", \"anna\", \"josh\"};\nstd::vector\u003cint\u003e           ages = {     24,    41,     16,     59};\n\nauto stream = ranges::views::zip(logins, ages)\n\n  // create a batched stream out of the raw data\n  | hipipe::create\u003clogin, age\u003e(2)\n\n  // make everyone older by one year\n  | hipipe::transform(from\u003cage\u003e, to\u003cage\u003e, [](int a) { return a + 1; })\n\n  // increase each letter in the logins by one (i.e., a-\u003eb, e-\u003ef ...)\n  | hipipe::transform(from\u003clogin\u003e, to\u003clogin\u003e, [](char c) { return c + 1; }, dim\u003c2\u003e)\n\n  // increase the ages by the length of the login\n  | hipipe::transform(from\u003clogin, age\u003e, to\u003cage\u003e, [](std::string l, int a) {\n        return a + l.length();\n    })\n\n  // probabilistically rename 50% of the people to \"buzz\"\n  | hipipe::transform(from\u003clogin\u003e, to\u003clogin\u003e, 0.5, [](std::string) -\u003e std::string {\n        return \"buzz\";\n    })\n\n  // drop the login column from the stream\n  | hipipe::drop\u003clogin\u003e\n\n  // introduce the login column back to the stream\n  | hipipe::transform(from\u003cage\u003e, to\u003clogin\u003e, [](int a) {\n        return \"person_\" + std::to_string(a) + \"_years_old\";\n    })\n\n  // filter only people older than 30 years\n  | hipipe::filter(from\u003clogin, age\u003e, by\u003cage\u003e, [](int a) { return a \u003e 30; })\n\n  // asynchronously buffer the stream during iteration\n  | hipipe::buffer(2);\n\n// extract the ages from the stream to std::vector\nages = hipipe::unpack(stream, from\u003cage\u003e);\nassert((ages == std::vector\u003cint\u003e{45, 64}));\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiterait%2Fhipipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiterait%2Fhipipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiterait%2Fhipipe/lists"}