https://github.com/iterait/hipipe
Super fast C++17 data transformation pipeline (with Python interface).
https://github.com/iterait/hipipe
hpc stream
Last synced: about 1 year ago
JSON representation
Super fast C++17 data transformation pipeline (with Python interface).
- Host: GitHub
- URL: https://github.com/iterait/hipipe
- Owner: iterait
- License: mit
- Created: 2018-09-26T12:41:29.000Z (almost 8 years ago)
- Default Branch: dev
- Last Pushed: 2023-08-22T05:43:42.000Z (almost 3 years ago)
- Last Synced: 2025-05-08T04:55:42.370Z (about 1 year ago)
- Topics: hpc, stream
- Language: C++
- Homepage: https://hipipe.org/
- Size: 3.02 MB
- Stars: 17
- Watchers: 7
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# HiPipe
[](https://circleci.com/gh/iterait/hipipe/tree/dev)
[](LICENSE)
[]()
[]()
__HiPipe__ is a C++ library for efficient data processing. Its main purpose is to simplify
and accelerate data preparation for deep learning models, but it is generic enough to be used
in many other areas.
__HiPipe__ lets the programmer build intuitive data streams that transform,
combine and filter the data that pass through. Those streams are compiled,
batched, and asynchronous, therefore maximizing the utilization of the provided
hardware.
- [Documentation and API reference](https://hipipe.org/).
- [Installation guide](https://hipipe.org/installation.html).
## Example
```c++
std::vector logins = {"marry", "ted", "anna", "josh"};
std::vector ages = { 24, 41, 16, 59};
auto stream = ranges::views::zip(logins, ages)
// create a batched stream out of the raw data
| hipipe::create(2)
// make everyone older by one year
| hipipe::transform(from, to, [](int a) { return a + 1; })
// increase each letter in the logins by one (i.e., a->b, e->f ...)
| hipipe::transform(from, to, [](char c) { return c + 1; }, dim<2>)
// increase the ages by the length of the login
| hipipe::transform(from, to, [](std::string l, int a) {
return a + l.length();
})
// probabilistically rename 50% of the people to "buzz"
| hipipe::transform(from, to, 0.5, [](std::string) -> std::string {
return "buzz";
})
// drop the login column from the stream
| hipipe::drop
// introduce the login column back to the stream
| hipipe::transform(from, to, [](int a) {
return "person_" + std::to_string(a) + "_years_old";
})
// filter only people older than 30 years
| hipipe::filter(from, by, [](int a) { return a > 30; })
// asynchronously buffer the stream during iteration
| hipipe::buffer(2);
// extract the ages from the stream to std::vector
ages = hipipe::unpack(stream, from);
assert((ages == std::vector{45, 64}));
```