https://github.com/tsung-wei-huang/dtcraft
A High-performance Cluster Computing Engine
https://github.com/tsung-wei-huang/dtcraft
cluster-computing dataflow-programming distributed-systems event-driven-programming network-programming parallel-computing reactor stream-processing-engine streaming
Last synced: 2 months ago
JSON representation
A High-performance Cluster Computing Engine
- Host: GitHub
- URL: https://github.com/tsung-wei-huang/dtcraft
- Owner: tsung-wei-huang
- License: mit
- Created: 2017-11-30T01:32:17.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-05-04T20:34:01.000Z (about 6 years ago)
- Last Synced: 2025-03-27T23:51:08.466Z (3 months ago)
- Topics: cluster-computing, dataflow-programming, distributed-systems, event-driven-programming, network-programming, parallel-computing, reactor, stream-processing-engine, streaming
- Language: C++
- Homepage: https://tsung-wei-huang.github.io/DtCraft/
- Size: 45.3 MB
- Stars: 145
- Watchers: 16
- Forks: 25
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- Changelog: ChangeLog
- License: LICENSE
- Authors: AUTHORS
Awesome Lists containing this project
README
# What's DtCraft?
DtCraft is a general-purpose distributed programming system based on data-parallel streams. It offers a new powerful programming model called *stream graph* to build parallel and distributed workloads. Once an application is cast into this framework, the kernel transparently performs job distribution for you. You don't have to worry about system programming and can focus on high-level development!
| Unified Framework | Stream Graph |
| --------------- | ----------- |
|  |  |
Whether you are application developers or domain-specific engineers, you can use DtCraft for:
- Distributed computing
- Event-driven programming
- Network programming
- Data stream processing
To get your first DtCraft program up and running, visit QuickStart.
# Design Goals
The goal of DtCraft is to help you write simple, easy, and effective code at cluster scale. The example below demonstrates a simple application to run two programs on two machines sending each other a message.
```cpp
#include
using namespace std::literals; // for the use of string literal
using namespace dtc::literals; // for the use of memory literal
int main(int argc, char* argv[]) {
dtc::Graph G;
auto A = G.vertex();
auto B = G.vertex();
auto lambda = [] (dtc::Vertex& v, dtc::InputStream& is) {
if(std::string s; is(s) != -1) {
std::cout << "Received: " << s << '\n';
return dtc::Event::REMOVE;
}
return dtc::Event::DEFAULT;
};
auto AB = G.stream(A, B).on(lambda);
auto BA = G.stream(B, A).on(lambda);
A.on([&AB] (dtc::Vertex& v) { (*v.ostream(AB))("hello world from A"s); });
B.on([&BA] (dtc::Vertex& v) { (*v.ostream(BA))("hello world from B"s); });
G.container().add(A).cpu(1).memory(1_GB);
G.container().add(B).cpu(1).memory(1_GB);
dtc::Executor(G).run();
}
```
There are myriads of cluster computing frameworks such as Hadoop MapReduce, Apache Spark, Dryad, and Ray. We believe each has its reason to exist. DtCraft targets at:
- **Programming model**. DtCraft leverages modern C++17 to offer a new programming model for cluster computing. Our model is very general and can implement most distributed computing patterns.
- **Performance**. DtCraft is designed completely from the ground up using advanced software techniques in order to deliver the best performance.
- **Productivity**. DtCraft allows you to easily bring up a parallel and distributed workload in only a few lines of code. It takes only a few steps to set up a cluster to get things up and running.
# System Requirements
To install and run DtCraft, you only need the following:
- 64-bit Linux machine(s) with Kernel 3.8 or higher
- GNU C++ Compiler G++ v7.2 with -std=c++1z
- GNU Autotool (autoconf, automake, libtool)
# Learn More
+ Official website
# Get Involved
+ Report bugs/issues by submitting a GitHub issue.
+ Submit contributions using pull requests.