https://github.com/mbraceproject/mbrace.core
MBrace Core Libraries & Runtime Foundations
https://github.com/mbraceproject/mbrace.core
Last synced: 9 months ago
JSON representation
MBrace Core Libraries & Runtime Foundations
- Host: GitHub
- URL: https://github.com/mbraceproject/mbrace.core
- Owner: mbraceproject
- License: apache-2.0
- Created: 2014-10-14T10:39:25.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2020-03-12T19:13:30.000Z (almost 6 years ago)
- Last Synced: 2025-03-28T08:05:59.191Z (10 months ago)
- Language: F#
- Homepage: http://mbrace.io/
- Size: 4.71 MB
- Stars: 211
- Watchers: 35
- Forks: 46
- Open Issues: 19
-
Metadata Files:
- Readme: README.md
- License: License.md
Awesome Lists containing this project
README
# MBrace Core Libraries
[](https://www.nuget.org/packages/MBrace.Core/)
[](https://ci.appveyor.com/project/dsyme/mbrace-core/branch/master)
[](https://travis-ci.org/mbraceproject/MBrace.Core)
[](https://opensource.org/licenses/MIT)
This repository contains core libraries and runtime foundation components
for the MBrace cloud programming model and frameworks.
For a first introduction to MBrace please refer to the main website at [mbrace.io](http://www.mbrace.io/).
If you have any questions regarding MBrace don't hesitate to create an issue or ask one of the [maintainers](#maintainers).
You can also follow the official MBrace twitter account [@mbracethecloud](https://twitter.com/mbracethecloud).
### MBrace.Core [](https://www.nuget.org/packages/MBrace.Core/)
[MBrace.Core](https://github.com/mbraceproject/MBrace.Core/tree/master/src/MBrace.Core) is a standalone class library that contains the core MBrace programming model, used to author general-purpose, runtime-agnostic distributed computation. It is centered on the concept of *cloud workflows*, a composable, language-integrated API based on F# computation expressions. It can be used to author specialized cloud libraries like MBrace.Flow.
[MBrace.Core.Tests](https://github.com/mbraceproject/MBrace.Core/tree/master/tests/MBrace.Core.Tests) defines a comprehensive suite of abstracted NUnit/FsCheck based tests for use by MBrace runtime implementations. Useful for verifying that an MBrace implementation is up to spec regarding distribution semantics, serialization and fault tolerance. See [MBrace.Thespian.Tests](https://github.com/mbraceproject/MBrace.Core/tree/master/tests/MBrace.Thespian.Tests) and [MBrace.Azure.Tests](https://github.com/mbraceproject/MBrace.Azure/tree/master/tests/MBrace.Azure.Tests) for samples that make use of the test suites.
### MBrace.Flow [](https://www.nuget.org/packages/MBrace.Flow/)
[MBrace.Flow](https://github.com/mbraceproject/MBrace.Core/tree/master/src/MBrace.Flow) is a distributed streaming library written on top of MBrace.Core. It enables distributed computation using functional pipeline declarations like the following:
```fsharp
CloudFlow.OfHttpFileByLine "http://my.server.local/large.txt"
|> CloudFlow.collect (fun line -> line.Split [|' '; ',' ; '.'|])
|> CloudFlow.filter (fun w -> w.Length > 3)
|> CloudFlow.map (fun w -> w.ToLower())
|> CloudFlow.countBy id
|> CloudFlow.sortBy (fun (_,c) -> -c) 10
|> CloudFlow.toArray
```
It is written on top of the [Nessos.Streams](http://nessos.github.io/Streams) library, a fast streaming library inspired by [Java 8 Streams](http://www.oracle.com/technetwork/articles/java/ma14-java-se-8-streams-2177646.html).
### MBrace.CSharp [](https://www.nuget.org/packages/MBrace.CSharp/)
[MBrace.CSharp](https://github.com/mbraceproject/MBrace.Core/tree/master/src/MBrace.CSharp) contains C# friendly wrapper APIs for the MBrace core programming model.
# MBrace Implementations
### MBrace.Runtime [](https://www.nuget.org/packages/MBrace.Runtime/)
[MBrace.Runtime](https://github.com/mbraceproject/MBrace.Core/tree/master/src/MBrace.Runtime) provides an extensive set of foundations and common components for quickly developing MBrace runtimes on top of [FsPickler](http://nessos.github.io/FsPickler)/[Vagabond](http://nessos.github.io/Vagabond). It removes the hassle of working with MBrace semantics and lets the runtime developer focus on providing cloud infrastructure implementations such as:
* Cloud file storage.
* Cloud table storage.
* Work item queue/scheduler.
Refer to [MBrace.Thespian](https://github.com/mbraceproject/MBrace.Core/tree/master/samples/MBrace.Thespian) and [MBrace.Azure](https://github.com/mbraceproject/MBrace.Azure) for examples of MBrace runtimes that are implemented on top of MBrace.Runtime.
We've done work that effectively decouples the runtime implementation of MBrace from any particular cloud vendor. Details such as use of Vagabond, cloud workflow execution semantics, serialization and caching are handled by MBrace.Runtime, which can be used to target any infrastructure that supplies the following resources:
* A distributed storage: e.g. S3, Blob Storage, HDFS, etc.;
* Distributed queues with fault tolerance;
* A small collection of simple synchronization primitives, typically implemented on top of a database; and
* Stateless worker instances that can talk to the above services.
As a result, we have been able to quickly provide MBrace implementations for 3 different settings: standalone, Azure and AWS, all with relatively small codebases: 2900, 6800 and 6200 LoC respectively.
Conceivably it is now possible to provide MBrace bindings for any cloud infrastructure that provides aforementioned resources, private or public. We believe there's space for collaboration here.
### MBrace.Thespian [](https://www.nuget.org/packages/MBrace.Thespian/)
[MBrace.Thespian](https://github.com/mbraceproject/MBrace.Core/tree/2ee52e8518e5a1450bf357495580d6b88e171477/src/MBrace.Thespian) is a simple MBrace cluster implementation on top of MBrace.Runtime and the [Thespian](http://nessos.github.io/Thespian) actor library. Not intended for production deployments, it is used for testing MBrace core development.
### MBrace.Azure [](https://www.nuget.org/packages/MBrace.Azure/)
[MBrace.Azure](http://github.com/mbraceproject/MBrace.Azure/) is an MBrace framework implementation on top of Azure PaaS components. Enables easy deployment of scalable MBrace clusters using worker roles. It also supports on-site cluster deployments using Azure storage/service bus components for communication.
### MBrace.AWS [](https://www.nuget.org/packages/MBrace.AWS/)
[MBrace.AWS](http://github.com/mbraceproject/MBrace.AWS/) is an MBrace framework implementation on top of AWS PaaS components. Enables easy deployment of scalable MBrace clusters using Elastic Beanstalk & Docker. It also supports on-site cluster deployments using AWS S3/DynamoDB/SQS components for communication.
## Building and Running unit tests for MBrace.Core
Building MBrace Core requires Visual Studio 2015/F# 4.0. You can build the project either from Visual Studio or by running `build.cmd Build` if using cmd/powershell or `./build.sh Build` if using bash/sh.
Unit tests can be run by calling `build.cmd RunTests` or `./build.sh RunTests`.
You can skip the time consuming Thespian by adding the `-ef IgnoreClusterTests` flag in the above commands.
Alternatively you can run individual tests by opening test assemblies found in the repository's `/bin` folder
using [NUnit-GUI](http://www.nunit.org/index.php?p=nunit-gui&r=2.2.10). Building the solution produces the following
test assemblies:
* `MBrace.Core.Tests.dll` for testing the core MBrace library.
* `MBrace.Runtime.Tests.dll` for testing MBrace.Runtime components.
* `MBrace.Thespian.Tests.dll` for testing the MBrace Thespian implementation.
When performing changes to MBrace.Core, MBrace.Flow or MBrace.Runtime it is essential to verify that the MBrace.Thespian tests are still passing.
## Contributing
The MBrace project is happy to accept quality contributions from the .NET community.
If you would like to get involved, here are a few places you could have a look at:
* MBrace Libraries: we are looking for data scientists and domain experts who can help us develop specialized libraries that run on top of MBrace.Core. Examples include [Machine Learning](https://spark.apache.org/docs/1.1.0/mllib-guide.html) and [Graph analytics](http://spark.apache.org/graphx/) libraries.
* MBrace Runtimes: help MBrace grow by extending support to your favorite cloud service. It could be AWS, YARN/HDFS or your private infrastructure.
* C# Support: As of MBrace 1.0, development of [MBrace.CSharp](#mbrace.csharp) has been suspended. We are looking for working C# developers interested in extending MBrace support to C# and testing deployments using the upcoming C# Interactive.
* Documentation & Code Samples: help improve MBrace documentation and coding samples. See the [MBrace.StarterKit](https://github.com/mbraceproject/MBrace.StarterKit) for current coding samples and the [mbrace-docs](https://github.com/mbraceproject/mbrace-docs) repo for documentation and the [mbrace.io](http://www.mbrace.io/) website.
### CI NuGet feeds
* Vagabond: https://ci.appveyor.com/nuget/vagabond-6odcsr0usg1g
* MBrace.Core: https://ci.appveyor.com/nuget/mbrace-core-gjo4016ri68g
* MBrace.Azure: https://ci.appveyor.com/nuget/mbrace-azure-jglidppatqhw
* MBrace.AWS: https://ci.appveyor.com/nuget/mbrace-aws-drjomsmfdib7
## License
This project is subject to the [Apache Licence, Version 2.0](License.md).
## Maintainers
* [@eiriktsarpalis](https://twitter.com/eiriktsarpalis)
* [@nickpalladinos](https://twitter.com/nickpalladinos)
* [@krontogiannis](https://twitter.com/krontogiannis)
* [@dsyme](https://twitter.com/dsyme)
* [@isaac_abraham](https://twitter.com/isaac_abraham)