https://github.com/affogarty/niccolo
Easy-to-use orchestration framework for .NET for Apache Spark applications.
https://github.com/affogarty/niccolo
apache-spark csharp dotnet easy-to-use orchestration poco prototyping simple
Last synced: 3 months ago
JSON representation
Easy-to-use orchestration framework for .NET for Apache Spark applications.
- Host: GitHub
- URL: https://github.com/affogarty/niccolo
- Owner: AFFogarty
- License: apache-2.0
- Created: 2020-07-07T04:05:27.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2020-07-11T04:12:33.000Z (over 5 years ago)
- Last Synced: 2025-01-08T07:44:39.001Z (9 months ago)
- Topics: apache-spark, csharp, dotnet, easy-to-use, orchestration, poco, prototyping, simple
- Language: C#
- Homepage:
- Size: 22.5 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Niccolo
Niccolo is an easy-to-use orchestration framework for
[.NET for Apache Spark](https://github.com/dotnet/spark) applications. Niccolo makes it quick and
easy to write .NET applications and submit them to Apache Spark clusters.
## Use cases
Niccolo can be used for:
1. Rapid development and prototyping of applications.
2. [POCO](https://en.wikipedia.org/wiki/Plain_old_CLR_object) code-first approach to application
scheduling and management.
## Sample
The following example shows how to use Niccolo to write a simple .Net for Apache Spark application
and submit it to a local cluster.
```csharp
using Microsoft.Spark.Sql;
using Niccolo.Cluster.Local;
using Niccolo.Interfaces;
namespace Niccolo.Example
{
///
/// Example Spark batch application.
///
class TestSparkApp : ISparkBatchApplication
{
public SparkSession BuildSparkSession()
{
return SparkSession.Builder().GetOrCreate();
}
public void Run(SparkSession spark)
{
var data = spark.Range(25);
data.Show();
}
}
class Program
{
static void Main(string[] args)
{
// Reference to the local Spark cluster.
var cluster = new LocalSparkCluster();
// Start the batch application on the local cluster.
ISparkJob job = cluster.RunBatchApplication(typeof(TestSparkApp));
// Wait for the batch application to finish.
job.RunningTask.Wait();
}
}
}
```