https://github.com/aagumin/goflint

Spark-submit golang wrapper
https://github.com/aagumin/goflint

apache-spark cli-wrapper go golang spark spark-submit

Last synced: 6 months ago
JSON representation

Spark-submit golang wrapper

• Host: GitHub
• URL: https://github.com/aagumin/goflint
• Owner: aagumin
• License: mit
• Created: 2025-04-09T18:04:04.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2025-05-06T13:37:06.000Z (9 months ago)
• Last Synced: 2025-06-27T10:11:30.816Z (7 months ago)
• Topics: apache-spark, cli-wrapper, go, golang, spark, spark-submit
• Language: Go
• Homepage:
• Size: 114 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# GoFlint 🔥

[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=aagumin_goflint&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=aagumin_goflint)

[![Go build](https://github.com/aagumin/goflint/actions/workflows/go.yml/badge.svg)](https://github.com/aagumin/goflint/actions/workflows/go.yml)

[![Go Report Card](https://goreportcard.com/badge/github.com/aagumin/goflint)](https://goreportcard.com/report/github.com/aagumin/goflint)

[![Maintainability](https://api.codeclimate.com/v1/badges/aabaf6a2d52511b3d581/maintainability)](https://codeclimate.com/github/aagumin/goflint/maintainability)

[![Test Coverage](https://api.codeclimate.com/v1/badges/aabaf6a2d52511b3d581/test_coverage)](https://codeclimate.com/github/aagumin/goflint/test_coverage)

[![Known Vulnerabilities](https://snyk.io/test/github/aagumin/goflint/badge.svg)](https://app.snyk.io/org/aagumin/project/f1c4783a-33e3-4ad8-8c2d-1dc1f67e1866)

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Faagumin%2Fgoflint.svg?type=shield&issueType=license)](https://app.fossa.com/projects/git%2Bgithub.com%2Faagumin%2Fgoflint?ref=badge_shield&issueType=license)

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Faagumin%2Fgoflint.svg?type=shield&issueType=security)](https://app.fossa.com/projects/git%2Bgithub.com%2Faagumin%2Fgoflint?ref=badge_shield&issueType=security)

> **[WIP]** Project in active dev state.

**GoFlint** is an idiomatic Go library for submitting Apache Spark jobs via `spark-submit`.  

Designed for clarity and type safety, it wraps Spark’s CLI with a fluent Go API.

## Features

- **Fluent API** for all `spark-submit` options (masters, deploy modes, args, etc.)

- **Context support** for cancellation/timeouts

- **Extensible** with custom logging and monitoring

- **Zero dependencies** (except Go’s standard library)

## Installation

```bash

go get github.com/aagumin/flint

```

## Example usage

```go

package main

import (

   "context"

   "fmt"

   "os"

   "path"

   "github.com/aagumin/goflint/flint"

   sc "github.com/aagumin/goflint/flint/sparkconf"

)

func main() {

   xx := map[string]string{"spark.driver.port": "4031", "spark.driver.host": "localhost"}

   sparkCfg := sc.NewFrozenConf(xx)

   scalaExamples := path.Join(os.Getenv("SPARK_HOME"), "examples/jars/spark-examples_2.12-3.5.3.jar")

   submit := flint.NewSparkApp(

      flint.WithApplication(scalaExamples),

      flint.WithSparkConf(sparkCfg),

      flint.WithName("GoFlint"),

      flint.WithMainClass("org.apache.spark.examples.parkPi"),

   )

   base := submit.Build()

   updatedSubmit := flint.ExtendSparkApp(

      &base,

      flint.WithMaster(""),

      // Other options...

   )

   app := updatedSubmit.Build()

   ctx := context.Background()

   _, err = app.Submit(ctx)

   if err != nil {

      fmt.Println(err)

   }

}

```

## Design Principles

1. **Idiomatic Go**

    - Errors as `error`, not panics

    - `context.Context` support

    - Interfaces for extensibility

2. **Spark Compatibility**

    - Maps 1:1 with `spark-submit` CLI

    - No hidden magic – transparent command generation

3. **Batteries Included**

    - Defaults for quick starts

    - Extensible for complex cases

---

## Roadmap

- [ ] Async job monitoring

- [ ] YARN/K8s auth helpers

- [ ] Prometheus metrics integration

---

## Contributing

PRs welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

MIT © Arsen Gumin