https://github.com/exacaster/lighter

REST API for Apache Spark on K8S or YARN
https://github.com/exacaster/lighter

apache-spark jupyter k8s livy spark sparkmagic yarn

Last synced: about 10 hours ago
JSON representation

REST API for Apache Spark on K8S or YARN

• Host: GitHub
• URL: https://github.com/exacaster/lighter
• Owner: exacaster
• License: mit
• Created: 2021-10-13T07:10:14.000Z (about 3 years ago)
• Default Branch: master
• Last Pushed: 2024-10-30T07:55:59.000Z (about 2 months ago)
• Last Synced: 2024-10-30T08:29:56.535Z (about 2 months ago)
• Topics: apache-spark, jupyter, k8s, livy, spark, sparkmagic, yarn
• Language: Java
• Homepage:
• Size: 6.06 MB
• Stars: 91
• Watchers: 4
• Forks: 21
• Open Issues: 10
• Metadata Files:
  • Readme: README.md
  • Contributing: docs/CONTRIBUTING.md
  • License: LICENSE.txt
  • Code of conduct: docs/CODE_OF_CONDUCT.md
  • Codeowners: docs/CODEOWNERS

Awesome Lists containing this project

README

        
# Lighter

Lighter is an opensource application for interacting with [Apache Spark](https://spark.apache.org/) on [Kubernetes](https://kubernetes.io/) or [Apache Hadoop YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html). It is heavily inspired by [Apache Livy](https://livy.incubator.apache.org/) and has some overlapping features.

Lighter supports:

- Interactive Python Sessions through [Sparkmagic](https://github.com/jupyter-incubator/sparkmagic) kernel

- Batch application submissions through the REST API

You can read a breaf description on how Lighter works [here](./docs/architecture.md).

## Using Lighter

- [Quickstart with Minikube](./quickstart/README.md)

- [Installation on Kubernetes](./docs/kubernetes.md)

- [Installation on Docker](./docs/docker.md)

- [Configuration Properties](./docs/configuration.md)

- [Configuring Sparkmagic](./docs/sparkmagic.md)

- [Using REST API](./docs/rest.md)

## Developing Lighter

### Building

To build Docker image, containing application, you need to run

```

docker build -t lighter .

```

### Spark versions

Lighter will always try to support the latest patch version for all officially recommended [Apache Spark](https://spark.apache.org/) releases i.e.: 3.4.3 and 3.5.3.

### Contributing

See [Contribution guide](./docs/CONTRIBUTING.md)

## License

Lighter is [MIT](./LICENSE.txt) licensed.