Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/redislabs/spark-redis

A connector for Spark that allows reading and writing to/from Redis cluster
https://github.com/redislabs/spark-redis

dataframe java redis spark

Last synced: about 1 month ago
JSON representation

A connector for Spark that allows reading and writing to/from Redis cluster

Host: GitHub
URL: https://github.com/redislabs/spark-redis
Owner: RedisLabs
License: bsd-3-clause
Created: 2015-08-16T08:39:33.000Z (about 9 years ago)
Default Branch: master
Last Pushed: 2024-06-03T11:36:56.000Z (5 months ago)
Last Synced: 2024-09-24T09:02:00.629Z (about 2 months ago)
Topics: dataframe, java, redis, spark
Language: Scala
Homepage:
Size: 2.18 MB
Stars: 936
Watchers: 71
Forks: 369
Open Issues: 140
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        [![license](https://img.shields.io/github/license/RedisLabs/spark-redis.svg)](https://github.com/RedisLabs/spark-redis)

[![Release](https://img.shields.io/github/release/RedisLabs/spark-redis.svg?sort=semver)](https://github.com/RedisLabs/spark-redis/releases/latest)

[![CircleCI](https://circleci.com/gh/RedisLabs/spark-redis/tree/master.svg?style=svg)](https://circleci.com/gh/RedisLabs/spark-redis/tree/master)

[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.redislabs/spark-redis_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.redislabs/spark-redis_2.12)

[![Javadocs](https://www.javadoc.io/badge/com.redislabs/spark-redis_2.12.svg)](https://www.javadoc.io/doc/com.redislabs/spark-redis_2.12)

[![Discord](https://img.shields.io/discord/697882427875393627?style=flat-square)](https://discord.gg/Bjt6TcgV)

[![Codecov](https://codecov.io/gh/RedisLabs/spark-redis/branch/master/graph/badge.svg)](https://codecov.io/gh/RedisLabs/spark-redis)

# Spark-Redis

A library for reading and writing data in [Redis](http://redis.io) using [Apache Spark](http://spark.apache.org/).

Spark-Redis provides access to all of Redis' data structures - String, Hash, List, Set and Sorted Set - from Spark as RDDs. It also supports reading and writing with DataFrames and Spark SQL syntax.

The library can be used both with Redis stand-alone as well as clustered databases. When used with Redis cluster, Spark-Redis is aware of its partitioning scheme and adjusts in response to resharding and node failure events.

Spark-Redis also supports Spark Streaming (DStreams) and Structured Streaming.

## Version compatibility and branching

The library has several branches, each corresponds to a different supported Spark version. For example, 'branch-2.3' works with any Spark 2.3.x version.

The master branch contains the recent development for the next release.

| Spark-Redis                                                               | Spark | Redis            | Supported Scala Versions | 

|---------------------------------------------------------------------------|-------| ---------------- | ------------------------ |

| [master](https://github.com/RedisLabs/spark-redis/)                       | 3.2.x | >=2.9.0          | 2.12                     | 

| [3.0](https://github.com/RedisLabs/spark-redis/tree/branch-3.0)           | 3.0.x | >=2.9.0          | 2.12                     | 

| [2.4, 2.5, 2.6](https://github.com/RedisLabs/spark-redis/tree/branch-2.4) | 2.4.x | >=2.9.0          | 2.11, 2.12               | 

| [2.3](https://github.com/RedisLabs/spark-redis/tree/branch-2.3)           | 2.3.x | >=2.9.0          | 2.11                     | 

| [1.4](https://github.com/RedisLabs/spark-redis/tree/branch-1.4)           | 1.4.x |                  | 2.10                     | 

## Known limitations

* Java, Python and R API bindings are not provided at this time

## Additional considerations

This library is a work in progress so the API may change before the official release.

## Documentation

Please make sure you use documentation from the correct branch ([2.4](https://github.com/RedisLabs/spark-redis/tree/branch-2.4#documentation), [2.3](https://github.com/RedisLabs/spark-redis/tree/branch-2.3#documentation), etc). 

  - [Getting Started](doc/getting-started.md)

  - [RDD](doc/rdd.md)

  - [Dataframe](doc/dataframe.md)

  - [Streaming](doc/streaming.md)

  - [Structured Streaming](doc/structured-streaming.md)

  - [Cluster](doc/cluster.md)

  - [Java](doc/java.md)

  - [Python](doc/python.md)

  - [Configuration](doc/configuration.md)

  - [Dev environment](doc/dev.md)

## Contributing

You're encouraged to contribute to the Spark-Redis project. 

There are two ways you can do so:

### Submit Issues

If you encounter an issue while using the library, please report it via the project's [issues tracker](https://github.com/RedisLabs/spark-redis/issues).

### Author Pull Requests

Code contributions to the Spark-Redis project can be made using [pull requests](https://github.com/RedisLabs/spark-redis/pulls). To submit a pull request:

 1. Fork this project.

 2. Make and commit your changes.

 3. Submit your changes as a pull request.