https://github.com/grailbio/bigslice

A serverless cluster computing system for the Go programming language
https://github.com/grailbio/bigslice

bigdata cluster computing etl go golang machinelearning mapreduce

Last synced: about 1 month ago
JSON representation

A serverless cluster computing system for the Go programming language

• Host: GitHub
• URL: https://github.com/grailbio/bigslice
• Owner: grailbio
• License: apache-2.0
• Created: 2019-09-10T21:35:17.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2023-05-19T00:55:45.000Z (over 1 year ago)
• Last Synced: 2024-08-02T01:25:42.786Z (4 months ago)
• Topics: bigdata, cluster, computing, etl, go, golang, machinelearning, mapreduce
• Language: Go
• Homepage: https://bigslice.io/
• Size: 2.66 MB
• Stars: 548
• Watchers: 27
• Forks: 35
• Open Issues: 24
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-repositories - grailbio/bigslice - A serverless cluster computing system for the Go programming language (Go)
• awesome-golang-repositories - bigslice
• go-awesome - Bigslice

README

        
# Bigslice

Bigslice is a serverless cluster data processing system for [Go](https://golang.org).

Bigslice exposes composable API

that lets the user express

data processing tasks in terms of a series of

data transformations that invoke user code.

The Bigslice runtime then

transparently parallelizes and distributes the work,

using the [Bigmachine](https://github.com/grailbio/bigmachine)

library to create an ad hoc cluster on a cloud provider.

- website: [bigslice.io](https://bigslice.io/)

- API documentation: [godoc.org/github.com/grailbio/bigslice](https://godoc.org/github.com/grailbio/bigslice)

- issue tracker: [github.com/grailbio/bigslice/issues](https://github.com/grailbio/bigslice/issues)

- [![CI](https://github.com/grailbio/bigslice/workflows/CI/badge.svg)](https://github.com/grailbio/bigslice/actions?query=workflow%3ACI) [![Full Test](https://github.com/grailbio/bigslice/workflows/Full%20Test/badge.svg)](https://github.com/grailbio/bigslice/actions?query=workflow%3A%22Full+Test%22)

# Developing Bigslice

Bigslice uses Go modules to capture its dependencies;

no tooling other than the base Go install is required.

```

$ git clone https://github.com/grailbio/bigslice

$ cd bigslice

$ GO111MODULE=on go test

```

If tests fail with `socket: too many open files` errors, try increasing the maximum number of open files.

```

$ ulimit -n 2000

```