https://github.com/garugaru/gearpump-swarm

Deploy gearpump on a docker-swarm cluster
https://github.com/garugaru/gearpump-swarm

bigdata dataprocessing docker real-time swarm

Last synced: about 1 month ago
JSON representation

Deploy gearpump on a docker-swarm cluster

• Host: GitHub
• URL: https://github.com/garugaru/gearpump-swarm
• Owner: GaruGaru
• License: apache-2.0
• Created: 2017-10-16T16:12:48.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2017-10-18T18:19:12.000Z (over 8 years ago)
• Last Synced: 2025-06-17T23:02:36.052Z (12 months ago)
• Topics: bigdata, dataprocessing, docker, real-time, swarm
• Language: Shell
• Size: 28.3 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Run & Scale Apache Gearpump on a docker swarm cluster

[![Codefresh build status]( https://g.codefresh.io/api/badges/build?repoOwner=GaruGaru&repoName=gearpump-swarm&branch=master&pipelineName=gearpump-swarm&accountName=garugaru&type=cf-1)]( https://g.codefresh.io/repositories/GaruGaru/gearpump-swarm/builds?filter=trigger:build;branch:master;service:59e5b94a3523010001347f4a~gearpump-swarm)

[![Docker Pulls](https://img.shields.io/docker/pulls/garugaru/gearpump.svg)]()

[![Docker Stars](https://img.shields.io/docker/stars/garugaru/gearpump.svg)]()

**Apache Gearpump**

	Apache Gearpump is a lightweight real-time big data streaming engine

	

**Docker Swarm**

	Docker Swarm is a clustering and scheduling tool for Docker containers

## Docker Swarm + Apache Gearpump = ♥

### Deploy gearpump cluster master + 3 workers + ui

	docker stack deploy -c docker-compose.yml gearpump

#### Web UI

	http://dockerhost:8090/

#### Submitting an application

In order to submit a new application you can use the web ui or mount a volume on the master container and use the deploy command:

	bin/gear app -jar applications/your-app.jar your.main.class 

#### Scaling Workers

To scale your workers just use the docker-swarm command, new workers will be automatically configured and registered on the cluster

	 docker service scale gearpump_gearpump-worker= 

	

### Performances

The current setup is able to reach **200.000+ msg/sec** with the base example application in a 3 nodes ( 2 core, 2 gb memory) cluster (2 workers, 1 master)

	

### Example docker swarm stack

	version: '3'

	services:

	  gearpump-master:

	    hostname: gearpump-master

	    image: garugaru/gearpump:b1880f4

	    entrypoint: ./entrypoint.sh

	    command: ./bin/master -ip gearpump-master -port 3000

	    environment:

		- MASTERS=["gearpump-master:3000"]

		- HOSTNAME=gearpump-master    

	    deploy:

		mode: replicated

		replicas: 1

	  gearpump-ui:

	    image: garugaru/gearpump:b1880f4

	    entrypoint: ./entrypoint.sh

	    command: ./bin/services

	    environment:

		- MASTERS=["gearpump-master:3000"]

	    ports:

	      - "8090:8090"

	    depends_on:

		- gearpump-master

	    deploy:

		mode: replicated

		replicas: 1

	  gearpump-worker:

	    hostname: "{{.Node.Hostname}}"

	    entrypoint: ./entrypoint.sh

	    image: garugaru/gearpump:b1880f4

	    command: ./bin/worker 

	    environment:

		- MASTERS=["gearpump-master:3000"] 

	    depends_on:

		- gearpump-master

		- gearpump-ui

	    deploy:

		mode: replicated

		replicas: 3