https://github.com/haraqa/haraqa
High Availability Routing and Queueing Application
https://github.com/haraqa/haraqa
docker go message-queue
Last synced: 6 months ago
JSON representation
High Availability Routing and Queueing Application
- Host: GitHub
- URL: https://github.com/haraqa/haraqa
- Owner: haraqa
- License: mit
- Created: 2019-12-24T21:43:25.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-09-21T02:54:50.000Z (almost 5 years ago)
- Last Synced: 2024-08-05T09:15:40.708Z (almost 2 years ago)
- Topics: docker, go, message-queue
- Language: Go
- Size: 966 KB
- Stars: 50
- Watchers: 5
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
HARAQA - High Availability Routing And Queueing Application
[](https://pkg.go.dev/github.com/haraqa/haraqa?tab=doc)
[](https://goreportcard.com/report/haraqa/haraqa)
[](https://github.com/haraqa/haraqa/blob/master/LICENSE)
[](https://github.com/haraqa/haraqa/blob/master/.github/workflows/go.yml)
[](https://coveralls.io/github/haraqa/haraqa)
[](https://hub.docker.com/r/haraqa/haraqa/)
[](https://github.com/haraqa/haraqa/releases)
**haraqa** is designed to be a developer friendly, scalable message queue for data
persistence and real-time data streaming between microservices.
Haraqa provides high-throughput, low-latency, fault-tolerant pipelines for architectures of any size.
### Table of Contents
* [About the Project](#about-the-project)
* [Overview](#overview)
* [Persistence and Replication](#persistence-and-replication)
* [Getting Started](#getting-started)
* [API Docs](#api-docs)
* [Server](#server)
* [Client](#client)
* [Contributing](#contributing)
* [License](#license)
## About the Project
### Overview
Haraqa is meant for handling and persisting data in a distributed system.
One or more servers can be used to send and receive messages.
Each server has a set of 'topics', a set of messages stored in the order received.
A Haraqa client can produce and/or consume from a server's topics.
Clients can produce messages one at a time or in batches.
Clients consume messages by making a request for a specific offset and limit.
The messages can be consumed one at a time or in batches.
### Persistence and Replication
Each server, after receiving a message from a producer, can save the message to multiple volumes.
These volumes are meant to be distributed in the architecture, such as having
multiple PersistentVolumes in a Kubernetes cluster, EBS in AWS, or Persistent Disks in Google Cloud.
The server reads messages from the last volume when sending to consumer clients.
If a volume is removed or corrupted during a restart the server repopulates the data from the other volumes.
## Getting started
### API Docs
* [Swagger API Documentation](https://haraqa.github.io/haraqa/cmd/server/swagger.html)
* [Redocs API Documentation](https://haraqa.github.io/haraqa/cmd/server/redocs.html)
* [Swagger yaml](https://github.com/haraqa/haraqa/blob/master/cmd/server/swagger.yaml)
The docker server also includes local api documentation at the `/docs/redocs` and `/docs/swagger` endpoints.
### Server
The recommended deployment strategy is to use [Docker](https://hub.docker.com/r/haraqa/haraqa)
```
docker run -it -p 4353:4353 -v $PWD/vol1:/vol1 haraqa/haraqa /vol1
```
```
docker run -it [port mapping] [volume mounts] haraqa/haraqa [flags] [volumes]
```
To run from source, navigate to cmd/server and run the main.go file.
```
cd cmd/server
go run main.go vol1
```
##### Flags:
```
-http uint Port to listen on (default 4353)
-cache boolean Enable queue file caching (default true)
-cors boolean Enable CORS (default true)
-docs boolean Enable Docs pages (default true)
-entries integer The number of msg entries per queue file before creating a new file (default 5000)
-limit integer Default batch limit for consumers (default -1)
-ballast integer Garbage collection memory ballast size in bytes (default 1073741824)
-prometheus boolean Enable prometheus metrics (default true)
```
##### Volumes:
Volumes will be written to in the order given and recovered from in the reverse
order. Consumer requests are read from the last volume. For this reason it's
recommended to use a local volume last.
For instance, given
```
docker run haraqa/haraqa /vol1 /vol2 /vol3
```
When a client produces a message, it will be written to /vol1, then /vol2, then /vol3.
When a client consumes a message, it will be read from /vol3.
During recovery, if data exists in /vol3 it will be replicated to volumes /vol1 and /vol2.
If /vol3 is empty, /vol2 will be replicated to /vol1 and /vol3.
### Client
```
go get github.com/haraqa/haraqa
```
## Contributing
We want this project to be the best it can be and all feedback, feature requests or pull requests are welcome.
## License
MIT © 2019 [haraqa](https://github.com/haraqa/) and [contributors](https://github.com/haraqa/haraqa/graphs/contributors). See `LICENSE` for more information.