{"id":13826714,"url":"https://github.com/haraqa/haraqa","last_synced_at":"2025-12-25T15:06:40.661Z","repository":{"id":57505164,"uuid":"230006404","full_name":"haraqa/haraqa","owner":"haraqa","description":"High Availability Routing and Queueing Application","archived":false,"fork":false,"pushed_at":"2021-09-21T02:54:50.000Z","size":989,"stargazers_count":50,"open_issues_count":0,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-08-05T09:15:40.708Z","etag":null,"topics":["docker","go","message-queue"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/haraqa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-24T21:43:25.000Z","updated_at":"2024-05-20T11:44:34.000Z","dependencies_parsed_at":"2022-08-22T08:50:22.536Z","dependency_job_id":null,"html_url":"https://github.com/haraqa/haraqa","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haraqa%2Fharaqa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haraqa%2Fharaqa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haraqa%2Fharaqa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haraqa%2Fharaqa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/haraqa","download_url":"https://codeload.github.com/haraqa/haraqa/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225476383,"owners_count":17480215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","go","message-queue"],"created_at":"2024-08-04T09:01:43.062Z","updated_at":"2025-12-25T15:06:40.621Z","avatar_url":"https://github.com/haraqa.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003ca href=\"https://haraqa.github.io\"\u003e\n    \u003cimg alt=\"HARAQA\" src=\"https://raw.githubusercontent.com/haraqa/haraqa/media/mascot.png\"/\u003e\n  \u003c/a\u003e\n\u003c/h1\u003e\n\n\u003ch2 align=\"center\"\u003e\u003cstrong\u003eHARAQA\u003c/strong\u003e - High Availability Routing And Queueing Application\u003c/h2\u003e\n\n[![GoDoc](https://godoc.org/github.com/haraqa/haraqa?status.svg)](https://pkg.go.dev/github.com/haraqa/haraqa?tab=doc)\n[![Report Card](https://goreportcard.com/badge/github.com/haraqa/haraqa)](https://goreportcard.com/report/haraqa/haraqa)\n[![License](https://img.shields.io/github/license/haraqa/haraqa.svg)](https://github.com/haraqa/haraqa/blob/master/LICENSE)\n[![Build](https://github.com/haraqa/haraqa/workflows/build/badge.svg)](https://github.com/haraqa/haraqa/blob/master/.github/workflows/go.yml)\n[![Coverage Status](https://coveralls.io/repos/github/haraqa/haraqa/badge.svg)](https://coveralls.io/github/haraqa/haraqa)\n[![Docker Build](https://github.com/haraqa/haraqa/actions/workflows/docker.yml/badge.svg)](https://hub.docker.com/r/haraqa/haraqa/)\n[![Release](https://img.shields.io/github/release/haraqa/haraqa.svg)](https://github.com/haraqa/haraqa/releases)\n\n**haraqa** is designed to be a developer friendly, scalable message queue for data\npersistence and real-time data streaming between microservices. \nHaraqa provides high-throughput, low-latency, fault-tolerant pipelines for architectures of any size.\n\n\n### Table of Contents\n* [About the Project](#about-the-project)\n  * [Overview](#overview)\n  * [Persistence and Replication](#persistence-and-replication)\n* [Getting Started](#getting-started)\n  * [API Docs](#api-docs)\n  * [Server](#server)\n  * [Client](#client)\n* [Contributing](#contributing)\n* [License](#license)\n\n## About the Project\n\n### Overview\nHaraqa is meant for handling and persisting data in a distributed system. \nOne or more servers can be used to send and receive messages. \nEach server has a set of 'topics', a set of messages stored in the order received.\n\nA Haraqa client can produce and/or consume from a server's topics. \nClients can produce messages one at a time or in batches.\nClients consume messages by making a request for a specific offset and limit. \nThe messages can be consumed one at a time or in batches.\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://raw.githubusercontent.com/haraqa/haraqa/media/haraqa_overview.svg\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/haraqa/haraqa/media/haraqa_overview.svg\"/\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n### Persistence and Replication\nEach server, after receiving a message from a producer, can save the message to multiple volumes.\nThese volumes are meant to be distributed in the architecture, such as having\nmultiple PersistentVolumes in a Kubernetes cluster, EBS in AWS, or Persistent Disks in Google Cloud. \nThe server reads messages from the last volume when sending to consumer clients.\n\nIf a volume is removed or corrupted during a restart the server repopulates the data from the other volumes.\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://raw.githubusercontent.com/haraqa/haraqa/media/haraqa_volumes.svg\"\u003e\n    \u003cimg alt=\"Data can be replicated to external volumes and volumes local to where the server runs\" src=\"https://raw.githubusercontent.com/haraqa/haraqa/media/haraqa_volumes.svg\"/\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n\u003c!--\n### Usecases\n* #### Log Aggregation\n  * [Example](https://github.com/haraqa/haraqa/tree/master/internal/examples/logs).\n  Haraqa can be used by services to persist logs for debugging or auditing.\n* #### Message routing between clients\n  * [Example](https://github.com/haraqa/haraqa/tree/master/internal/examples/message_routing).\nhttp clients can send and receive messages asynchronously through POST and GET requests\nto a simple REST server. These messages are stored in haraqa in a topic unique to each client.\n* #### Time series data\n  * [Example](https://github.com/haraqa/haraqa/tree/master/internal/examples/time_series).\n  Metrics can be stored in a topic and later used for graphing or more complex analysis.\n* #### Aggregation for emails or notifications\n  * [Example](https://github.com/haraqa/haraqa/tree/master/internal/examples/emails).\n  Notifications can be aggregated and sent out in batches for daily/weekly emails or push notifications.\n\n--\u003e\n\n## Getting started\n\n### API Docs\n* [Swagger API Documentation](https://haraqa.github.io/haraqa/cmd/server/swagger.html)\n* [Redocs API Documentation](https://haraqa.github.io/haraqa/cmd/server/redocs.html)\n* [Swagger yaml](https://github.com/haraqa/haraqa/blob/master/cmd/server/swagger.yaml)\n\nThe docker server also includes local api documentation at the `/docs/redocs` and `/docs/swagger` endpoints.\n\n### Server\nThe recommended deployment strategy is to use [Docker](https://hub.docker.com/r/haraqa/haraqa)\n```\ndocker run -it -p 4353:4353 -v $PWD/vol1:/vol1 haraqa/haraqa /vol1\n```\n```\ndocker run -it [port mapping] [volume mounts] haraqa/haraqa [flags] [volumes]\n```\n\nTo run from source, navigate to cmd/server and run the main.go file.\n```\ncd cmd/server\ngo run main.go vol1\n```\n\n##### Flags:\n```\n  -http    uint    Port to listen on (default 4353)\n  -cache   boolean Enable queue file caching (default true)\n  -cors    boolean Enable CORS (default true)\n  -docs    boolean Enable Docs pages (default true)\n  -entries integer The number of msg entries per queue file before creating a new file (default 5000)\n  -limit   integer Default batch limit for consumers (default -1)\n  -ballast integer Garbage collection memory ballast size in bytes (default 1073741824)\n  -prometheus boolean Enable prometheus metrics (default true)\n```\n\n##### Volumes:\nVolumes will be written to in the order given and recovered from in the reverse\norder. Consumer requests are read from the last volume. For this reason it's\nrecommended to use a local volume last.\n\nFor instance, given\n```\ndocker run haraqa/haraqa /vol1 /vol2 /vol3\n```\n\nWhen a client produces a message, it will be written to /vol1, then /vol2, then /vol3.\nWhen a client consumes a message, it will be read from /vol3.\n\nDuring recovery, if data exists in /vol3 it will be replicated to volumes /vol1 and /vol2.\nIf /vol3 is empty, /vol2 will be replicated to /vol1 and /vol3.\n\n### Client\n```\ngo get github.com/haraqa/haraqa\n```\n\u003c!---\n##### Client Code Examples\nClient examples can be found in the\n[godoc documentation](https://pkg.go.dev/github.com/haraqa/haraqa?tab=doc#pkg-overview)\n\n##### Additional Examples\nAdditional examples are located in the internal examples folder [internal/examples](https://github.com/haraqa/haraqa/tree/master/internal/examples)\n\n\u003cdetails\u003e\u003csummary\u003eHello World Quickstart\u003c/summary\u003e\n\u003cp\u003e\n\n```\npackage main\n\nimport (\n  \"context\"\n  \"log\"\n\n  \"github.com/haraqa/haraqa\"\n)\n\nfunc main() {\n  client, err := haraqa.NewClient(haraqa.WithAddr(\"127.0.0.1\"))\n  if err != nil {\n    panic(err)\n  }\n  defer client.Close()\n\n  var (\n    ctx    = context.Background()\n    topic  = []byte(\"my_topic\")\n    msg1   = []byte(\"hello\")\n    msg2   = []byte(\"world\")\n    offset = 0\n    limit  = 2048\n  )\n\n  // produce messages in a batch\n  err = client.Produce(ctx, topic, msg1, msg2)\n  if err != nil {\n    panic(err)\n  }\n\n  // consume messages in a batch\n  msgs, err := client.Consume(ctx, topic, offset, limit, nil)\n  if err != nil {\n    panic(err)\n  }\n\n  log.Println(msgs)\n}\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n#### Command Line Client\n\nSee the [hrqa repository](https://github.com/haraqa/hrqa) for more details\n\n```\ngo get github.com/haraqa/hrqa\n```\n--\u003e\n\n## Contributing\n\nWe want this project to be the best it can be and all feedback, feature requests or pull requests are welcome.\n\n## License\n\nMIT © 2019 [haraqa](https://github.com/haraqa/) and [contributors](https://github.com/haraqa/haraqa/graphs/contributors). See `LICENSE` for more information.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharaqa%2Fharaqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharaqa%2Fharaqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharaqa%2Fharaqa/lists"}