{"id":13800564,"url":"https://github.com/gojek/ziggurat","last_synced_at":"2025-12-12T01:11:51.552Z","repository":{"id":37692959,"uuid":"163260126","full_name":"gojek/ziggurat","owner":"gojek","description":"A stream processing framework to build stateless applications on Kafka","archived":false,"fork":false,"pushed_at":"2024-04-29T10:33:16.000Z","size":1610,"stargazers_count":391,"open_issues_count":2,"forks_count":64,"subscribers_count":34,"default_branch":"master","last_synced_at":"2024-05-04T10:03:10.334Z","etag":null,"topics":["clojure","framework","kafka-streams"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gojek.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-27T07:06:12.000Z","updated_at":"2024-05-03T08:02:52.000Z","dependencies_parsed_at":"2022-07-09T04:17:03.661Z","dependency_job_id":"85b2fd96-1f4b-451c-8967-5efcdf7bac7b","html_url":"https://github.com/gojek/ziggurat","commit_stats":{"total_commits":1144,"total_committers":56,"mean_commits":"20.428571428571427","dds":0.75,"last_synced_commit":"0841311f8ddf81a2eaf2a1b634ee3d2f7e50e09b"},"previous_names":["gojektech/ziggurat"],"tags_count":177,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojek%2Fziggurat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojek%2Fziggurat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojek%2Fziggurat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojek%2Fziggurat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gojek","download_url":"https://codeload.github.com/gojek/ziggurat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226213,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","framework","kafka-streams"],"created_at":"2024-08-04T00:01:13.758Z","updated_at":"2025-12-12T01:11:51.510Z","avatar_url":"https://github.com/gojek.png","language":"Clojure","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# Ziggurat\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/gojektech/ziggurat/wiki/images/logo-ziggurat.png\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://travis-ci.com/gojek/ziggurat\"\u003e\n    \u003cimg src=\"https://travis-ci.com/gojek/ziggurat.svg?branch=master\" alt=\"Build Status\" /\u003e\n  \u003c/a\u003e\n  \u003ca href='https://coveralls.io/github/gojek/ziggurat?branch=master'\u003e\n    \u003cimg src='https://coveralls.io/repos/github/gojek/ziggurat/badge.svg?branch=master' alt='Coverage Status' /\u003e\n  \u003c/a\u003e\n  \u003ca href='https://clojars.org/tech.gojek/ziggurat'\u003e\n    \u003cimg src='https://img.shields.io/clojars/v/tech.gojek/ziggurat.svg' alt='Clojars Project' /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n## Table of Contents\n\n- [Wiki](https://github.com/gojek/ziggurat/wiki)\n- [Release Notes](https://github.com/gojek/ziggurat/wiki/Release-Notes)\n- [Upgrade Guide](https://github.com/gojek/ziggurat/wiki/Upgrade-guide)\n- [Description](#description)\n- [Dev Setup](#dev-setup)\n- [Usage](#usage)\n- [Configuration](doc/configuration.md)\n- [Contribution Guidelines](#contribution)\n- [License](#license)\n- [Changelog](CHANGELOG.md)\n\n## Description\n\nZiggurat is a framework built to simplify stream processing on Kafka. It can be used to create a full-fledged Clojure app that reads and processes messages from Kafka. Ziggurat abstracts the following features:\n\n- Reading messages from Kafka\n- Retrying failed messages via RabbitMQ\n- Setting up an HTTP server\n\nRefer to [concepts](doc/CONCEPTS.md) to understand the concepts referred to in this document.\n\n### Important Concepts and Usage Docs\n\n- [Ziggurat HTTP Server](doc/CONCEPTS.md#Http-Server)\n- [Toggle Streams on a Running Actor](doc/CONCEPTS.md#toggle-streams-in-running-actor)\n- [Middlewares in Ziggurat](doc/middleware.md)\n- [Consuming and Publishing Messages to Kafka](doc/kafka_produce_consume.md)\n- [Connecting RabbitMQ and using channels](doc/rmq_channels.md)\n- [Configuration and Config Description](doc/configuration.md)\n- To read about all concepts, please refer the [Concepts file](doc/CONCEPTS.md)\n\n## Dev Setup\n\n### For Mac Users Only\n\n1. Install Clojure: `brew install clojure`\n2. Install Leiningen: `brew install leiningen`\n3. Run docker-compose: `docker-compose up`. This starts:\n  - Kafka on localhost:9092\n  - ZooKeeper on localhost:2181\n  - RabbitMQ on localhost:5672\n4. Run tests: `make test`\n\n### Running a Cluster Setup Locally\n\n- Run `make setup-cluster`. This clears up the volume and starts:\n  - 3 Kafka brokers on localhost:9091, localhost:9092, and localhost:9093\n  - Zookeeper on localhost:2181\n  - RabbitMQ on localhost:5672\n\n### Running Tests via a Cluster\n\n- Run `make test-cluster`. This uses `config.test.cluster.edn` instead of `config.test.edn`.\n\n## Usage\n\nAdd this to your `project.clj`:\n\n```clojure\n[tech.gojek/ziggurat \"4.11.1\"]\n\n\n_Please refer [clojars](https://clojars.org/tech.gojek/ziggurat) for the latest stable version_\n\nTo start a stream (a thread that reads messages from Kafka), add this to your core namespace.\n\n```clojure\n(require '[ziggurat.init :as ziggurat])\n\n(defn start-fn []\n    ;; your logic that runs at startup goes here\n)\n\n(defn stop-fn []\n    ;; your logic that runs at shutdown goes here\n)\n\n(defn main-fn\n  [{:keys [message metadata] :as message-payload}]\n    (println message)\n    :success)\n\n(def handler-fn\n    (-\u003e main-fn\n      (middleware/protobuf-\u003ehash ProtoClass :stream-id)))\n;; Here ProtoClass refers to the fully qualified name of the Java class which the code is used to de-serialize the message.\n\n(ziggurat/main start-fn stop-fn {:stream-id {:handler-fn handler-fn}})\n```\n\n_NOTE: this example assumes that the message is serialized in Protobuf format_\n\nPlease refer the [Middleware section](doc/middleware.md) for understanding `handler-fn` here.\n\n- The main-fn is the function that will be applied to every message that is read from the Kafka stream.\n- The main-fn will take map as an argument that takes 2 keys i.e\n  - message - It is the byte[] array received from kafka.\n  - metadata\n    - topic - It is the topic from where kafka message is consumed.\n    - timestamp - It is ingestion timestamp in kafka.\n    - partition - The partition from message is consumed.\n    - rabbitmq-retry-count - The number of retries done by rabbitmq for given message.\n- The main-fn returns a keyword which can be any of the below words\n  - :success - The message was successfully processed and the stream should continue to the next message\n  - :retry - The message failed to be processed and it should be retried via RabbitMQ.\n  - :dead-letter - The message is not retried and is directly pushed to the dead letter queue\n  - :skip - The message should be skipped without reporting its failure or retrying the message. Same as :success except that a different metric is published to track skipped messages\n- The start-fn is run at the application startup and can be used to initialize connection to databases, http clients, thread-pools, etc.\n- The stop-fn is run at shutdown and facilitates graceful shutdown, for example, releasing db connections, shutting down http servers etc.\n\n\n\n## Multiple stream routes\n- Ziggurat enables reading from multiple streams and applying same/different functions to the messages. `:stream-id` is a unique identifier per stream which needs to be included in config.edn file\n- All configs, queues and metrics will be namespaced under this id.\n\n```clojure\n(ziggurat/main start-fn stop-fn {:stream-id-1 {:handler-fn handler-fn-1}\n                                         :stream-id-2 {:handler-fn handler-fn-2}})\n```\n\n```clojure\n(require '[ziggurat.init :as ziggurat])\n\n(defn start-fn []\n    ;; your logic that runs at startup goes here\n)\n\n(defn stop-fn []\n    ;; your logic that runs at shutdown goes here\n)\n\n\n(defn api-handler [_request]\n  {:status  200\n   :headers {\"Content-Type\" \"application/json\"}\n   :body    (get-resource)})\n\n(def routes [[\"v1/resources\" {:get api-handler}]])\n\n(defn main-fn\n  [{:keys [message metadata] :as message-payload}]\n    (println message)\n    :success)\n\n(def handler-fn\n    (-\u003e main-fn\n      (middleware/protobuf-\u003ehash ProtoClass :stream-id)))\n\n(ziggurat/main start-fn stop-fn {:stream-id {:handler-fn handler-fn}} routes)\n\n```\n\n_NOTE: this example assumes that the message is serialized in Protobuf format_\n\n\n## Deprecation Notice\n* Sentry has been deprecated from version 4.6.3. \n\n## Contribution\n\n- For dev setup and contributions please refer to CONTRIBUTING.md\n\n## License\n\n```\nCopyright 2018, GO-JEK Tech \u003chttp://gojek.tech\u003e\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgojek%2Fziggurat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgojek%2Fziggurat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgojek%2Fziggurat/lists"}