{"id":15056726,"url":"https://github.com/walmartlabs/bigben","last_synced_at":"2025-10-04T16:31:47.811Z","repository":{"id":53422738,"uuid":"139198574","full_name":"walmartlabs/bigben","owner":"walmartlabs","description":"BigBen - a generic, multi-tenant, time-based event scheduler and cron scheduling framework","archived":true,"fork":false,"pushed_at":"2021-03-30T20:13:17.000Z","size":973,"stargazers_count":198,"open_issues_count":19,"forks_count":54,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-07-08T03:48:36.156Z","etag":null,"topics":["cassandra","cron","distributed","events","hazelcast"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/walmartlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-29T21:41:14.000Z","updated_at":"2025-06-19T08:08:57.000Z","dependencies_parsed_at":"2022-08-27T18:52:41.604Z","dependency_job_id":null,"html_url":"https://github.com/walmartlabs/bigben","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/walmartlabs/bigben","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walmartlabs%2Fbigben","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walmartlabs%2Fbigben/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walmartlabs%2Fbigben/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walmartlabs%2Fbigben/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/walmartlabs","download_url":"https://codeload.github.com/walmartlabs/bigben/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/walmartlabs%2Fbigben/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278342671,"owners_count":25971396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","cron","distributed","events","hazelcast"],"created_at":"2024-09-24T21:55:51.736Z","updated_at":"2025-10-04T16:31:47.452Z","avatar_url":"https://github.com/walmartlabs.png","language":"Kotlin","funding_links":[],"categories":["任务调度","others","Kotlin"],"sub_categories":[],"readme":"***\n# NOTICE:\n \n## This repository has been archived and is not supported.\n \n[![No Maintenance Intended](http://unmaintained.tech/badge.svg)](http://unmaintained.tech/)\n***\nNOTICE: SUPPORT FOR THIS PROJECT HAS ENDED \n\nThis projected was owned and maintained by Walmart. This project has reached its end of life and Walmart no longer supports this project.\n\nWe will no longer be monitoring the issues for this project or reviewing pull requests. You are free to continue using this project under the license terms or forks of this project at your own risk. This project is no longer subject to Walmart's bug bounty program or other security monitoring.\n\n\n## Actions you can take\n\nWe recommend you take the following action:\n\n  * Review any configuration files used for build automation and make appropriate updates to remove or replace this project\n  * Notify other members of your team and/or organization of this change\n  * Notify your security team to help you evaluate alternative options\n\n## Forking and transition of ownership\n\nFor [security reasons](https://www.theregister.co.uk/2018/11/26/npm_repo_bitcoin_stealer/), Walmart does not transfer the ownership of our primary repos on Github or other platforms to other individuals/organizations. Further, we do not transfer ownership of packages for public package management systems.\n\nIf you would like to fork this package and continue development, you should choose a new name for the project and create your own packages, build automation, etc.\n\nPlease review the licensing terms of this project, which continue to be in effect even after decommission.\n\n# BigBen\n`BigBen` is a generic, multi-tenant, time-based event scheduler and cron \nscheduling framework based on `Cassandra` and `Hazelcast`\n\nIt has following features:\n* **Distributed** - `BigBen` uses a distributed design and can be deployed on 10's or 100's of machines and can be dc-local or cross-dc\n* **Horizontally scalable** - `BigBen` scales linearly with the number of machines. \n* **Fault tolerant** - `BigBen` employs a number of failure protection modes and \ncan withstand arbitrary prolonged down times\n* **Performant** - `BigBen` can easily scale to 10,000's or even millions's of event triggers with a \nvery small cluster of machines. It can also easily manage million's of crons running in a distributed manner\n* **Highly Available** - As long as a single machine is available in the cluster, `BigBen` will guarantee the \nexecution of events (albeit with a lower throughput)\n* **Extremely consistent** - `BigBen` employs a single master design (the master itself is highly available with \n`n-1` masters on standby in an `n` cluster machine) to ensure that no two nodes fire the same event or execute \nthe same cron.  \n* **NoSql based** - `BigBen` comes with default implementation with `Cassandra` but can be easily extended\nto support other `NoSql` or even `RDBMS` data stores  \n* **Auditable** - `BigBen` keeps a track of all the events fired and crons executed with a configurable \nretention \n* **Portable, cloud friendly** - `BigBen` comes as application bundled as `war` or an embedded lib as `jar`, \nand can be deployed on any cloud, `on-prem` or `public`   \n\n## Use cases\n`BigBen` can be used for a variety of time based workloads, both single trigger based or repeating crons. \nSome of the use cases can be\n* **Delayed execution** - E.g. if a job is to be executed 30 mins from now\n* **System retries** - E.g. if a service A wants to call service B and service B is down at the moment, then \nservice A can schedule an exponential backoff retry strategy with retry intervals of \n1 min, 10 mins, 1 hour, 12 hours, and so on.\n* **Timeout tickers** - E.g. if service A sends a message to service B via `Kafka` and expects a response in 1 min, \nthen it can schedule a `timeout check` event to be executed after 1 min\n* **Polling services** - E.g. if service A wants to poll service B at some frequency, it can schedule a cron \nto be executed at some specified frequency\n* **Notification Engine** - `BigBen` can be used to implement `notification engine` with scheduled deliveries, \nscheduled polls, etc\n* **Workflow state machine** - `BigBen` can be used to implement a distributed `workflow` with state suspensions, \nalerts and monitoring of those suspensions.\n\n## Architectural Goals\n`BigBen` was designed to achieve the following goals:\n* Uniformly distributed storage model\n    * Resilient to hot spotting due to sudden surge in traffic\n* Uniform execution load profile in the cluster\n    * Ensure that all nodes have similar load profiles to minimize misfires\n* Linear Horizontal Scaling \n* Lock-free execution\n    * Avoid resource contentions\n* Plugin based architecture to support variety of data bases like `Cassandra, Couchbase, Solr Cloud, Redis, RDBMS`, etc\n* Low maintenance, elastic scaling   \n\n## Design and architecture\nSee the blog published at [Medium](https://medium.com/walmartlabs/an-approach-to-designing-distributed-fault-tolerant-horizontally-scalable-event-scheduler-278c9c380637)\nfor a full description of various design elements of `BigBen`\n\n## Events Inflow\n`BigBen` can receive events in two modes:\n* **kafka** - inbound and outbound Kafka topics to consume event requests and publish event triggers\n* **http** - HTTP APIs to send event requests and HTTP APIs to receive event triggers.\n\n*It is strongly recommended to use `kafka` for better scalability*\n\n### Event Inflow diagram\n![inflow](/docs/assets/inflow.png \"Events Inflow diagram\")\n\n*Request and Response channels can be mixed. For example, the event requests can be sent through HTTP APIs but \nthe event triggers (response) can be received through a Kafka Topic.*\n\n## Event processing guarantees\n`BigBen` has a robust event processing guarantees to survive various failures. \nHowever, `event-processing` is not same as `event-acknowledgement`. \n`BigBen` works in a no-acknowledgement mode (*at least for now*). \nOnce an event is triggered, it is either published to `Kafka` or \nsent through an `HTTP API`. Once the `Kafka` producer returns success, or `HTTP API` returns non-500 status code, \nthe event is **assumed** to be processed and marked as such in the system. \nHowever, for whatever reason if the event was not processed and resulted in an error \n(e.g. `Kafka` producer timing out, or `HTTP API` throwing `503`), \nthen the event will be retried multiple times as per the strategies discussed below\n\n### Event misfire strategy\nMultiple scenarios can cause `BigBen` to be not able to trigger an event on time. Such scenarios are called \nmisfires. Some of them are:\n* `BigBen`'s internal components are down during event trigger. \nE.g. \n    * `BigBen`'s data store is down and events could not be fetched\n    * `VMs` are down\n\n* `Kafka` Producer could not publish due to loss of partitions / brokers or any other reasons\n* `HTTP API` returned a 500 error code\n* Any other unexpected failure\n\nIn any of these cases, the event is first retried in memory using an exponential back-off strategy. \n\nFollowing parameters control the retry behavior:\n\n* _event.processor.max.retries_ - how many in-memory retries will be made before declaring the event as error, default is 3\n* _event.processor.initial.delay_ - how long in seconds the system should wait before kicking in the retry, default is 1 second\n* _event.processor.backoff.multiplier_ - the back off multiplier factor, default is 2. E.g. the intervals would be 1 second, 2 seconds, 4 seconds.\n\nIf the event still is not processed, then the event is marked as `ERROR`. \nAll the events marked `ERROR` are retried up to a configured limit called `events.backlog.check.limit`. \nThis value can be an arbitrary amount of time, e.g. 1 day, 1 week, or even 1 year. E.g. if the the limit\nis set at `1 week` then any event failures will be retried for `1 week` after which, they will be permanently \nmarked as `ERROR` and ignored. The `events.backlog.check.limit` can be changed at any time by changing the \nvalue in `bigben.yaml` file and bouncing the servers.\n\n### Event bucketing and shard size\n`BigBen` shards events by minutes. However, since it's not known in advance how many events will be \nscheduled in a given minute, the buckets are further sharded by a pre defined shard size. The shard size is a \ndesign choice that needs to be made before deployment. Currently, it's not possible to \nchange the shard size once defined. \n\nAn undersized shard value has minimal performance impact, however an oversized shard value may \nkeep some machines idling. The default value of `1000` is good enough for most practical purposes as long as \nnumber of events to be scheduled per minute exceed `1000 x n`, where `n` is the number of machines in the cluster.\nIf the events to be scheduled are much less than `1000` then a smaller shard size may be chosen.      \n\n### Multi shard parallel processing\nEach bucket with all its shards is distributed across the cluster for execution with an algorithm that ensures a \nrandom and uniform distribution. The following diagram shows the execution flow.  \n![shard design](https://cdn-images-1.medium.com/max/1600/1*euaHLOnw6G96SigfXxWhtA.png \"BigBen processing flow\")\n\n### Multi-tenancy\nMultiple tenants can use `BigBen` in parallel. Each one can configure how the events will be delivered once triggered.\nTenant 1 can configure the events to be delivered in `kafka` topic `t1`, where as tenant 2 can have them delivered\nvia a specific `http` url. The usage of tenants will become more clearer with the below explanation of `BigBen` APIs\n\n## Docker support\nBigBen is dockerized and image (`bigben`) is available on docker hub. The code also contains \nscripts, which start `cassandra`, `hazelcast` and `app`.\nTo quickly set up the application for local dev testing, do the following steps:\n1. `git clone $repo`\n2. `cd bigben/build/docker`\n3. execute `./docker_build.sh`\n4. start cassandra container by executing `./cassandra_run.sh`\n5. start app by executing `./app_run.sh`\n6. To run multiple app nodes `export NUM_INSTANCES=3 \u0026\u0026 ./app_run.sh`\n6. wait for application to start on port `8080`\n7. verify that `curl http://localhost:8080/ping` returns `200`\n8. Use `./cleanup.sh` to stop and remove all `BigBen` related containers \n\n## Non-docker execution\n`BigBen` can be run without docker as well. Following are the steps\n1. `git clone $repo`\n2. `cd bigben/build/exec`\n3. execute `./build.sh`\n4. execute `./app_run.sh`\n\n## Env properties\nYou can set the following environment properties\n1. `APP_CONTAINER_NAME` (default bigben_app)\n2. `SERVER_PORT` (default 8080)\n3. `HZ_PORT` (default 5701)\n4. `NUM_INSTANCES` (default 1)\n5. `LOGS_DIR` (default bigben/../bigben_logs)\n6. `CASSANDRA_SEED_IPS` (default $HOST_IP)\n7. `HZ_MEMBER_IPS` (default $HOST_IP)\n8. `JAVA_OPTS`\n\n#How to override default config values?\n`BigBen` employs an extensive override system to allow someone to override \nthe default properties. The order of priority is system properties \u003e system env variables \u003e\noverrides \u003e defaults\nThe overrides can be defined in `config/overrides.yaml` file.\nThe `log4j.xml` can also be changed to change log behavior without \nrecompiling binaries\n\n## How to setup `Cassandra` for `BigBen`?\nFollowing are the steps to set up `Cassandra`:\n1. git clone the `master` branch\n2. Set up a Cassandra cluster\n3. create a keyspace `bigben` in `Cassandra` cluster with desired replication\n4. Open the file `bigben-schema.cql` and execute `cqlsh -f bigben-schema.cql`\n\n## APIs\n\n### cluster\n`GET /events/cluster`\n* response sample (a 3 node cluster running on single machine and three different ports (5701, 5702, 5703)):\n```json\n{\n    \"[127.0.0.1]:5702\": \"Master\",\n    \"[127.0.0.1]:5701\": \"Slave\",\n    \"[127.0.0.1]:5703\": \"Slave\"\n}\n``` \nThe node marked `Master` is the master node that does the scheduling.\n\n### tenant registration\nA tenant can be registered by calling the following API\n\n`POST /events/tenant/register`\n* payload schema\n```json\n{\n  \"$schema\": \"http://json-schema.org/draft-04/schema#\",\n  \"type\": \"object\",\n  \"properties\": {\n    \"tenant\": {\n      \"type\": \"string\"\n    },\n    \"type\": {\n      \"type\": \"string\"\n    },\n    \"props\": {\n      \"type\": \"object\"\n    }\n  },\n  \"required\": [\n    \"tenant\",\n    \"type\",\n    \"props\"\n  ]\n}\n```\n* `tenant` - specifies a tenant and can be any arbitrary value.\n* `type` - specifies the type of `tenant`. One of the three types can be used\n    * MESSAGING - specifies that `tenant` wants events delivered via a messaging queue. Currently, `kafka` \n    is the only supported messaging system.\n    * HTTP - specifies that `tenant` wants events delivered via an http callback URL. \n    * CUSTOM_CLASS - specifies a custom event processor implemented for custom processing of events\n* `props` - A bag of properties needed for each type of tenant. \n\n* kafka sample:\n```json\n{\n    \"tenant\": \"TenantA/ProgramB/EnvC\",\n    \"type\": \"MESSAGING\",\n    \"props\": {\n        \"topic\": \"some topic name\",\n        \"bootstrap.servers\": \"node1:9092,node2:9092\"\n    }\n}\n```\n* http sample\n```json\n{\n     \"tenant\": \"TenantB/ProgramB/EnvC\",\n     \"type\": \"HTTP\",\n     \"props\": {\n          \"url\": \"http://someurl\",\n          \"headers\": {\n            \"header1\": \"value1\",\n            \"header2\": \"value2\"\n          }\n     }\n}\n```     \n\n### fetch all tenants:\n`GET /events/tenants`\n\n### event scheduling\n`POST /events/schedule`\n\n`Payload - List\u003cEventRequest\u003e`\n\n`EventRequest` schema:\n \n```json\n{\n  \"$schema\": \"http://json-schema.org/draft-04/schema#\",\n  \"type\": \"object\",\n  \"properties\": {\n    \"id\": {\n      \"type\": \"string\"\n    },\n    \"eventTime\": {\n      \"type\": \"string\",\n      \"description\": \"An ISO-8601 formatted timestamp e.g. 2018-01-31T04:00.00Z\"\n    },\n    \"tenant\": {\n      \"type\": \"string\"\n    },\n    \"payload\": {\n      \"type\": \"string\",\n      \"description\": \"an optional event payload, must NOT be null with deliveryOption = PAYLOAD_ONLY\"\n    },\n    \"mode\": { \n      \"type\": \"string\",\n      \"enum\": [\"UPSERT\", \"REMOVE\"],\n      \"default\": \"UPSERT\",\n      \"description\": \"Use REMOVE to delete an event, UPSERT to add/update an event\"\n    },\n    \"deliveryOption\": {\n      \"type\": \"string\",\n      \"enum\": [\"FULL_EVENT\", \"PAYLOAD_ONLY\"],\n      \"default\": \"FULL_EVENT\",\n      \"description\": \"Use FULL_EVENT to have full event delivered via kafka/http, PAYLOAD_ONLY to have only the payload delivered\"\n    }\n  },\n  \"required\": [\n    \"id\",\n    \"eventTime\",\n    \"tenant\"\n  ]\n}\n```\n\n### find an event\n`GET /events/find?id=?\u0026tenant=?`\n\n### dry run\n`POST /events/dryrun?id=?\u0026tenant=?`\n\nfires an event without changing its final status\n\n## cron APIs\ncoming up...\n        \n    \n    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwalmartlabs%2Fbigben","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwalmartlabs%2Fbigben","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwalmartlabs%2Fbigben/lists"}