{"id":15048326,"url":"https://github.com/github/kestrel","last_synced_at":"2025-10-04T08:32:00.799Z","repository":{"id":65974859,"uuid":"107738299","full_name":"github/kestrel","owner":"github","description":"simple, distributed message queue system (inactive)","archived":true,"fork":true,"pushed_at":"2016-01-22T07:54:28.000Z","size":249309,"stargazers_count":5,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-09-25T21:10:43.862Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://twitter.github.io/kestrel","language":"Scala","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"twitter-archive/kestrel","license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/github.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-21T00:17:08.000Z","updated_at":"2024-07-31T03:20:54.000Z","dependencies_parsed_at":"2023-02-19T18:01:03.173Z","dependency_job_id":null,"html_url":"https://github.com/github/kestrel","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fkestrel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fkestrel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fkestrel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Fkestrel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/github","download_url":"https://codeload.github.com/github/kestrel/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219876581,"owners_count":16554769,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T21:10:50.571Z","updated_at":"2025-10-04T08:31:55.467Z","avatar_url":"https://github.com/github.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Kestrel\n=======\n\n[![Project status](https://img.shields.io/badge/status-deprecated-orange.svg)](#status)\n\nKestrel is based on Blaine Cook's \"starling\" simple, distributed message\nqueue, with added features and bulletproofing, as well as the scalability\noffered by actors and the JVM.\n\nEach server handles a set of reliable, ordered message queues. When you put a\ncluster of these servers together, *with no cross communication*, and pick a\nserver at random whenever you do a `set` or `get`, you end up with a reliable,\n*loosely ordered* message queue.\n\nIn many situations, loose ordering is sufficient. Dropping the requirement on\ncross communication makes it horizontally scale to infinity and beyond: no\nmulticast, no clustering, no \"elections\", no coordination at all. No talking!\nShhh!\n\nFor more information about what it is and how to use it, check out\nthe included [guide](https://github.com/robey/kestrel/blob/master/docs/guide.md).\n\nKestrel has a mailing list here:\n[kestrel-talk@googlegroups.com](http://groups.google.com/group/kestrel-talk)\n\nAuthor's address: Robey Pointer \\\u003crobeypointer@gmail.com\u003e\n\nStatus\n------\n\nWe've deprecated Kestrel because internally we've shifted our attention to an alternative project based on DistributedLog, and we no longer have the resources to contribute fixes or accept pull requests. While Kestrel is a great solution up to a certain point (simple, fast, durable, and easy to deploy), it hasn't been able to cope with Twitter's massive scale (in terms of number of tenants, QPS, operability, diversity of workloads etc.) or operating environment (an Aurora cluster without persistent storage).\n\nFeatures\n--------\n\nKestrel is:\n\n- fast\n\n    It runs on the JVM so it can take advantage of the hard work people have\n    put into java performance.\n\n- small\n\n    Currently about 2500 lines of scala, because it relies on Netty (a rough\n    equivalent of Danger's ziggurat or Ruby's EventMachine) -- and because\n    Scala is extremely expressive.\n\n- durable\n\n    Queues are stored in memory for speed, but logged into a journal on disk\n    so that servers can be shutdown or moved without losing any data.\n\n- reliable\n\n    A client can ask to \"tentatively\" fetch an item from a queue, and if that\n    client disconnects from kestrel before confirming ownership of the item,\n    the item is handed to another client. In this way, crashing clients don't\n    cause lost messages.\n\n\nAnti-Features\n-------------\n\nKestrel is not:\n\n- strongly ordered\n\n    While each queue is strongly ordered on each machine, a cluster will\n    appear \"loosely ordered\" because clients pick a machine at random for\n    each operation. The end result should be \"mostly fair\".\n\n- transactional\n\n    This is not a database. Item ownership is transferred with acknowledgement,\n    but kestrel does not support grouping multiple operations into an atomic\n    unit.\n\n\nDownloading it\n--------------\n\nThe latest release is always on the homepage here:\n\n- [http://robey.github.io/kestrel/](http://robey.github.io/kestrel/)\n\nOr the latest development versions \u0026 branches are on github:\n\n- [http://gitub.com/robey/kestrel](https://github.com/twitter/kestrel)\n\n\nBuilding it\n-----------\n\nKestrel requires java 6 and sbt 0.11.2. Presently some sbt plugins used by kestrel\ndepend on that exact version of sbt. On OS X 10.5, you may have to hard-code\nan annoying `JAVA_HOME` to use java 6:\n\n    $ export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home\n\nBuilding from source is easy:\n\n    $ sbt clean update package-dist\n\nScala libraries and dependencies will be downloaded from maven repositories\nthe first time you do a build. The finished distribution will be in `dist`.\n\n\nRunning it\n----------\n\nYou can run kestrel by hand, in development mode, via:\n\n    $ ./dist/kestrel-VERSION/scripts/devel.sh\n\nLike all ostrich-based servers, it uses the \"stage\" property to determine\nwhich config file to load, so `devel.sh` sets `-Dstage=development`.\n\nWhen running it as a server, a startup script is provided in\n`dist/kestrel-VERSION/scripts/kestrel.sh`. The script assumes you have\n`daemon`, a standard daemonizer for Linux, but also available\n[here](http://libslack.org/daemon/) for all common unix platforms.\n\nThe created archive `kestrel-VERSION.zip` can be expanded into a place\nlike `/usr/local` (or wherever you like) and executed within its own folder as\na self-contained package. All dependent jars are included. The current startup\nscript, however, assumes that kestrel has been deployed to\n`/usr/local/kestrel/current` (e.g., as if by capistrano), and the startup\nscript loads kestrel from that path.\n\nThe default configuration puts logfiles into `/var/log/kestrel/` and queue\njournal files into `/var/spool/kestrel/`.\n\nThe startup script logs extensive GC information to a file named `stdout` in\nthe log folder. If kestrel has problems starting up (before it can initialize\nlogging), it will usually appear in `error` in the same folder.\n\n\nConfiguration\n-------------\n\nQueue configuration is described in detail in `docs/guide.md` (an operational\nguide). Scala docs for the config variables are\n[here](http://robey.github.io/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html).\n\n\nPerformance\n-----------\n\nSeveral performance tests are included. To run them, first start up a kestrel instance\nlocally.\n\n    $ sbt clean update package-dist\n    $ ./dist/kestrel-VERSION/scripts/devel.sh\n\n## Put-many\n\nThis test just spams a kestrel server with \"put\" operations, to see how\nquickly it can absorb and journal them.\n\nA sample run on a 2010 MacBook Pro:\n\n    $ ./dist/kestrel/scripts/load/put-many -n 100000\n    Put 100000 items of 1024 bytes to localhost:22133 in 1 queues named spam\n      using 100 clients.\n    Finished in 6137 msec (61.4 usec/put throughput).\n    Transactions: min=71.00; max=472279.00 472160.00 469075.00;\n      median=3355.00; average=5494.69 usec\n    Transactions distribution: 5.00%=485.00 10.00%=1123.00 25.00%=2358.00\n      50.00%=3355.00 75.00%=4921.00 90.00%=7291.00 95.00%=9729.00\n      99.00%=50929.00 99.90%=384638.00 99.99%=467899.00\n\n## Many-clients\n\nThis test has one producer that trickles out one item at a time, and a pile of\nconsumers fighting for each item. It usually takes exactly as long as the\nnumber of items times the delay, but is useful as a validation test to make\nsure kestrel works as advertised without blowing up.\n\nA sample run on a 2010 MacBook Pro:\n\n    $ ./dist/kestrel/scripts/load/many-clients\n    many-clients: 100 items to localhost using 100 clients, kill rate 0%,\n      at 100 msec/item\n    Received 100 items in 11046 msec.\n\nThis test always takes about 11 seconds -- it's a load test instead of a\nspeed test.\n\n## Flood\n\nThis test starts up one producer and one consumer, and just floods items\nthrough kestrel as fast as it can.\n\nA sample run on a 2010 MacBook Pro:\n\n    $ ./dist/kestrel/scripts/load/flood\n    flood: 1 threads each sending 10000 items of 1kB through spam\n    Finished in 1563 msec (156.3 usec/put throughput).\n    Consumer(s) spun 0 times in misses.\n\n## Packing\n\nThis test starts up one producer and one consumer, seeds the queue with a\nbunch of items to cause it to fall behind, then does cycles of flooding items\nthrough the queue, separated by pauses. It's meant to test kestrel's behavior\nwith a queue that's fallen behind and *stays* behind indefinitely, to make\nsure the journal files are packed periodically without affecting performance\ntoo badly.\n\nA sample run on a 2010 MacBook Pro:\n\n    $ ./dist/kestrel/scripts/load/packing -c 10 -q small\n    packing: 25000 items of 1kB with 1 second pauses\n    Wrote 25000 items starting at 0.\n    cycle: 1\n    Wrote 25000 items starting at 25000.\n    Read 25000 items in 5279 msec. Consumer spun 0 times in misses.\n    cycle: 2\n    Wrote 25000 items starting at 50000.\n    Read 25000 items in 4931 msec. Consumer spun 0 times in misses.\n    ...\n    cycle: 10\n    Wrote 25000 items starting at 250000.\n    Read 25000 items in 5304 msec. Consumer spun 0 times in misses.\n    Read 25000 items in 3370 msec. Consumer spun 0 times in misses.\n\nYou can see the journals being packed in the kestrel log. Like\n\"many-clients\", this test is a load test instead of a speed test.\n\n## Leaky-reader\n\nThis test starts a producer and several consumers, with the consumers\noccasionally \"forgetting\" to acknowledge an item that they've read. It\nverifies that the un-acknowledged items are eventually handed off to another\nconsmer.\n\nA sample run:\n\n    $ ./dist/kestrel/scripts/load/leaky-reader -n 100000 -t 10\n    leaky-reader: 10 threads each sending 100000 items through spam\n    Flushing queues first.\n    1000\n    2000\n    100000\n    Finished in 40220 msec (40.2 usec/put throughput).\n    Completed all reads\n\nLike \"many-clients\", it's just a load test.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Fkestrel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgithub%2Fkestrel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Fkestrel/lists"}