{"id":13550704,"url":"https://github.com/Asana/kraken","last_synced_at":"2025-04-03T00:34:40.354Z","repository":{"id":6214003,"uuid":"7445181","full_name":"Asana/kraken","owner":"Asana","description":"Distributed Pubsub Server for Realtime Apps","archived":true,"fork":false,"pushed_at":"2020-12-03T20:57:03.000Z","size":323,"stargazers_count":333,"open_issues_count":1,"forks_count":35,"subscribers_count":284,"default_branch":"master","last_synced_at":"2024-04-16T01:02:12.616Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Asana.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-01-04T18:31:37.000Z","updated_at":"2024-02-24T13:08:10.000Z","dependencies_parsed_at":"2022-09-09T21:50:39.041Z","dependency_job_id":null,"html_url":"https://github.com/Asana/kraken","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asana%2Fkraken","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asana%2Fkraken/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asana%2Fkraken/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asana%2Fkraken/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Asana","download_url":"https://codeload.github.com/Asana/kraken/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246916734,"owners_count":20854511,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:01:36.406Z","updated_at":"2025-04-03T00:34:35.343Z","avatar_url":"https://github.com/Asana.png","language":"Erlang","funding_links":[],"categories":["Erlang","others","Tools per Language"],"sub_categories":["Erlang"],"readme":"![Kraken](https://raw.github.com/Asana/kraken/master/images/kraken_128.png)\n\n### Overview\n\nKraken is a distributed pubsub server that is designed to power collaborative realtime apps like [Asana](http://www.asana.com).\n\nApplications use Kraken to transmit and receive messages through topics. These messages will typically contain just enough information\nto identify the set of data that was changed by the client just before the message was published. When other clients\nreceive these messages, they will figure out which data changed and reload it from the datastore so that they are eventually brought up to date.\n\nKraken is not a general purpose message bus like [RabbitMQ](http://www.rabbitmq.com/).\n\n### Building Kraken for the first time\n\nWe recommend running Kraken with [Erlang R15B03](http://www.erlang.org/download.html) and above, but it will likely work fine with older versions of Erlang as well.\n\nDownload the latest Kraken release (or clone the git project), and then use the following command from the root of the Kraken directory to build Kraken for the first time:\n\n    ./rebar get-deps compile\n\nYou can now run Kraken in the foreground like this:\n\n    bin/kraken run\n\nThis will start Kraken up with the default config. It will listen to new TCP connections on port 12355.\n\n### Running Kraken\n\nKraken comes with a little bash script that provides a handful of useful commands:\n\nRunning Kraken in the foreground, with a live erlang shell\n\n    bin/kraken run\n\nStarting and Stopping Kraken in the background\n\n    bin/kraken start\n    bin/kraken stop\n\nChecking if Kraken is currently running in the background\n\n    bin/kraken status\n\nChanging the Kraken log level while it is running\n\n    bin/kraken change_log_level [debug|info|warn|error]\n\nDumping information about every client queue\n\n    bin/kraken dump_queues\n\nDumping the list of topics for a particular client's queue\n\n    bin/kraken dump_queue_topics \u003cpid from dump_queues\u003e\n\nDumping all topics with a count of subscribers\n\n    bin/kraken dump_topics\n\n### Configuring Kraken\n\nBefore running Kraken in production, you will want to customize some of the config options. Kraken is built as a standard OTP application, so you can modify config options directly from the command line or by specifying a custom erlang config.\n\n#### Supported options\n\n* **pid_file**: If specified, then the system process id of the erlang node will be written to this file.\n* **listen_ip**: The IP address for the Kraken server to listen to new connections on.\n* **tcp_server_port**: The port for the Kraken server to listen to new connections on.\n* **num_router_shards**: The number of router shards to run. A good starting point is 2x the number of cores on the machine.\n* **router_min_fanout_to_warn**: Octopus will log warnings if a message ends up being distributed to this many or more subscribers.\n\n**Specifying options at the command line**\n\nYou can specify Kraken options at the command line when starting Kraken as follows:\n\n    bin/kraken run -kraken num_router_shards 8 -kraken router_min_fanout_to_warn 1000\n\nYou need to prefix each argument with \"-kraken\" to let erlang know that you are customizing the kraken application environment. Erlang lets you run multiple applications on a single node.\n\n**Specifying options in a config file**\n\nKraken options can also be specified in an erlang config file. Here is an example config file:\n\n    [{octopus, [\n       {pid_file, \"/var/run/kraken.pid\"},\n       {log_file, \"/var/log/kraken.log\"},\n       {max_tcp_clients, 30000},\n       {num_router_shards, 8}]}].\n\nIf you stored the config file in /etc/kraken.config, you could tell Erlang to use the config when you start it as follows:\n\n    bin/kraken start -config /etc/kraken\n\nNote that Erlang requires you to exclude the extension when you specify the config file.\n\n### Kraken clients\n\nKraken currently includes two official clients for Erlang and Node.js. The [Kraken protocol](https://github.com/Asana/Kraken/blob/master/src/kraken_memcached.erl) is based on the Memcached protocol, so it shouldn't take very long to create a client in the language of your choice. Please let us know if you create a new client!\n\nHere is an example of working with Kraken using the [Node.js client](https://github.com/Asana/kraken-node-client):\n\n    js\u003e kraken1 = new Kraken(\"localhost\", 12355);\n    js\u003e kranen2 = new Kraken(\"localhost\", 12355);\n    js\u003e kraken1.subscribe([\"topicA\", \"topicB\"]);\n    js\u003e kraken2.publish([\"topicA\"], \"hi there!\");\n    js\u003e console.log(kraken1.receiveMessages());\n    js\u003e kraken2.unsubscribe([\"topicA\"]);\n\nAs you see above, the Memcached based protocol requires clients to poll for new messages. Most good message bus proctocols (like AMQP) have some kind of polling in the form of a heart beat so that clients can detect dead connections sooner than later. In Kraken, the receive command is the way to receive new messages and the heartbeat at the same time. A decent machine should be able to handle thousands of clients polling once every couple of seconds without a problem. It probably wouldn't take very long to add a new protocol to Kraken that pushes messages to clients if you need it! Kraken was designed with the goal of supporting multiple protocols in the future.\n\n### How do I use Kraken to build realtime apps?\n\nKraken was designed to forward data invalidation messages between application servers. It's up to the application designer to figure out how to scope these messages to topics, and what they should contain. For example, a simple TODO list app may have topics corresponding to each of the lists that a user can see. This app would publish invalidation messages corresponding to the ids of tasks that have changed through the topics corresponding to the lists that the task is and was a member of. When other application servers receive these messages, they would reload the state of the tasks referenced in the invalidations messages to ensure they are still up to date.\n\n### How does Kraken scale?\n\nAs far as we know, very well. Kraken has been powering the Asana service since mid 2010, and has yet to crash or fail in any way. At Asana, we have 10s of thousands of clients connected to each Kraken node.\n\nThere are two ways of scaling Kraken beyond a single machine:\n\n1. You can shard the topic space so that each machine is responsible for a portion of the topics. This will typically decrease the total number of messages that a given node needs to process and reduce the amount of memory required to keep track of all the routing information.\n\n2. You can run Kraken nodes that proxy to other Kraken nodes. The proxy nodes will aggregate connections and routing information from their clients and forward on the minimal amount of information necessary to ensure they stay up to date. The proxy nodes then become a single client to the Kraken nodes that they connect to, substantially decreasing the total number of clients and messages that any single Kraken node needs to handle!\n\n### Authors and Contributors\n\nKraken was developed at [Asana](http://www.asana.com/jobs).\nThe original version was written by Kris Rasmussen ([@krisr](https://github.com/krisr)) in 2010.\nIt was rewritten to be retroactive by Samvit Ramadurgam ([@samvit](https://github.com/samvit)) in 2013.\nThe Kraken mascot was designed by Stephanie Hornung.\n\n### Support or Contact\nHaving trouble with Kraken? Check out the documentation at https://github.com/Asana/Kraken/wiki or file an issue at https://github.com/Asana/Kraken/issues and we’ll help you sort it out.\n\n### Current Development\n\n* **Retroactive Subscription**: Sometimes clients want to subscribe to topics retroactively and receive messages that have already flown through the system.\nThis is useful for situations where clients want to avoid repeated synchonous roundtrips to kraken as the set of topics they are interested in expands,\nbut don't want to miss out on messages that get sent between the times when a subscription is needed and when the batch-subscription is actually established.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAsana%2Fkraken","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAsana%2Fkraken","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAsana%2Fkraken/lists"}