{"id":13498107,"url":"https://github.com/lg/murder","last_synced_at":"2025-09-28T21:30:43.261Z","repository":{"id":786191,"uuid":"481811","full_name":"lg/murder","owner":"lg","description":"Large scale server deploys using BitTorrent and the BitTornado library (NOTE: project no longer maintained)","archived":true,"fork":false,"pushed_at":"2017-01-04T02:33:40.000Z","size":168,"stargazers_count":2529,"open_issues_count":8,"forks_count":275,"subscribers_count":139,"default_branch":"master","last_synced_at":"2024-03-25T14:04:12.253Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://twitter.com","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-01-21T07:05:36.000Z","updated_at":"2024-03-23T14:21:01.000Z","dependencies_parsed_at":"2022-07-05T14:32:10.040Z","dependency_job_id":null,"html_url":"https://github.com/lg/murder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lg%2Fmurder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lg%2Fmurder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lg%2Fmurder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lg%2Fmurder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lg","download_url":"https://codeload.github.com/lg/murder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234563133,"owners_count":18853059,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T20:00:51.016Z","updated_at":"2025-09-28T21:30:42.699Z","avatar_url":"https://github.com/lg.png","language":"Ruby","readme":"Murder by Larry Gadea \u003clg@twitter.com\u003e and Matt Freels \u003cfreels@twitter.com\u003e  \n\nCopyright 2010-2012 Twitter Inc.\n\n**WARNING: This project, though still functional, is no longer maintained**\n\nDESCRIPTION\n-----------\n\nMurder is a method of using Bittorrent to distribute files to a large amount\nof servers within a production environment. This allows for scaleable and fast\ndeploys in environments of hundreds to tens of thousands of servers where\ncentralized distribution systems wouldn't otherwise function. A \"Murder\" is\nnormally used to refer to a flock of crows, which in this case applies to a\nbunch of servers doing something.\n\nFor an intro video, see:\n[Twitter - Murder Bittorrent Deploy System](http://vimeo.com/11280885)\n\n\nQUICK START\n-----------\n\nFor the impatient, `gem install murder` and add these lines to your Capfile:\n\n    require 'murder'\n\n    set :deploy_via, :murder\n    after 'deploy:setup', 'murder:distribute_files'\n    before 'murder:start_seeding', 'murder:start_tracker'\n    after 'murder:stop_seeding', 'murder:stop_tracker'\n\n\nHOW IT WORKS\n------------\n\nIn order to do a Murder transfer, there are several components required to be\nset up beforehand -- many the result of BitTorrent nature of the system. Murder\nis based on BitTornado.\n\n- A torrent tracker. This tracker, started by running the 'murder_tracker.py'\nscript, runs a self-contained server on one machine. Although technically this\nis still a centralized system (everyone relying on this tracker), the\ncommunication between this server and the rest is minimal and normally\nacceptable. To keep things simple tracker-less distribution (DHT) is currently\nnot supported. The tracker is actually just a mini-httpd that hosts a\n/announce path which the Bittorrent clients update their state onto.\n\n- A seeder. This is the server which has the files that you'd like to\ndeploy onto all other servers. The files are placed into a directory\nthat a torrent gets created from. Murder will tgz up the directory and\ncreate a .torrent file (a very small file containing basic hash\ninformation about the tgz file). This .torrent file lets the peers\nknow what they're downloading. The tracker keeps track of which\n.torrent files are currently being distributed. Once a Murder transfer\nis started, the seeder will be the first server many machines go to to\nget pieces. These pieces will then be distributed in a tree-fashion to\nthe rest of the network, but without necessarily getting the parts\nfrom the seeder.\n\n- Peers. This is the group of servers (hundreds to tens of thousands) which\nwill be receiving the files and distributing the pieces amongst themselves.\nOnce a peer is done downloading the entire tgz file, it will continue seeding\nfor a while to prevent a hotspot effect on the seeder.\n\n\nCONFIGURATION AND USAGE\n-----------------------\n\nMurder integrates with Capistrano. The most simple way to use it is as\na deploy strategy, by setting `:deploy_via` to `:murder`. By default,\nmurder makes the same assumptions that cap makes. All servers without\n`:no_release =\u003e true` will act as peers. Additionally, murder will\nautomatically use the first peer as both tracker and seeder. you may\nredefine the `tracker`, `seeder` and `peer` roles yourself to change\nthese defaults, for instance, if you want to set up a dedicated\ntracker.\n\nAll involved servers must have python installed and the related murder\nsupport files (BitTornado, etc.). To upload the support files to the\ntracker, seeder, and peers, run:\n\n    cap murder:distribute_files\n\nBy default, these will go in `shared/murder` in your apps deploy\ndirectory. Override this by setting the variable\n`remote_murder_path`. For convenience, you can add an after hook to\nrun this on `deploy:setup`:\n\n    after 'deploy:setup', 'murder:distribute_files'\n\nBefore deploying, you must start the tracker:\n\n    cap murder:start_tracker\n\nTo have this happen automatically during a deploy, add the following hooks:\n\n    before 'murder:start_seeding', 'murder:start_tracker'\n    after 'murder:stop_seeding', 'murder:stop_tracker'\n\nAt this point you should be able to deploy normally:\n\n    cap deploy\n\n\nMANUAL USAGE (murder without a deploy strategy)\n-----------------------------------------------\n\nMurder can also be used as a general mechanism to distribute files\nacross a generic set of servers. To do so create a Capfile, require\nmurder, and manually define roles:\n\n    require 'rubygems'\n    require 'murder'\n\n    set :remote_murder_path, '/opt/local/murder' # or some other directory\n\n    role :peer, 'host1', 'host2', 'host3', 'host4', 'host5', host6', host7'\n    role :seeder, 'host1'\n    role :tracker, 'host1'\n\nTo distribute a directory of files, first make sure that murder is set\nup on all hosts, then manually run the murder cap tasks:\n\n1. Start the tracker:\n\n        cap murder:start_tracker\n\n2. Create a torrent from a directory of files on the seeder, and start seeding:\n\n        scp -r ./files host1:~/files\n        cap murder:create_torrent tag=\"Deploy1\" files_path=\"~/files\"\n        cap murder:start_seeding tag=\"Deploy1\"\n\n3. Distribute the torrent to all peers:\n\n        cap murder:peer tag=\"Deploy1\" destination_path=\"/tmp\"\n\n4. Stop the seeder and tracker:\n\n        cap murder:stop_seeding tag=\"Deploy1\"\n        cap murder:stop_tracker\n\nWhen this finishes, all peers will have the files in /tmp/Deploy1\n\n\nTASK REFERENCE\n--------------\n\n`distribute_files`:\n  SCPs a compressed version of all files from ./dist (the python Bittorrent\nlibrary and custom scripts) to all server. The entire directory is sent,\nregardless of the role of each individual server. The path on the server is\nspecified by remote_murder_path and will be cleared prior to transferring\nfiles over.\n\n`start_tracker`:\n  Starts the Bittorrent tracker (essentially a mini-web-server) listening on\nport 8998.\n\n`stop_tracker`:\n  If the Bittorrent tracker is running, this will kill the process. Note that\nif it is not running you will receive an error.\n\n`create_torrent`:\n  Compresses the directory specified by the passed-in argument 'files_path'\nand creates a .torrent file identified by the 'tag' argument. Be sure to use\nthe same 'tag' value with any following commands. Any .git directories will be\nskipped. Once completed, the .torrent will be downloaded to your local\n/tmp/TAG.tgz.torrent.\n\n`download_torrent`:\n  Although not necessary to run, if the file from create_torrent was lost, you\ncan redownload it from the seeder using this task. You must specify a valid\n'tag' argument.\n\n`start_seeding`:\n  Will cause the seeder machine to connect to the tracker and start seeding.\nThe ip address returned by the 'host' bash command will be announced to the\ntracker. The server will not stop seeding until the stop_seeding task is\ncalled. You must specify a valid 'tag' argument (which identifies the .torrent\nin /tmp to use)\n\n`stop_seeding`:\n  If the seeder is currently seeding, this will kill the process. Note that if\nit is not running, you will receive an error. If a peer was downloading from\nthis seed, the peer will find another host to receive any remaining data. You\nmust specify a valid 'tag' argument.\n\n`stop_all_seeding`:\n  Identical to stop_seeding, except this will kill all seeding processes. No\n'tag' argument is needed.\n\n`peer`:\n  Instructs all the peer servers to connect to the tracker and start download\nand spreading pieces and files amongst themselves. You must specify a valid\n'tag' argument. Once the download is complete on a server, that server will\nfork the download process and seed for 30 seconds while returning control to\nCapistrano. Cap will then extract the files to the passed in\n'destination_path' argument to destination_path/TAG/*. To not create this tag\nnamed directory, pass in the 'no_tag_directory=1' argument. If the directory\nis empty, this command will fail. To clean it, pass in the\n'unsafe_please_delete=1' argument. The compressed tgz in /tmp is never\nremoved. When this task completes, all files have been transferred and moved\ninto the requested directory.\n\n`stop_all_peering`:\n  Sometimes peers can go on forever (usually because of an error). This\ncommand will forcibly kill all \"murder_client.py peer\" commands that are\nrunning.\n\nCONFIG REFERENCE\n----------------\n\nVariables\n---------\n\n`default_tag`:\n  A tag name to use by default such that a tag parameter doesn't need to be\nmanually entered on every task. Not recommended to be used since files will be\noverwritten.\n\n`default_seeder_files_path`:\n  A path on the seeder's file system where the files to be distributed are\nstored.\n\n`default_destination_path`:\n  A path on the peers' file system where the files that were distributed\nshould be decompressed into.\n\n`remote_murder_path`:\n A path where murder will look for its support files on each host. `cap\nmurder:distribute_files` will upload murder support files here.\n\n`default_temp_path`:\n  The base path to use for temporary files, the default for this is `/tmp`.\nThis can be overridden with the `temp_path` environment variable.\n\nRoles\n-----\n\n`tracker`:\n  Host on which to run the BitTorrent tracker\n\n`seeder`:\n  Host which will be the source of the files to be distributed via BitTorrent\n\n`peers`:\n  All hosts to which files should be distributed\n","funding_links":[],"categories":["Delivery","Python","Ruby","others","Distributed computing projects"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flg%2Fmurder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flg%2Fmurder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flg%2Fmurder/lists"}