{"id":13837407,"url":"https://github.com/ezmobius/nanite","last_synced_at":"2025-07-10T18:33:10.761Z","repository":{"id":388447,"uuid":"5964","full_name":"ezmobius/nanite","owner":"ezmobius","description":"self assembling fabric of ruby daemons","archived":false,"fork":false,"pushed_at":"2010-12-22T10:21:32.000Z","size":29179,"stargazers_count":734,"open_issues_count":13,"forks_count":63,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-07-02T14:51:57.989Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ezmobius.png","metadata":{"files":{"readme":"README.rdoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2008-03-31T08:32:08.000Z","updated_at":"2025-03-15T00:01:22.000Z","dependencies_parsed_at":"2022-07-16T14:18:24.764Z","dependency_job_id":null,"html_url":"https://github.com/ezmobius/nanite","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/ezmobius/nanite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezmobius%2Fnanite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezmobius%2Fnanite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezmobius%2Fnanite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezmobius%2Fnanite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ezmobius","download_url":"https://codeload.github.com/ezmobius/nanite/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezmobius%2Fnanite/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264631213,"owners_count":23640941,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T15:01:08.340Z","updated_at":"2025-07-10T18:33:10.428Z","avatar_url":"https://github.com/ezmobius.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"= Nanite : A self-assembling fabric of Ruby daemons\n\nGoogle Group:  http://groups.google.com/group/nanite\nirc.freenode.net:  #nanite\n\n== Intro\n\nNanite is a new way of thinking about building cloud ready web applications. Having\na scalable message queueing back-end with all the discovery and dynamic load based\ndispatch that Nanite has is a very scalable way to construct web application back-ends.\n\nA Nanite system has two types of components. There are nanite agents, these are the\ndaemons where you implement your code functionality as actors. And then there are\nmappers.\n\nMappers are the control nodes of the system. There can be any number of mappers, these\ntypically run inside of your merb or rails app running on the thin webserver\n(eventmachine is needed) but you can also run command line mappers from the shell.\n\nEach Nanite agent sends a ping to the mapper exchange every @ping_time seconds. All of\nthe mappers are subscribed to this exchange and they all get a copy of the ping with your\nstatus update. If the mappers do not get a ping from a certain agent within a timeout\n@ping_time the mappers will remove that agent from any dispatch.  When the agent comes\nback online or gets less busy it will re-advertise itself to the mapper exchange therefore\nadding itself back to the dispatch.  This makes for a very nice self-healing cluster of\nworker processes and a cluster of front-end mapper processes.\n\nIn your Nanites you can have any number of actor classes.  These actors are like controllers\nin Rails or Merb and this is where you implement your own custom functionality.  An actor looks\nlike:\n\n  class Foo\n    include Nanite::Actor\n    expose :bar\n    def bar(payload)\n      \"got payload: #{payload}\"\n    end\n  end\n  Nanite::Dispatcher.register(Foo.new)\n\nThe methods that you 'expose' on the actors are advertised to the mappers like:\n\n  Foo#bar   =\u003e /foo/bar\n  Mock#list =\u003e /mock/list\n\nEvery agent advertises its status every time it pings the mapper cluster.\nThe default status that is advertised is the load average as a float. This\nis used for the default request dispatching based on least loaded server.\n\nYou can change what is advertised as status to anything you want that is\ncomparable(\u003c=\u003e) by doing something like this in your agent's init.rb file:\n\n  status_proc = lambda { MyApp.some_statistic_indicating_load }\n\nThis proc will be recalled every @ping_time and sent to the mappers.\n\nThis is the 'fitness function' for selecting the least loaded Nanite. You can\ndream up any scheme of populating this function so the mappers can select the\nright Nanites based on their status.\n\n\n== A quick note about security:\n\nNanite security is based upon RabbitMQ vhosts so anything attached to one vhost\ncan talk to anything else on the same vhost.  So you generally want one vhost\nper app space.\n\n\n== Installation\n\nNanite has a lot of moving parts.  Follow the directions below and we'll get you up\nand running in no-time.\n\n=== Install Erlang (OS X)\n\nSee the Erlang website for the latest info : http://www.erlang.org/download.html\n\nIn your chosen source dir which we'll refer to as \u003cSRC\u003e:\n\n  cd \u003cSRC\u003e\n  wget http://www.erlang.org/download/otp_src_R12B-5.tar.gz\n  tar -zxvf otp_src_R12B-5.tar.gz\n  cd otp_src_R12B-5\n  ./configure --enable-hipe --enable-darwin-universal\n  make\n  sudo make install\n\n=== Install Erlang (Linux .deb)\n\n  sudo apt-get install erlang-nox\n\n=== Install Nanite Ruby Gem\n\nInstalling the gem gives us access to the various Nanite commands in the default binary path.\n\n  cd \u003cSRC\u003e\n  git clone git://github.com/ezmobius/nanite.git\n  cd nanite\n  rake gem\n  sudo gem install pkg/nanite-\u003cVERSION\u003e.gem\n\n\n=== Install EventMachine Ruby Gem\n\n  sudo gem install eventmachine\n\n=== Install AMQP Ruby Gem\n\n  # install version \u003e= 0.6.0 from RubyForge\n  sudo gem install amqp\n\n  # or install from source\n  cd \u003cSRC\u003e\n  git clone git://github.com/tmm1/amqp.git\n  cd amqp \u0026\u0026 rake gem \u0026\u0026 sudo gem install amqp-\u003cVERSION\u003e.gem\n\n=== Install RabbitMQ from source tarball (OS X and generic Linux)\n\nIn short, you'll need a working erlang installation, python, and simplejson. If your python (\"python -V\") is 2.6, simplejson is included\nin your distribution. Otherwise, install simplejson:\n\n  easy_install simplejson\n\nIf you don't have easy_install, install setuptools from http://pypi.python.org/pypi/setuptools:\n\n  # Get the latest .egg file for your version of python (\"python -V\"), then run it:\n  wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg\n  sh setuptools-0.6c9-py2.5.egg\n\nand install simplejson, which is needed for the \"make\" step below.\n\nThese instructions assume the latest RabbitMQ release 1.5.3:\n\n  # Download somewhere\n  cd /root\n  wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.3/rabbitmq-server-1.5.3.tar.gz\n  \n  # Go to your erlang lib directory, usually /usr/lib/erlang/lib or:\n  cd /usr/local/lib/erlang/lib\n  \n  tar -zxf ~/rabbitmq-server-1.5.3.tar.gz\n  cd rabbitmq-server-1.5.3\n  make\n  \n  # There is no \"make install\" phase.\n\nBe sure to add the /usr/local/lib/erlang/lib/rabbitmq-server-1.5.3/scripts to your $PATH.\n\nThe following websites may also be useful:\n* RabbitMQ website for the latest info : http://www.rabbitmq.com/download.html\n* RabbitMQ server source download : http://www.rabbitmq.com/server.html\n* RabbitMQ build instructions : http://www.rabbitmq.com/build-server.html\n* RabbitMQ install instructions : http://www.rabbitmq.com/install.html\n\n\n=== Install RabbitMQ (Linux .deb)\n\n  wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.3/rabbitmq-server_1.5.3-1_all.deb\n  sudo apt-get install logrotate\n  sudo dpkg -i rabbitmq-server_1.5.3-1_all.deb\n\n== Test your installation\n\n=== Test your Erlang install\n\nStart an Erlang shell\n\n  erl\n\nEnter the following commands in the Erlang shell (When installed correctly each should print a great deal of erlang config info):\n\n  rabbit:module_info().\n\nExit the Erlang shell with the following command (or Ctrl-c):\n\n  q().\n\n\n=== Start RabbitMQ\n\nAll directories will be automatically set up on first run.\n\nTo run RabbitMQ in the foreground:\n  sudo rabbitmq-server\n\nTo run RabbitMQ in the background:\n  sudo rabbitmq-server -detached\n\nTo check status:\n  rabbitmqctl status\n\nTo stop server:\n  rabbitmqctl stop\n\nYou can learn more about RabbitMQ admin here: http://www.rabbitmq.com/admin-guide.html\n\n\n=== Test Ruby/EventMachine/AMQP end-to-end with a running RabbitMQ instance\n\nYou can test that all the moving parts are working end-to-end by running one of the AMQP example programs.  There are a number of example tests provided with AMQP but the one we will try simply queues and prints two messages immediately, and another one five seconds later.\n\n  # this test can only be run if you made a local clone\n  # of the amqp repository during the amqp installation above\n  # (it does not matter however if the gem was installed from src or rubyforge)\n  cd \u003cSRC\u003e/amqp\n  ruby examples/mq/simple-get.rb\n\n\n=== Test Nanite (finally)\n\nFirst we'll do a little setup and run a script which adds an agent account (nanite), a mapper account (mapper)\nand a '/nanite' vhost to RabbitMQ.  RabbitMQ broker must of course be running before you run this script.\n\n  cd nanite\n  ./examples/rabbitconf.rb\n\nNow lets run a few agents.  Each of these is a long running process and needs to run in its own shell.  Each agent also needs its own unique identity, or token to differentiate it from the other running agents.  You can try opening a couple of them now if you like.  The agents will generally provide no output when running.  You can always get more help about the nanite command with 'nanite --help'.\n\nFirst shell\n\n  cd examples/simpleagent\n  nanite-agent --token fred\n\nSecond shell\n\n  cd examples/simpleagent\n  nanite-agent --token bob\n\nNow run a mapper. Mappers can be run from within your Merb or Rails app, from an interactive irb shell, or from the command line.  For this example we'll run it from the command line so open a third shell window and run the following:\n\n  cd examples/simpleagent\n  ./cli.rb\n\nWhich should soon return something like the following.\n\n  {\"bob\"=\u003e\"hello nanite\"}                                 # where the '--token bob' parameter was passed\n  {\"55a7f300c454203eacc218b6fbd2edc6\"=\u003e\"hello nanite\"}    # where no '--token' was passed.  auto-generated identity.\n\nNow if you want to make this interesting, you can issue a Ctrl-c in one of the agent's windows to kill it.  And then run cli.rb again.  You should see that you still get a result back since the mapper is finding the remaining agent to do its work.\n\nIf you want to try requesting work to be done by an agent from an interactive command shell try the following. This assumes that you have the agents running as indicated in the example above ('\u003e\u003e' is the nanite shell prompt).\n\n  cd nanite\n  ./bin/nanite-mapper -i -u mapper -p testing -v /nanite\n  Starting mapper console\n  \u003e\u003e request('/simple/echo') {|res| p res }\n\nBy default this will dispatch to the agent with the lowest reported load average.\n\nThere are a few other selectors as well:\n\n  # run this request on *all* agents that expose the /foo/bar Foo#bar actor\n  \u003e\u003e request('/foo/bar', 'hi', :selector =\u003e :all) {|res| p res }\n\n  # run this request on one random agent that expose the /whatever/hello Whatever#hello actor\n  \u003e\u003e request('/whatever/hello', 42, :selector =\u003e :random) {|res| p res }\n\nYou can create your own selectors based on arbitrary stuff you put in status from your agents see cluster.rb for examples of how least_loaded, all and random are implemented.\n\nYou can run as many mappers as you want, they will all be hot masters.\n\nThe calls are asynchronous. This means the block you pass to Nanite::Agent#request is not run until the response from the agent(s) have returned. So keep that in mind. Should you need to poll from an ajax web app for results you should have your block stuff the results in the database for any web front end to pick up with the next poll.\n\nAnother option to test your agents is to use nanite-admin\n\n  $ nanite-admin \n  starting nanite-admin\n  \u003e\u003e Thin web server (v1.0.1 codename ?)\n  \u003e\u003e Maximum connections set to 1024\n  \u003e\u003e Listening on 0.0.0.0:4000, CTRL+C to stop\n\nand bring up the Nanite Control Tower at http://localhost:4000 .\n\nHave fun!\n\n== Web Framework Integration\n\nThe integration of Nanite into web frameworks depends on the web server running our application and, more importantly, if it uses EventMachine or not.\n\nThin is EventMachine-based, so we only need to make sure that the EventMachine reactor is already 'heated' up\n\n  Thread.new do\n    until EM.reactor_running?\n      sleep 1\n    end\n    Nanite.start_mapper(:host =\u003e 'localhost', :user =\u003e 'mapper', :pass =\u003e 'testing', :vhost =\u003e '/nanite', :log_level =\u003e 'info')\n  end\n  \nMongrel on the other hand does not use EventMachine and therefore requires to wrap the start of our mapper\n  \n  Thread.new do\n    EM.run do \n      Nanite.start_mapper(:host =\u003e 'localhost', :user =\u003e 'mapper', :pass =\u003e 'testing', :vhost =\u003e '/nanite', :log_level =\u003e 'info')\n    end\n  end\n\nUsing nanite with Passenger:\n\nif defined?(PhusionPassenger)\n PhusionPassenger.on_event(:starting_worker_process) do |forked|\n   if forked\n     if EM.reactor_running?\n       EM.stop_event_loop\n       EM.release_machine\n       EM.instance_variable_set( '@reactor_running', false )\n     end\n     Thread.current[:mq] = nil\n     AMQP.instance_variable_set('@conn', nil)\n   end\n\n   th = Thread.current\n   Thread.new{\n     Nanite.start_mapper(:host =\u003e 'localhost', :user =\u003e 'mapper', :pass =\u003e 'testing', :vhost =\u003e '/nanite', :log_level =\u003e 'info')\n   }\n   Thread.stop\n end\nend\n\n=======\nWhere to put the mapper initialization code depends on the framework and our preference. \nFor Rails the canonical place to start our mapper is within nanite.rb (or any other filename you prefer) in config/initalizers.\nIn Merb we can use init.rb in config.\n\n== Security\n\nNanite implements a secure serializer which can be used in place of the other serializers to encrypt the \nAMQP messages exchanged between the mappers and the agents.\n\nThe secure serializer uses X.509 certificates and cryptographic keys to sign and encrypt the messages.\n\nIt is important to understand that:\n1. A certificate only includes the public key component of a cryptographic key \n   pair.\n2. Signing requires the use of a certificate and its private key, checking the\n   signature then only requires the certificate (the idea is that only the\n   signer has the secret private key and thus can sign but anyone can check the \n   signature).\n3. Encrypting only requires the certificate but decrypting also requires the\n   private key (anyone can encrypt the data but only the intended recipient can\n   decrypt it).\n\nA signing serializer thus needs access to the signer certificate and private \nkey. An encrypting serializer *also* needs access to the intended recipients\ncertificates. There needs to be a way to dynamically retrieve the corresponding\ncertificates. This is done using certificate stores.\n\nCertificate stores associate identities with certificates. The identity is \nassociated when the data is serialized and can be keyed off to retrieve the \nright certificate upon deserialization.\n\nNanite provides a static store implementation which can be used when the \ncertificates used for serialization are always the same and can be kept in \nmemory. Nanite also provides a certificate store proxy cache which can be\nassociated with any store implementation and will cache the most used \ncertificates.\n\nThe serializer should be initialized prior to being used by calling the 'init'\nmethod:\n\n  # Initialize serializer, must be called prior to using it.\n  #\n  #  - 'identity':   Identity associated with serialized messages\n  #  - 'cert':       Certificate used to sign serialized messages and\n  #                  decrypt encrypted messages\n  #  - 'key':        Private key corresponding to 'cert'\n  #  - 'store':      Certificate store. Exposes certificates used for\n  #                  encryption and signature validation.\n  #  - 'encrypt':    Whether data should be signed and encrypted ('true')\n  #                  or just signed ('false'), 'true' by default.\n  #\n  def SecureSerializer.init(identity, cert, key, store, encrypt = true)\n\nThe 'secure' agent example (examples\\secure) shows how the mappers and agents should be\nconfigured to use the secure serializer.\n\n== Troubleshooting\n\n** IMPORTANT **\nIf you are using Apple's built in Ruby that comes with Leopard (10.5.x) or Tiger (10.4.x) then your\nREADLINE lib is hosed and the interactive shell will not work. As soon as you drop into a shell\nApple's fakey READLINE will halt the event machine loop so you won't see any nanites\nregistering. I don't have a good workaround except to tell you not to use Apple's Ruby,\nbuild your own or use ports.\n\n\n** What to do if rabbitconf.rb dies with: {badrpc,nodedown} and nothing you do seems to matter **\n\nIf rabbitconf.rb dies saying \"{badrpc,nodedown}\" it means that for some reason,\nthe rabbitmqctl program rabbitconf.rb is using to setup the agent accounts for nanite is\nnot able to connect to your RabbitMQ server. Assuming RabbitMQ is running and is\nknown to work (try the examples that come with the amqp library), then there's a chance\nsomething is going wrong with using short node names in your Erlang install.\n\nThe easiest way to verify it is by starting two separate Erlang shells like this (note that \"odin\" is my hostname):\n\n  $ erl -sname fred\n  (fred@odin)1\u003e\n\n  $ erl -sname bob\n  (bob@odin)1\u003e\n\nAnd then trying to 'ping' one from the other. In the 'fred' node you can do that like this:\n\n  (fred@odin)1\u003e net_adm:ping(bob@odin).\n  pang\n\nNote : If your hostname has a hyphen in it (e.g. macbook-pro) you should include the user@hostname in quotes like:\n\n  net_adm:ping('bob@macbook-pro').\n\nIf you see 'pang' (which is apparently Swedish for something like \"crap! it's broken.\") then short name distribution isn't working for you, and you need to fall back to using a full name. If you see 'pong', then that's not actually your problem.\n\nFirst, verify that your system's hostname is set to a fully qualified domain name. On OS X it can end in '.local':\n\n  hostname\n  odin.local\n\nThen test that *this* will work by starting a node with the full name of 'fred@\u003cyour hostname\u003e':\n\n  erl -name fred@odin.local\n  (fred@odin.local)1\u003e\n\nthen bob@\u003cyourhostname\u003e, and then finally try to ping fred:\n\n  erl -name bob@odin.local\n  (bob@odin.local)1\u003e net_adm:ping(fred@odin.local).\n  pong\n\nIn my case, it looks like that worked. Now... on to getting rabbitconf.rb to run! To do that, you need to edit the 'rabbitmq-server' and 'rabbitmqctl' scripts in your RabbitMQ distribution and edit the -sname arguments to use -name and a full name.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fezmobius%2Fnanite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fezmobius%2Fnanite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fezmobius%2Fnanite/lists"}