{"id":16733097,"url":"https://github.com/quchen/amoeba","last_synced_at":"2025-03-23T15:31:02.781Z","repository":{"id":8784635,"uuid":"10474119","full_name":"quchen/amoeba","owner":"quchen","description":"Amœba is a distributed network.","archived":false,"fork":false,"pushed_at":"2014-05-16T19:49:30.000Z","size":5948,"stargazers_count":18,"open_issues_count":18,"forks_count":1,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-18T21:50:58.712Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quchen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-06-04T08:26:16.000Z","updated_at":"2023-10-02T12:29:53.000Z","dependencies_parsed_at":"2022-09-24T00:24:12.036Z","dependency_job_id":null,"html_url":"https://github.com/quchen/amoeba","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quchen%2Famoeba","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quchen%2Famoeba/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quchen%2Famoeba/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quchen%2Famoeba/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quchen","download_url":"https://codeload.github.com/quchen/amoeba/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245122701,"owners_count":20564358,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T23:48:41.087Z","updated_at":"2025-03-23T15:31:01.416Z","avatar_url":"https://github.com/quchen.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"Amœba\n======\n\nAmœba is a program for setting up a distributed network. The name comes from\nthe hope that eventually, the network will be so robust that you can poke any of\nits parts without endangering its overall integrity.\n\nThis is how a small network looks like. It consists of 359 nodes, each trying to\nmaintain 5-10 incoming and outgoing connections. Initially, three independent\nclusters were created; the last 30 nodes knew of these clusters and joined them\ntogether over the course of a minute. Darker colours indicate a more central\nrole of a node in the network ([betweenness centrality][wiki-betweenness]).\nClick for higher resolution.\n\n[![(Picture missing, uh oh)](doc/network_example.png)][network-hires]\n\n[network-hires]: doc/network_example_hires_amoeba_n359_5-10-3cluster.png\n\n\n\nThe current development stage is unstable alpha.\n\nBranch  | Status\n:-----: | :-----:\nmaster  | [![(Travis image broken)](https://travis-ci.org/quchen/amoeba.png?branch=master)][travis]\ndevelop | [![(Travis image broken)](https://travis-ci.org/quchen/amoeba.png?branch=develop)][travis]\n\n\n\n[travis]: https://travis-ci.org/quchen/amoeba\n[wiki-betweenness]: http://en.wikipedia.org/wiki/Betweenness_centrality\n\n\n\n\n\nExplanation in simple terms\n---------------------------\n\n(This section is for you if you're wondering what the fuzz about the above\ncrumbled up piece of strings and dots is.)\n\nI was always superficially fascinated by complex systems and networks, most\nnotably by what is called *emergence*: the appearance of complex behaviour in\nsystems made up from simple rules. A single ant does not do anything complex,\nand neither do ten of them. Put a thousand together though and you will discover\nthat, although each is still individually doing the same things as before, it\nwill amount to something much bigger than what you would have expected from the\nindividual: complex structures of air tunnels or [fungus farms][leafcutter].\nAnother example is any living organism: even if you understood how every cell\nworked exactly, you still would have no idea about whether (or why) putting them\ntogether in some way can make up the organism that I am right now, typing this\nparagraph.\n\nNetworks also exhibit a lot of emergent properties, and contrary to living\norganisms they are much more suitable to being simulated and applied by\ncomputers. A network in this sense is simply a number of constituents with\nconnections to other constituents. These networks can consist of people (where\nconncetions can be \"who likes who\" or \"have met each other at some point\"),\ncomputers (\"connected over the internet\", \"contains parts made by the same\nmanufacturer\"), languages (\"what words can be used after others\") and many other\nthings.\n\nAmœba is a program that creates a computer network. I came up with the idea\naround the time of the first Bitcoin boom in 2013; the Torrent network did also\nseem somewhat interesting to me. So I thought \"why not implement a basic version\nof something like that yourself?\" - generously estimating 500 lines of code to\nget the core done. Months and thousands of lines of code added/removed/edited\nlater, a satisfying first version is still just barely on the horizon, but it's\nfinished enough to be able to play around with it. The \"crumbled up piece of\nstrings and dots\" above is a snapshot of an Amœba network, a few seconds before\nI terminated half of it to see whether it would survive that without clustering\ninto many disconnected components. Research has begun! :-)\n\n[leafcutter]: https://en.wikipedia.org/wiki/Leaf_cutter_ant\n\n\n\n\n\n\n\nPlanned features\n----------------\n\n- Anonymity: every node only knows about its immediate neighbours. Unless\n  explicitly added, the origin of a signal sent over the network is untraceable.\n\n- The network should be as robust as possible against malicious participants.\n\n- Even large network outages should not lead to disconnected components.\n\n- Neat live graph drawing to have a global view of the entire network (or at\n  least the nodes choosing to be published).\n\nAlso see the [issues list][issues] on GitHub.\n\n[issues]: https://github.com/quchen/amoeba/issues\n\n\n\n\n\nResearch goals\n--------------\n\n- Structure\n\n  - What node degrees are necessary for a robust network?\n\n  - How long, if possible, does it take an almost dead node to heal again?\n\n  - How does the network diameter scale with number of participants? This is\n    important to ensure the shortest paths between arbitrary nodes stay short,\n    ensuring fast message delivery.\n\n  - What are the timescales for bootstrapping the network, adding or removing\n    one node or many nodes?\n\n  - How do two disconnected network components meld together when a connection\n    is introduced?\n\n- Integrity\n\n  - What attacks can be prevented by design?\n\n  - Worst case scenarios: what are malicious participants allowed to do?\n\n\nThese goals are subject to certain constraints:\n\n- A node only knows its immediate neighbours. (No global knowledge hacks.)\n\n- Network dynamics are static: a link is only broken by technical failure, and\n  not as part of the network dynamics.\n\n\n\n\n\nNetwork description\n-------------------\n\n\n![(Picture missing, uh oh)](doc/network_schema.png\n                            \"Network structure of a small system\")\n\nThe picture shows the network structure of a small Amœba network. Blue arrows\nare ordinary connections, while red ones stand for local direct connections,\nused by special network services.\n\n\n\n### Normal nodes\n\n- All nodes run identical programs.\n\n- Each node has a number of upstream and downstream neighbours, which are\n  allowed to send data to the current node, or accept data sent by it,\n  respectively. It has no knowledge about the network other than its neighbours.\n\n- Nodes have a minimum and maximum number of neighbours for both upstream and\n  downstream connections (independent of each other). If there is a deficit of\n  connections, nodes will request new neighbours from the network; if there is a\n  surplus, no new connections will be accepted; if there is neither, no requests\n  will be sent, but incoming requests will be processed.\n\n- If a node has a deficit in connections, it will randomly tell one of its\n  neighbours of it. This is called an *edge request*, and contains its own\n  address, and parameters determining how far the request should travel through\n  the network. The edge request is relayed by receiving nodes a number of times,\n  passing it on to one of their own downstream neighbours, until eventually one\n  of them accepts the request, and establishes the desired connection with the\n  initially issuing node.\n\n  The number of edge requests depending on the deficit is currently a simple\n  square root:\n\n  ![(Picture missing, uh oh)](doc/request_per_deficit_function.png)\n\n- Nodes will attempt to minimize the number of connections above the minimum by\n  *pruning*. They will do so by telling downstream neighbours of their wish to\n  drop the connection, which will be accepted if this can be done without making\n  the partner go below the minimum limit.\n\n- Initial connection is made using a special bootstrap service, see the section\n  below.\n\n- To look at the large scale structure of the network, a special request can be\n  made by a special graph plot server. This request makes every client send a\n  list of all its neighbours to the plot server. (This is strictly a debugging\n  tool, since it opens the door for a truckload of attacks.)\n\n\n\n### Special services\n\nA central point in node design is that they reject signals from unregistered\norigins, so that spamming a single node from outside does not affect the network\nat all.\n\nHowever, this is sometimes too restrictive: for some services, it makes sense to\nbe able to issue signals, despite them not being part of the network. To solve\nthis problem, nodes can be spawned with a special direct communication channel\nthat can be used to send messages to it directly.\n\n\n\n#### Bootstrap server\n\nA bootstrap server is the first contact a node makes after startup, and issues\nedge requests on behalf of its client.\n\n\n\n#### Drawing server\n\nThe drawing server's purpose is creating a map of the network to study its\nstructure. Issues a signal that makes every (willing) node of the network send\nit a list of their downstream neighbours.\n\n\n\n\n\nKnown vulnerabilities and immunities\n------------------------------------\n\nThis is a list of known and feasible attacks on the current design:\n\n- Malicious single nodes\n\n  - DDoS: Spamming the network with loads of trashy messages. Especially\n    flooding signals (such as text messages) have a very large impact compared\n    to the single signal they are issued with. Furthermore messages are\n    anonymous, so if you have N neighbours, you can at least send N messages per\n    malicious node without any possible spam protection being able to jump in.\n\n  - Spamming the bootstrap server with requests yields an arbitrary amount of\n    new edge requests, allowing a node to quickly connect to a large part of the\n    network.\n\n  - The drawing server capabilities are currently available to everyone, and not\n    just legitimate drawing servers. In other words, getting a list of all nodes\n    is trivial.\n\n  - *Resistent*: SlowLoris attacks. Since signal sizes are relatively small,\n    connection timeouts can be short.\n\n  - Node crawling: Although nodes only retain the addresses of downstream\n    neighbours (remember the upstream connection is one-way, clients will not\n    send or handle signals issued the wrong way), edge requests carry valid\n    server addresses and traverse part of the network before they are accepted.\n    Specialized nodes could simply store all valid addresses they encounter.\n    This is completely undetectable by other nodes and, while not dangerous on\n    its own, can lead to knowledge about all of the network's participants. The\n    node database could be shared with malicious nodes that could harness that\n    information for attacks on the network.\n\n  - There is no message size limit at the moment. A node will read incoming data\n    until the connection for a single signal times out.\n\n- Malicious sub-networks\n\n  - Nodes accepting all edge requests they encounter can build up a connection\n    to a large portion of the network. This opens the door for new attacks:\n\n    - Filtering: Certain signals could be thrown away, for example edge requests\n      from a certain IP range so that no new nodes can connect to the network\n\n    - Altering signals\n\n  - A test where two neighbours compare some of their neighbours and drop common\n    ones could battle the possibility of nodes gathering up too many connections\n    illegally.\n\n  - Bootstrap takeover: if an attacker manages to replace all neighbours (both\n    up- and downstream) of a bootstrap server's pool, all new nodes connecting\n    via that server will be relayed into the malicious network.\n\n- Malicious swarms - right now it's trivial to spawn thousands of new nodes\n  simultaneously. No matter how many honest nodes there are, it is very easy to\n  drown them in a network controlled by a few actors behaving like they are\n  many.\n\n- Killing all bootstrap servers makes it impossible to discover the network.\n\n\n\n\n\nDocumentation\n-------------\n\n\n\n### Client structure\n\nThe picture below sketches the flow of information in a single Amœba client.\n\n- Network connections are shown in red. Nodes first connect to another node's\n  server (dashed red), which then relays them to their own private worker in the\n  target node (by spawning a new worker, yellow dashed), at which point the data\n  flows directly from node to worker.\n\n- Workers take input from upstream nodes and formulate a response based on them.\n  This response is then sent over the internal channels (blue) to the clients.\n\n- Clients have persistent connections to downstream neighbours open (red network\n  connections), and send the instructions received from the channels to them.\n\n- The client pool watches internal databases to determine whether there are\n  enough workers and clients. If not, it instructs existing clients to send\n  requests for further neighbours.\n\n![(Picture missing, uh oh)](doc/information_flow.png\n                            \"Flow of information in an Amœba client\")\n\n\n\n### The protocol\n\nThe protocol type used by Amœba can be found in `src/Types/Signal.hs`. All\nsignals are sent downstream, with one exception where relevant data actually\nflows upstream. Unless otherwise noted, the server answers signals with a\n`ServerSignal`, which can basically be `OK` or one of multiple possible errors.\nA usual request consists of a node sending a signal downstream and waiting for\nthe response, terminating the worker if it is not positive.\n\nSignals are divided in two main groups, normal and special. Normal signals are\nwhat usual nodes routinely use:\n\n- `EdgeRequest` contains information for establishing a new edge in the network\n- `KeepAlive` is sent in case there haven't been any useful signals, but the\n  connection should not time out\n- `ShuttingDown` is sent as a courtesy to other nodes, so they can remove a\n  terminating node before the timeout kicks in\n- `Flood` signals are distributed to every node in the network. Current\n  instances are text messages and a request to send a `NeighbourList` to a\n  drawing server.\n- `Prune` is a request to downstream neighbours to drop the connection. This is\n  an effort to keep the number of connections as low as possible, and the\n  request will be accepted if dropping the connection does not cross the minimum\n  threshold.\n\nNormal signals are filtered: only when they're coming from known upstream nodes\nthey are processed. Special signals circumvent this, as some processes\ninherently require unknown nodes to establish connections.\n\n- `BootstrapRequest` is sent to the bootstrap server, and instructs it to send\n  out `EdgeRequest`s on behalf of the contacting node.\n- `Handshake` is what actually establishes a new connection. Sent to a new\n  downstream neighbour, it adds the issuer to its list of known nodes and\n  answers with `OK`; the issuer then does its own bookkeeping, and answers back\n  `OK` as well, finalizing the deal with mutual agreement.\n- `HandshakeRequest`s prompt another node to send back a `Handshake`. This\n  allows `Handshake` to be used to establish incoming connections, not just\n  outgoing ones by sending it directly.\n- `NeighbourList` is sent from nodes to the drawing server. It contains the list\n  of downstream neighbours.\n\n\n\n### Terminology, abbreviations\n\nThese may help reading the source comments:\n\n        Name | Meaning\n-----------: | -----------------------------------------------------------------\n          _* | Accessor functions that don't do any computation otherwise. When dependencies permit, the lenses generated from these are used.\n          *H | Handler. Signals or commands are delegated to these for processing.\n         BSS | Bootstrap server\n         DSN | Downstream node, i.e. a neighbouring node the current sends commands do. (S, T, U in the picture above.)\n         LDC | Local direct connection. Used by the node pool to send signals directly to its nodes instead of taking a detour over the network.\n        ST1C | Server to one (unspecified/arbitrary) client channel\n         STC | Server to client channel\n        STSC | Server to specific client channel\n         USN | Upstream node, i.e. a neighbouring node the current gets commands sent by. (A, B, C in the picture above.)\n\nSometimes, I like to use capital letters at the end of identifiers to tag\nfunctions with a purpose. This is usually local to a single module. If you see\nsuspiciously looking names like `fooX` or `barL`, have a look at the module's\nhead comment.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquchen%2Famoeba","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquchen%2Famoeba","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquchen%2Famoeba/lists"}