Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ssbc/multiserver

A single interface that can work with multiple protocols, and multiple transforms of those protocols (eg, security layer)
https://github.com/ssbc/multiserver

Last synced: about 2 months ago
JSON representation

A single interface that can work with multiple protocols, and multiple transforms of those protocols (eg, security layer)

Awesome Lists containing this project

README 

        
# multiserver



A single interface that can work with multiple protocols,

and multiple transforms of those protocols (eg, security layer)



## motivation



Developing a p2p system is hard. Especially hard is upgrading protocol layers.

The contemporary approach is to [update code via a backdoor](https://whispersystems.org/blog/the-ecosystem-is-moving/),

but as easily as security can be added, [it can be taken away](https://nakamotoinstitute.org/trusted-third-parties/).



Before you can have a protocol, you need a connection between peers.

That connection is over some form of network transport,

probably encrypted with some encryption scheme, possibly

compression or other layers too.



Usually, two peers connect over a standard networking transport

(probably tcp) then they have a negotiation to decide

what the next layer (of encryption, for example) should be.

This allows protocol implementators to roll out improved

versions of the encryption protocol. However, it does

not allow them to upgrade the negotiation protocol!

If a negotiation protocol has a vulnerability it's much

harder to fix, and since the negotiation needs to be unencrypted,

it tends to reveal a lot about program the server is running.

[in my opinion, it's time to try a different way.](https://github.com/ipfs/go-ipfs/pull/34)



Some HTTP APIs provide upgradability in a better, simpler way by

putting a version number within the url. A new version of

the API can then be used without touching the old one at all.



multiserver adapts this approach to lower level protocols.

Instead of negotiating which protocol to use, run multiple

protocols side by side, and consider the protocol part of the address.



Most network systems have some sort of address look up,

there is peer identifier (such it's domain) and then

a system that is queried to map that domain to the lower level

network address (such as it's ip address, retrieved via a DNS (Domain Name System) request)

To connect to a website secured with https, first

you look up the domain via DNS, then connect to the server.

Then start a tls connection to that server, in which

a cyphersuite is negotiated, and a certificate is provided

by the server. (this certifies that the server really

owns that domain)



If it was using multiserver, DNS would respond with a list of cyphersuites,

(encoded as multiserver addresses) and then you'd connect directly to a server and start using the protocol, without negotiation.

p2p systems like scuttlebutt also usually have a lookup,

but usually mapping from a public key to an ip address.

Since a look up is needed anyway, it's a good place

to provide information about the protocol that server speaks!



This enables you to do two things, upgrade and bridging.



### upgrade



If a peer wants to upgrade from *weak* protocol

to a *strong* one, they simply start serving *strong* via another port,

and advertise that in the lookup system.

Now peers that have support for *strong* can connect via that protocol.



Once most peers have upgraded to strong, support for *weak* can be discontinued.



This is just how some services (eg, github) have an API version

in their URL scheme. It is now easy to use two different

versions in parallel. later, they can close down the old API.



``` js

var MultiServer = require('multiserver')

var chloride = require('chloride')

var keys = chloride.crypto_sign_keypair()

var appKey = "dTuPysQsRoyWzmsK6iegSV4U3Qu912vPpkOyx6bPuEk="



function accept_all (id, cb) {

  cb(null, true)

}

var ms = MultiServer([

  [ //net + secret-handshake

    require('multiserver/plugins/net')({port: 3333}),

    require('multiserver/plugins/shs')({

      keys: keys,

      appKey: appKey, //application key

      auth: accept_all

    }),

  ],

  [ //net + secret-handshake2

    //(not implemented yet, but incompatible with shs)

    require('multiserver/plugins/net')({port: 4444}),

    //this protocol doesn't exist yet, but it could.

    require('secret-handshake2')({

      keys: keys,

      appKey: appKey, //application key

      auth: accept_all

    }),

  ]

])



console.log(ms.stringify())



//=> net::3333~shs:;net::4444~shs2:



//run two servers on two ports.

//newer peers can connect directly to 4444 and use shs2.

//this means the protocol can be _completely_ upgraded.

ms.server(function (stream) {

  console.log('connection from', stream.address)

})



//connect to legacy protocol

ms.client('net::3333~shs:', function (err, stream) {

  //...

})



//connect to modern protocol

ms.client('net::4444~shs2:', function (err, stream) {

  //...

})



```



### bridging



By exposing multiple network transports as part of

the same address, you can allow connections from

peers that wouldn't have been able to connect otherwise.



Regular servers can do TCP. Desktop clients can speak TCP,

but can't create TCP servers that other desktop computers can connect to reliably.

Browsers can use WebSockets and WebRTC.

WebRTC gives you p2p, but needs an introducer.

Another option is [utp](https://github.com/mafintosh/utp-native)

- probably the most convenient, because it doesn't need an introducer

on _every connection_ (but it does require some bootstrapping),

but that doesn't work in the browser either.



``` js

var MultiServer = require('multiserver')



var ms = MultiServer([

  require('multiserver/plugins/net')({port: 1234}),

  require('multiserver/plugins/ws')({port: 2345})

])



//start a server (for both protocols!)

//returns function to close the server.

var close = ms.server(function (stream) {

  //handle incoming connection

})



//connect to a protocol. uses whichever

//handler understands the address (in this case, websockets)

var abort = ms.client('ws://localhost:1234', function (err, stream) {

  //...

})



//at any time abort() can be called to cancel the connection attempt.

//if it's called after the connection is established, it will

//abort the stream.

```



## address format



Addresses describe everything needed to connect to a peer.

each address is divided into protocol sections separated by `~`.

Each protocol section is divided itself by `:`. A protocol section

starts with a name for that protocol, and then whatever arguments

that protocol needs. The syntax of the address format is defined by [multiserver-address](https://github.com/ssbc/multiserver-address)



For example, the address for my ssb pubserver is:

```

net:wx.larpa.net:8008~shs:DTNmX+4SjsgZ7xyDh5xxmNtFqa6pWi5Qtw7cE8aR9TQ=

```

That says use the `net` protocol (TCP) to connect to the domain `wx.larpa.net`

on port `8008`, and then encrypt the session using `shs` ([secret-handshake](https://github.com/auditdrivencrypto/secret-handshake))

to the public key `DTNmX+4SjsgZ7xyDh5xxmNtFqa6pWi5Qtw7cE8aR9TQ=`.



Usually, the first section is a network protocol, and the rest are transforms,

such as encryption or compression.



Multiserver makes it easy to use multiple protocols at once. For example,

my pub server _also_ supports `shs` over websockets.



So, this is another way to connect:



```

wss://wx.larpa.net~shs:DTNmX+4SjsgZ7xyDh5xxmNtFqa6pWi5Qtw7cE8aR9TQ=

```



if your server supports multiple protocols, you can concatenate addresses with `;`

and multiserver will connect to the first address it understands.



```

net:wx.larpa.net:8008~shs:DTNmX+4SjsgZ7xyDh5xxmNtFqa6pWi5Qtw7cE8aR9TQ=;wss://wx.larpa.net~shs:DTNmX+4SjsgZ7xyDh5xxmNtFqa6pWi5Qtw7cE8aR9TQ=

```

This means use net, or wss. In some contexts, you might have a peer that understands

websockets but not net (for example a browser), as long as a server speaks at least

one protocol that a peer can understand, then they can communicate.



## scopes



address also have a scope. This relates to where they

can be connected to. Default supported scopes are:



* device - can connect only if on the same device

* local - can connect from same wifi (local network)

* public - can connect from public global internet.



some transport plugins work only on particular scopes.



when `stringify(scope)` is called, it will return

just the accessible addresses in that scope.



## plugins



A multiserver instance is set up by composing a selection

of plugins that construct the networking transports,

and transforms that instance supports.



There are two types of plugins, transports and transforms.



### `net({port,host,scope})`



TCP is a `net:{host}:{port}` port is not optional.



``` js

var Net = require('multiserver/plugins/net')`

Net({port: 8889, host: 'mydomain.com'}).stringify()  => 'net:mydomain.com:8889'

Net({port: 8889, host: 'fe80::1065:74a4:4016:6266:4849'}).stringify()  => 'net:fe80::1065:74a4:4016:6266:4849:8889'

Net({port: 8889, host: 'fe80::1065:74a4:4016:6266:4849', scope: 'device'}).stringify()  => 'net:fe80::1065:74a4:4016:6266:4849:8889'

```



### `WebSockets({host,port,scope,handler?,key?,cert?})`



create a websocket server. Since websockets are

just a special mode of http, this also creates a http

server. If `opts.handler` is provided, requests

to the http server can be handled, this is optional.



WebSockets `ws://{host}:{port}?` port defaults to 80 if not provided.



WebSockets over https is `wss://{host}:{port}?` where port is

443 if not provided.



If `opts.key` and `opts.cert` are provided as paths, a https server

will be spawned.



``` js

var WebSockets = require('multiserver/plugins/ws`)



var ws = WebSockets({

  port: 1234,

  host: 'mydomain.com',

  handler: function (req, res) {

    res.end('
hello
')

  },

  scope:...

})



ws.stringify() => 'ws://mydomain.com:1234'

```



### `Onion()`



Connect over tor using local proxy to dæmon (9050) or tor browser (9150). 

Both will be tried to find a suitable tor instance. 

The tor ports are unconfigurable. The standard

tor ports are always used.



This plugin does not support creating a server.

You should use tor's configuration files to send incoming connections to a `net`

instance as a hidden service.



An accepted onion address looks like: `onion:{host}:{port}`

port is not optional. This plugin does not return

an address, so you must construct this address manually.



``` js

var Onion = require('multiserver/plugins/onion`)



var onion = WebSockets({

  //no config is needed except scope, but you

  //surely will use this with "public" which is the default

  //scope:'public'

})



ws.stringify() => null

```



### `Bluetooth({bluetoothManager})`



The [multiserver-bluetooth](https://github.com/Happy0/multiserver-bluetooth) module implements a multiserver protocol for to communicate over Bluetooth Serial port.



### `reactnative = require('multiserver-rn-channel')`



The [multiserver-rn-channel](http://npm.im/multiserver-rn-channel) module implementes

a multiserver protocol for use inbetween the reactnative nodejs process and browser process.



### `SHS({keys,timeout?,appKey,auth})`



Secret-handshake is `shs:{public_key}:{seed}?`. `seed` is used to create

a one-time shared private key, that may enable a special access.

For example, you'll see that ssb invite codes have shs with two sections

following. Normally, only a single argument (the remote public key) is necessary.



``` js

var SHS = require('multiserver/plugins/shs')



var shs = SHS({

  keys: keys,

  timeout: //set handshake timeout, if unset falls through to secret-handshake default

  appKey: //sets an appkey

  auth: function (id, cb) {

    if(isNotAuthorized(id))

      cb(new Error())

    else

      cb(null, authenticationDetails)

  }

})

shs.stringify() => 'shs:{keys.publicKey.toString('base64')}

```



note, if the `auth` function calls back a truthy value,

it is considered authenticated. The value called back

may be an object that represents details of the authentication.

when a successful connection goes through `shs` plugin,

the stream will have an `auth` property, which is the value called back from `auth`,

and a `remote` property (the id of remote key).



### `Noauth({keys})`



This authenticates any connection without any encryption.

This should only be used on `device` scoped connections,

such as if net is bound strictly to localhost,

or a unix-socket. Do not use with ws or net bound to public addresses.



``` js

var Noauth = require('multiserver/plugins/noauth')



var noauth = Noauth({

  keys: keys

})

shs.stringify() => 'shs:{keys.publicKey.toString('base64')}



```



streams passing through this will look like an authenticated shs connection.



### `Unix = require('multiserver/plugins/unix-socket')`



network transport is unix socket. to connect to this

you must have access to the same file system as the server.



``` js

var Unix = require('multiserver/plugins/unix-socket')



var unix = Unix({

  path: where_to_put_socket,

  scope: ... //defaults to device

})



unix.stringify() => "unix:{where_to_put_socket}"

```



### createMultiServer([[transport,transforms...],...])



A server that runs multiple protocols on different ports can simply join them

with `;` and clients should connect to their preferred protocol.

clients may try multiple protocols on the same server before giving up,

but generally it's unlikely that protocols should not fail independently

(unless there is a bug in one protocol).



an example of a valid multiprotocol:

`net:{host}:{port}~shs:{key};ws:{host}:{port}~shs:{key}`



``` js

var MultiServer = require('multiserver')



var ms = MultiServer([

  [net, shs],

  [ws, shs],

  [unix, noauth]

])



ms.stringify('public') => "net:mydomain.com:8889~shs:;ws://mydomain.com:1234~shs:"

ms.stringify('device') => "unix:{where_to_put_socket}"



ms.server(function (stream) {

  //now that all the plugins are combined,

  //ready to use as an actual server.

})

```



## interfaces



To construct a useful multiserver instance,

one or more transport is each connected with zero

or more transforms. The combine function is

the default export from the `multiserver` module.



``` js

var MultiServer = require('multiserver')



var ms = MultiServer([

  [transport1, transform1],

  [transport2, transform2, transform3],

])



var close = ms.server(function (stream) {

  //called when a stream connects

}, onError, onListening)

```



```

createMultiServer([[Transform, Transports*,...]], *]) => MultiServer

```



a MultiServer has the same interface as a Transport,

but using a combined multiserver instance as a transport

is **not** supported.



## createTransport(Options) => Transport



The transport exposes a name and the ability to

create and connect to servers running that transport.



``` js

Transport => {

  // that describes the sub protocols

  name,

  // connect to server with address addr.

  client (addr, cb),

  // start the server

  server (onConnect, onError, onListening),

  // return string describing how to connect to the server, aka, "the address"

  // the address applies to a `scope`.

  stringify(scope),

  // parse the addr,

  // normally this would probably return the

  // Options used to create the transport.

  parse(string) => Options

}

```



## createTransform(options) => Transform



``` js

Transform => {

  name: string,

  create(Options) => (stream, cb(null, transformed_stream)),

  parse (str) => Options,

  stringify() => string,

}

```



note the create method on a Transform takes Options,

and returns a function that takes a stream and a callback,

and then calls back the transformed stream.

In all cases the stream is a [duplex stream](https://github.com/pull-stream/pull-stream)



## License



MIT