{"id":13806209,"url":"https://github.com/cagerton/dropthat","last_synced_at":"2025-05-13T21:32:44.461Z","repository":{"id":12022474,"uuid":"14606024","full_name":"cagerton/dropthat","owner":"cagerton","description":"Client side encryption can be part of this complete breakfast. Also Lua.","archived":false,"fork":false,"pushed_at":"2014-02-19T07:58:18.000Z","size":318,"stargazers_count":64,"open_issues_count":0,"forks_count":6,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-05-19T05:39:48.648Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"muhku/FreeStreamer","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cagerton.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-22T01:54:55.000Z","updated_at":"2023-01-14T20:38:11.000Z","dependencies_parsed_at":"2022-09-06T23:10:36.337Z","dependency_job_id":null,"html_url":"https://github.com/cagerton/dropthat","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cagerton%2Fdropthat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cagerton%2Fdropthat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cagerton%2Fdropthat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cagerton%2Fdropthat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cagerton","download_url":"https://codeload.github.com/cagerton/dropthat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":213078831,"owners_count":15534483,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T01:01:08.907Z","updated_at":"2024-08-04T01:05:54.122Z","avatar_url":"https://github.com/cagerton.png","language":"JavaScript","readme":"Building an OpenResty events server\n===================================\n\n[DropTh.at](https://dropth.at/) is a chat service demo built with OpenResty. It secures your messages using a combination of traditional https and clientside crypto. Chat rooms are randomly generated by the client and can be shared by sharing the room's full url. The encryption keys are stored in the #fragment-identifier and are not sent to the server. That means the url is the room password. The core server and client demo are [open source](https://github.com/cagerton/dropthat).\n\nReality check: It still relies on a trustworthy host and on SSL to deliver the initial html+js. It is just a is a demo. See [Below](#on-javascript-cryptography).\n\n### Context: Javascript EventSource API:\n\nI'm using the [EventSource API](http://www.w3.org/TR/eventsource/) for delivering messages.  It is [supported](http://caniuse.com/#feat=eventsource) by all the browsers that I currently care about (and can be shimmed into some by Microsoft).  The protocol is just a chunked HTTP stream with content-type \"text/event-source\" and formatted event messages.  It has a very low overhead and it plays nicely with the pub-sub pattern.\n\nHere are the headers and the first event in a stream (as rendered by curl):\n```http\n\u003c HTTP/1.1 200 OK\n\u003c Content-Type: text/event-stream\n\u003c Server: dropth.at\n\u003c Connection: keep-alive\n\u003c Transfer-Encoding: chunked\n\u003c\nevent:count\ndata:count=2\n\n```\nAnd here's how you could capture it:\n```javascript\nvar events = new EventSource( '\u003cevent-source-url\u003e' );\nevents.addEventListener('count', function(c){ console.log(\"Got \", c); });\n```\n\u003eGot count=2\n\n### Nginx, Lua, and OpenResty:\n\nThe server is built with [OpenResty](http://openresty.org/); that's Nginx, Lua, and a few extra modules. Leafo has a [nice intro](http://leafo.net/posts/creating_an_image_server.html) to building with OpenResty if you're not familiar. I highly suggest you check it out.\n\nI'll assume you understand the basics of Nginx and Lua from this point.  Here are my core pub/sub locations in the [nginx config](https://github.com/cagerton/dropthat/blob/master/chat.conf).\n```nginx\nworker_processes 1;\nhttp {\n    init_by_lua 'chat = require \"chat\"';\n\n    server {\n        listen 80 so_keepalive=20s:3s:6;\n        set_by_lua $dontcare 'chat.check_init()';\n\n        location ~ \"^/sub/(?P\u003cchannel\u003e[a-zA-Z\\d_-]+)$\" {\n            lua_socket_log_errors off;\n            lua_check_client_abort on;\n            content_by_lua 'chat.event_source_location(ngx.var.channel)';\n        }\n\n        location ~ \"^/pub/(?P\u003cchannel\u003e[a-zA-Z\\d_-]+)$\" {\n            client_max_body_size 32k;\n            content_by_lua '\n                ngx.req.read_body()\n                local message = ngx.escape_uri(ngx.var.request_body)\n                chat.publish_event(ngx.var.channel, nil, nil, message)\n            ';\n        }\n```\n* I've loaded the chat module as a thread global (happens once during while loading the config). \n* I'm start the a light background thread by calling chat.check_init()\n* The /sub/\u0026lt;channel-id\u003e location calls chat.event_source_location() function.\n* Likewise, the pub location really just hits chat.publish_event() function.\n\nNow that the special parts of the Nginx config file are out of the way, we'll jump in to [chat.lua](https://github.com/cagerton/dropthat/blob/master/chat.lua). The first thing we need is a way to format the event stream.\n```lua\nlocal http_header = \"HTTP/1.1 200 OK\\r\\n\"..\n                    \"Content-Type: text/event-stream\\r\\n\"..\n                    \"Server: dropth.at\\r\\n\"..\n                    \"Connection: keep-alive\\r\\n\"..\n                    \"Transfer-Encoding: chunked\\r\\n\\r\\n\"\n\nlocal function format_http_chunk(message)\n    -- http chunk format is hex-encoded length, newline, data, newline\n    local len = string.format(\"%x\\r\\n\", string.len(message))\n    return len..message..\"\\r\\n\"\nend\n\nlocal function format_event(id, event, message)\n    local buff = \"data:\"..message..\"\\r\\n\\r\\n\"\n    if event then\n        buff = \"event:\"..event..\"\\r\\n\"..buff\n    end\n    if id then\n        buff = \"id:\"..id..\"\\r\\n\"..buff\n    end\n    return format_http_chunk(buff)\nend\n```\nWell, that's pretty simple.  Next we'll handle the incoming subscriber connections.\n\n```lua\nfunction event_source_location(channel_id)\n    local sock, err = ngx.req.socket(true) -- hijack the request socket\n    sock:send(http_header)\n\n    local function cleanup()\n        remove_socket(channel_id, sock)\n        ngx.exit(499)\n    end\n    local ok, err = ngx.on_abort(cleanup)  -- handle disconnect\n\n    add_socket(channel_id, sock)           -- add socket to data structure\n    local loops=0\n    while 1 do\n        ngx.sleep(19.31)\n        send_blank(sock)     -- periodic tick message :\\r\\n\n    end\nend\n\nlocal function add_socket(channel_id, socket)\n    local channel = get_or_create_channel(channel_id)\n    channel.sockets[socket] = ngx.now()\n    update_channel(channel_id)\nend\n\nlocal function update_channel(channel_id)\n    local channel = channels[channel_id]\n    if not channel.announce_queued then\n        push_update_queue(channel_id)\n        channel.announce_queued = true\n    end\nend\n```\nFor reference, [ngx.sleep](https://github.com/chaoslawful/lua-nginx-module#ngxsleep) is nonblocking. In this case, the lua \"light thread\" handling this request wakes up every 19ish seconds just send a simple no-op message to keep the connection alive.\n\nWhen a web client posts a message, our Nginx config hands off to here:\n```lua\nfunction publish_event(channel_id, event_id, event, message)\n    local channel = channels[channel_id]\n    if channel then\n        local chunk = format_event(event_id, event, message) --chunk \u0026 event-stream format\n        for sock,start_time in pairs(channel.sockets) do\n            local bytes, err = sock:send(chunk)\n            if bytes ~= string.len(chunk) then\n                -- handle error here.\n                ngx.log(ngx.ERR, \"failed to write event\", err)\n            end\n        end\n    end\nend\n```\nThe server sends count events when the number of subscibers for a channel changes. I found out the hard way that these need to be rate limited while load testing with 20k clients on a virtual machine. Here's how the notification thread works:\n```lua\nfunction check_init()\n    ngx.timer.at(0, notify_thread) -- ngx.timer.at creates a new light-thread\n    check_init = function() end -- we only need that to happen once\nend\n\nlocal function notify_thread(premature)\n    local loops = 0\n    while true do\n        local channel_id = pop_update_queue() -- just has channel_ids\n        if channel_id then\n            local now = ngx.now();\n            local channel = channels[channel_id]\n            if channel and now \u003c 0.373 + channel.last_announce then\n                ngx.sleep(0.373) -- oversleeps a tiny bit,\n                                 -- but queue order means it's ok.\n            end\n            channel = channels[channel_id]\n            if channel then\n                channel.announce_queued = false\n                publish_channel_count(channel_id)  -- just counts clients \u0026 sends event\n            end\n        else\n            ngx.sleep(.127)\n        end\n    end\nend\n```\nPretty cool, huh? Try it out. Here's a [Dockerfile](https://github.com/cagerton/dropthat/blob/master/Dockerfile).\n\nI test this on a local VM.\n```bash\n$ curl localhost:8080/sub/0000000000000000000000\nevent:count\ndata:count=1\n\nevent:count\ndata:count=4663\n\nevent:count\ndata:count=15001\n\nevent:count\ndata:count=13977\n\nevent:count\ndata:count=13465\n\nevent:count\ndata:count=1\n\n```\nthe other tab:\n```bash\ncda@ubuntu:~$ sudo bash -c 'ulimit -n 16384 \u0026\u0026 ab -n 15000 -c 15000 localhost:8080/sub/0000000000000000000000'\nThis is ApacheBench, Version 2.3 \u003c$Revision: 655654 $\u003e\nCopyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/\nLicensed to The Apache Software Foundation, http://www.apache.org/\n\nBenchmarking localhost (be patient)\n^C\n\nServer Software:        dropth.at\nServer Hostname:        localhost\nServer Port:            8080\n\nDocument Path:          /sub/0000000000000000000000\nDocument Length:        0 bytes\n\nConcurrency Level:      15000\nTime taken for tests:   6.447 seconds\nComplete requests:      0\nFailed requests:        0\nWrite errors:           0\nTotal transferred:      2607156 bytes\nHTML transferred:       762156 bytes\ncda@ubuntu:~$ \n```\n\n\n\nOn Javascript Cryptography\n==========================\n\n### Preface\n\nJavascript Cryptography isn't [inherently doomed (Matasano)](http://www.matasano.com/articles/javascript-cryptography/), it's just useful for different types of problems. (JS Crypto can be an important part of this complete breakfast!)\n\n### Intro\n\nMy usage of encryption on the DropTh.at demo is very simple. The interesting crypto is handled by [SJCL](http://bitwiseshiftleft.github.io/sjcl/), the keys are stored in the fragment-identifier of the url (that part that doesn't get sent with the request; handle that url like a password), and the room channel ids are derived from a one way SHA hash of the room keys. Messages and images are encrypted using the same default settings in SJCL. It handles building random IVs, ensuring availability of sufficent entropy and encrypting with AES-CCM mode.  Remember that none of this is secure unless you can safely get the javascript and html (over SSL) in the first place.  There's lots of room for improvement here.\n\nHere are some of the things that JS crypto can add to traditional HTTPS.\n\n### Client Crypto is an implicit contract:\n\nWhen web services let clients encrypt their data without giving up their keys, it gives the clients an expectation of pricavy and forms an implicit contract like this:\n* We won't casually read your stuff on our severs.\n* We won't target ads based on your content.\n* We won't leak information by [de-duplicating](http://paranoia.dubfire.net/2011/04/how-dropbox-sacrifices-user-privacy-for.html) your data.\n* You won't delete your private media because of a bogus DMCA takedown on another user with the same files.\n* Our backups will never contain your plaintext.\n* Our cross-data-center links will never expose your unencrypted data.\n* We won't report your private data [to the police](http://sacramento.cbslocal.com/2013/11/21/googles-role-in-woodland-child-pornography-arrest-raises-privacy-concerns/)(Okay this time, but how about leaked NSA docs?)\n\nNormal cloud companies could get away with any of those things, but a client-encrypted service would lose all credibility over the first violation.\n\n### As a legal protection for the host from the client data:\n\nLets take [Mega](https://mega.co.nz/) as an example (I haven't verified this completely): they use client encryption and never see media files that are uploaded to them.  Since they're insulated from the data, it'll be hard to argue that they are complicity in copyright violations.\n\n### Untrusted Public CDNs:\n\nWe like Public CDNs (bootstrapcdn, cdnjs, googlecdn, etc), but we shouldn't have to trust them.  I'm sure they've all got great security, but an attack on a big public cdn could have wide reaching implications.  You can use Javascript to verify assets (both shared and private) before using/running them. Proof of concept: https://dropth.at/cors-cdn-demo\n\n### Password Hashing:\n\nYou all use BCrypt/SCrypt/PBKDF2, right?  These take up lots of CPU time by design and can be targeted for a DoS attack on your server.  Rate limiting can help, but it's not a silver bullet - especially if you're chewing through 100ms+ of cpu time for each attempt.  They also make RasPi servers cry.  With the careful application of client-side js crypto, you can offload some of the expensive work from your server to the client during heavy load (or all the time).  I'd reccomend wrapping the result again with a SHA and random salt on the server. [Previously](https://gist.github.com/cagerton/5485241#file-1crazy-md)\n\nPBKDF2 Sha256 with 25000 iterations:\n```javascript\nvar b64 = sjcl.codec.base64, sha2 = sjcl.hash.sha256, kdf = sjcl.misc.pbkdf2;\nfunction preHash(site, username, password){\n    var salt = b64.fromBits(sha2.hash(site + username)).slice(0,10);\n    return = b64.fromBits(kdf(password, salt, 25000));\n}\nvar username='cda', password='password', start = Date.now(),\n    output = preHash('dropth.at', username, password);\nconsole.log(\"Hashing took approx \",Date.now()-start, \"ms\");\n```\n\u003e Chrome: 168ms, Firefox: 150ms, Safari: 1260ms, Chrome on my Galaxy Nexus: 1343ms, Android browser: 1628ms.\n\n\u003e django.utils.crypto.pbkdf2: 105ms on my laptop.\n\n\nTODO\n============\n* iPhone canvas problem? Meh. Don't have one to test with; works on iPad.\n* Needs responsive layout. Sucks on a small screen.\n* Add disclaimer about old and/or shitty browsers.\n* Some misc fixups. Maybe CSRF (less important since rooms are secret) + asset domains?\n\n### Some ways you'll get boned:\n* Someone will try to run this without https and you'll get boned.\n* Someone will intercept the key when you send the room link...\n* You'll share the link#key with the wrong person...\n* Someone will dig through your history and get your key...\n* Someone will pwn my server or find an xss hold and inject js to steal the key....\n* Someone will get a \"trusted\" CA to issue a cert then mitm you. Then you'll get boned.\n* http://xkcd.com/538/ (Could be me + a court order to inject js to steal keys).\n","funding_links":[],"categories":["OpenResty"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcagerton%2Fdropthat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcagerton%2Fdropthat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcagerton%2Fdropthat/lists"}