Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jedi4ever/nodejs-ops
my notes on operational things in nodejs
https://github.com/jedi4ever/nodejs-ops
Last synced: about 1 month ago
JSON representation
my notes on operational things in nodejs
- Host: GitHub
- URL: https://github.com/jedi4ever/nodejs-ops
- Owner: jedi4ever
- Created: 2013-08-07T14:42:20.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2013-08-07T16:04:53.000Z (over 11 years ago)
- Last Synced: 2023-04-10T14:22:18.248Z (over 1 year ago)
- Size: 117 KB
- Stars: 28
- Watchers: 4
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# About this presentation
- Started looking at nodejs in anger about 1,5 month ago.
- Compiled a list of topics outside the usual (dev focused) tutorials
- Not all of this is battletested in production
- Beginners, there should be enough for you
- Advanced, Gurus , please chime in, I'd love your feedback!
# Things you should avoid (period)
## avoid typos, missing commas, etc..
- use jshint - in combination with grunt
## avoid using global space: (use var where you can)
- use ```var a = 'bla'``` ;
## avoid using eval, with, switch: (without defaults)
- use ```'use strict';``` or : ```$ node --use_strict```
## don't run npm as root:
- npm pre-install in package.json can be evil
## don't run as root:
- use process.setuid, gid -
- or sudo in your start script
- or use authbind or similar for non-priviledged users
http.createServer(app).listen(app.get('port'), function(){
console.log("Express server listening on port " + app.get('port'));
process.setgid(config.gid);
process.setuid(config.uid);
});
# Testing
- mocha, vows, expect, ...
- ``mocha watch``
- [sinon for spies, mocks, stubs](http://sinonjs.org/docs/)
- [nock for HTTP testing](https://github.com/flatiron/nock)
- [zombie for testing](http://zombie.labnotes.org/API)
- phantomjs
- passport-stub
- for socketio in async tests (force new connection)
[use a grunt setup - example](https://github.com/jedi4ever/socialapp/blob/master/Gruntfile.js)
# Versioning & Packaging
- use [semver](http://semver.org) versioning in package.json
## Dependencies in package.json
- dependencies , dev-dependencies
- optionalDependencies
- [shrinkwrap (freeze your dependencies)](http://blog.nodejs.org/2012/02/27/managing-node-js-dependencies-with-shrinkwrap/)
- [peerDependencies: use for plugins](http://blog.nodejs.org/2013/02/07/peer-dependencies/)
## Packaging
- ```npm pack```
- [bashpack](https://github.com/jedi4ever/bashpack) (turns a nodejs process in bash script)
# Error handling
## Try/catch , but ...
- Try/catch for errors, but this will not catch async errors
## Return an Error object and not a string
- The fundamental benefit of Error objects is that they automatically keep track of where they were built and originated
- ```callback(new Error('my own error'))```
-
## Better stacktraces
- [better-stack-traces](https://github.com/mjpizz/better-stack-traces)
- [LongJohn (eventemitters)](http://www.mattinsler.com/post/26396305882/announcing-longjohn-long-stack-traces-for-node-js)
- [Longer stacks in chai](https://npmjs.org/package/chai-stack)
-
-
Note that I recommend using ```function Foo() { ... }``` for constructors instead of ```var Foo = function() { ... }```
// Constructor
function Foo(bar) {
// always initialize all instance properties
this.bar = bar;
this.baz = 'baz'; // default value
}
// class methods
Foo.prototype.fooBar = function() {
};
// export the class
module.exports = Foo;
## Listen for ALL errors
- ```emit('error')``` makes the process exit if no listener:
- listen to http, redis connections , express, socketIO, log handler, metrics handler
- return a callback on error (as otherwise it will continue)
## Use domains to contain errors
- But imagine , 1 url gives an eror, it will actually exit the complete server! Lots of CPU power waisted for a single error
- Therefore the concept of domains were introduced
-
- with it you can create per 'domain' exceptions that will emit an on('error') to handle them
- Domains & express:
- But it seems connect [grabs express exception with a try catch in it's code](https://github.com/senchalabs/connect/blob/master/lib/proto.js#L160-L195)
- Stack overflow explanation:
- Domains & Connect:
- [Example on how to use connect-domain](http://masashi-k.blogspot.be/2012/12/express3-global-error-handling-domain.html)
The fundamental benefit of Error objects is that they automatically keep track of where they were built and originated
## process.uncaughtException
- it's very common in the nodejs world [to just exit on the exception](http://dshaw.github.io/2012-05-jsday/#/10)
**Not sure if this still makes sense with domains**
## return on a callback
function(err,callback) {
if (err) { return callback('err'); }
console.log('this will be printed too')
};
but: in loops or others ``returns`` means something else
## callback(err) vs emit('error')
**Not sure yet here**, both? only if callback, only if error listener?
-
-
# Logging
I like
- Has the logger.log, logger.info, logger.debug etc.
- Can log metadata, or json objects
- Can use multiple ``tags`` :
- Create as a singleton, require and reuse in other modules (module cache will reuse same object)
var logger = function (options) {
// If we have already been initialized
if (sharedLogger) {
return sharedLogger;
}
```var logger = require('../utils/logger')().loggers.get('express'); ```
## Multiple Outputs
- Console, File logging, rotation
- Also has a logstash output:
- TCP:
- UDP:
## Use in express
// enable web server logging; pipe those log messages through winston
// http://stackoverflow.com/questions/9141358/how-do-i-output-connect-expresss-logger-output-to-winston
var winstonStream = {
write: function(message, encoding){
logger.info(message.slice(0,-1)); //remove newline
}
};
expressApp.use(express.logger({stream: winstonStream}));
## Use in socketio
``` var ioServer = socketIO.listen(webServer, { logger: logger , log:true});```
# Clustering
# Keep it up
- Check if up -
- Use of upstart/ forever - just kidding
- [Stackoverflow discussion Domains or forever](http://stackoverflow.com/questions/14611749/forever-or-domain-which-of-them-is-better-for-node-js-continuous-work#comment20409657_14614261)
## The basics
Have multiple nodejs processes listen on the same socket.
The trick? pass the socket/file descriptor from a parent process and have the server.listen reuse that descriptor.
So **multiprocess** in their own memory space (but with ENV shared usually)
It does not balance, it leaves it to the kernel.
In the last nodejs > 0.8 there is a cluster module (functional although marked experimental)
-
- Simple cluster example:
- Simple cluster example + domains:
- Isaacs gist that was used as inspiration for the cluster doc -
Note: not yet found a 100% reason to favor multi process/cluster nodejs over nginx/haproxy stuff:
-
- Also see this [actionhero related blogpost on elegant downtime in relation to sockets , websockets etc..](http://blog.evantahler.com/blog/production-deployment-with-node-js-clusters.html)
- The rest off this post is to do clustering etc... yourself, otherwise you might want to check [actionhero.js](http://actionherojs.com/)
## Current Cluster/Domain Enhancements:
The core included module is basic, it tells you to take care of all the stuff you need in zerodowntime environments.
Most of the tools below allow you to:
- wait for workers to correctly close or sigKill if past timeout
- sigUSR2 to reload workers one by one
- some provide a cli to do the work using a socket/network connection
- kill a worker that has become unresponsive by waiting for a heartbeat
- put itself offline and not accepting any new requests
### Recluster:
- Code at:
- [Featured in this 10 steps to node Awesomeness](http://qzaidi.github.io/2013/05/14/node-in-production/)
- Good: Simple and in use
- Bad: No CLI , No domains, Cannot pass args
### Cluster2:
- Framework in use at ebay:
- Good: complete with monitoring, control URL
- Bad: Seems to be massive ...
### Naught:
Note: naught2 was a temporary fork, but it got merged into master again
talks about Zero Downtime Crashed by intelligently handling express errors with domains
-
- Good: simple, uses domains
- Bad: seems to do it's own logging
### cluster-master :
- Build by nodejs god @isaacs
- To be investigated
### Up:
- Seems to be the new learnbooost way for zero-downtime
-
### Others to check:
- All Cluster npm modules:
- Bowl:
- Herd:
- Jumpstarter:
- Multi Cluster:
- Pluribus:
- Simple node cluster:
## Older solutions:
(+2 years no updated & probably not nodejs > 0.8 compliant)
So you can safely ignore these, but they can give inspiration
- InfoQ blogpost on multi-core nodejs (is from 2010) -
- **The inspiration:** has some cool options [but _alas < 0.8_ compliant](https://github.com/LearnBoost/cluster/issues/168)
-
- +
[good writeup](http://www.sitepen.com/blog/2010/07/14/multi-node-concurrent-nodejs-http-server/)
-
## Worth mentioning:
-
# Profiling
**to be investigated**
-
-
## Profiling tools
-
-
-
-
- Callgrind -
-
-
-
-
-
## Connect
- Connect-profiler -
## Dtrace
-
-
-
-
-
-
# Memory
-
-
- Node-memwatch
-
-
-
# 'Known' Limits/Tuning
- ulimit Filedescriptors
- Nagle algorithm
- # ports
- eventemittors/listeners
- Set listen backlog, max agents and max open files limit. -
- limit posts
- throttle
# Metrics
Use statsd backend to send counters, timers etc..
- [sivy/node-statsd](https://github.com/sivy/node-statsd/) is feature complete and seems to be the most popular
- [dscape/lynx](https://github.com/dscape/lynx) has streams + (some wierd random/sampling stuff)
- [msiebuhr/node-statsd-client](https://github.com/msiebuhr/node-statsd-client) has express helpers & multi child options
- [fasterize/node-statsd-profiler](https://github.com/fasterize/node-statsd-profiler) has transformation function
- [godmodelabs/statistik](https://github.com/godmodelabs/statistik) has a CLI interface
##
- reuse UDP connection: yes
- prefix: yes
- suffix: yes
- dnscache: yes
- mock: yes
- samplerate: yes
- errors: yes (eventemitter & bubbleup)
- timing: yes
- count: yes
- increment: yes
- decrement: yes
- gauge: yes
- set: yes
- batch: yes
- callback: yes
##
- reuse UDP connection: yes (+ provide your own)
- prefix: yes (called scope)
- error: provide an error function
- network : USE of ephemeral sockets!
- samplerate: yes (use special random)
- batching yes:
- increment: yes
- decrement: yes
- timing: yes
- gauge: yes
- set: yes
__special__:
- uses as stream in/out: yes (uses parser)
##
- reuse UDP connections: yes (ephemeral socket)
- prefix: yes
- count: yes
- gauges: yes
- increment: yes
- decrement: yes
- sets: yes
- timings: yes (delays)
__special__
- socketTimeout: yes
- children (multi prefix)
- express helper: yes (or per URL)
##
- **reuse UDP: no**
- callback: yes (but default prints to console)
- samplerate: yes
- **prefix: no**
- suffix: yes
- batch: yes
- count: yes
- gauges: yes
- increment: yes
- decrement: yes
- sets: yes (called modify)
- timings: (delays)
##
(fork from node-statsd)
- samplerate: yes
- timing: yes
- count: yes
- increment: yes
- decrement: yes
- gauge: yes
- set: ??
- timingstart: yes
- timignend: yes
__special__:
- introduces: key aliases
- transformKey function: YES
##
Note: you can safely ignore this lib
- **reuse UDP connection: no**
- timing: yes
- count: yes
- increment: yes
- decrement: yes
- gauge: yes
- **errors: total ignore**
- **prefix: no**
- **samplerate: no**
- **callback: no**
##
Note: use node-statsd instead unless cli is something special for you. The feature set is smaller than node-statsd and no special features, so we'll ignore this
- udp connection reuse: no
- timing: yes
- counter: yes
- increment: yes
- decrement: yes
- gauge: yes
- rawSend: yes
- samplerate: yes
- **batch: no**
## Mixing in express & connect:
-
Puts your responsetimes & status in statsd
- Although the fork of @sansmischevia seems to be more advanced https://github.com/fetep/connect-logger-statsd/network
Has ignore list, sends full path if needed,
-
This focuses on writeHead, it will calculate the timeelapsed before sending back to the client
## Others:
Note: I've not included any specific backends here, we're focusing on generic statsd usage
- (logger -> statsd)
- (streaming parser)
-
- (dashboard)
## What is a set in statsd?
Sets are acting like simple counters, with the additional specificity that it ignores duplicate values.
Technically, all values are stored in a set, and the number of elements in the set is sent to graphite during flushes. Sets are also emptied during flushes (in the same way that counters are reset to 0).
We have been using it in production for a while now, and it is working as expected. The use case that was used during the development of that feature was the following (it has been since extended to other cases as well):
We want to graph the number of active logged in users on the website.
Maintaining that state across application servers to manually update gauges is non-trivial.
We send a message to statsd containing the id of the user making a request.
# Continous Integration
- Jenkins, Circle CI, TravisCI
- Chat bot in Campfire
# Continous delivery
## Configuration Mgmtm
- redis
- chef nodejs
### Fleet: Extending the easy Git -> deploy
- Initial blogpost on fleet:
- Fleet - uses drones & propagit:
- Propagit: A cascading git deployment:
- blogpost on fleet usage:
- Flotilla:
- All nodejs fleet modules:
-
-
-
-
-
Not related, but also cool: EC2-fleet
# Authz/Authn
- passport.js
- csrf in express
- helmet in express -
- use bcrypt password -
- link sessions socketio/express -
# Loadbalancer/SSL Termination
-
- proxy in express
- in socketIO proxy
- HonorCiphers
- SSL offload via HAProxy 1.5dev (also websockets)
brew install haproxy --devel
- remove header express version
- Oauth
- CA options is an array
- SSL correct settings
- Perfect secrecy
- ```process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';```
- offload your SSL
- CA param is an array (add provider Certs)
- strictCipher, SSL attacks
- perfect secrecy
- SSL insecure!
- express proxy setting (X-...
- in socket.io (authentication Secure..) , Proxy
// req.ip , req.ips , req.protocol (http,https)
if (settings.terminated) {
expressApp.enable('trust proxy');
}
# Security stuff
## Various
- input/output checker
console.log(with beep char);
JSON.parse
- Preinstall in npm
- The request size is also not limited by Node.js which means that a large POST request can be sent to fill the whole memory.
Mostly NPM is ran with root privileges.
Use objectProperties that cannot be changed
String sanitizer -
Fusker - fight back -
## Cookie/Sessions
- secure, http-only, signed cookies
- csrf attack connect.csrf
- helmet other security headers
- RedisStore backed
- connect-sessionIO
- redisstore with hiredis (reuse connection)
#CLI stuff
- commander
- shell.js
- ssh2