Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/treasure-data/serverengine
A framework to implement robust multiprocess servers like Unicorn
https://github.com/treasure-data/serverengine
Last synced: about 1 month ago
JSON representation
A framework to implement robust multiprocess servers like Unicorn
- Host: GitHub
- URL: https://github.com/treasure-data/serverengine
- Owner: treasure-data
- License: apache-2.0
- Created: 2013-06-01T22:26:47.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2024-10-22T02:25:19.000Z (2 months ago)
- Last Synced: 2024-10-24T09:21:46.053Z (2 months ago)
- Language: Ruby
- Size: 371 KB
- Stars: 759
- Watchers: 106
- Forks: 86
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Changelog: Changelog
- License: LICENSE
Awesome Lists containing this project
README
# ServerEngine
ServerEngine is a framework to implement robust multiprocess servers like Unicorn.
**Main features:**
```
Heartbeat via pipe
& auto-restart
/ \ ---+
+------------+ / +----------+ \ +--------+ |
| Supervisor |------| Server |------| Worker | |
+------------+ +----------+\ +--------+ | Multi-process
/ \ | or multi-thread
/ \ +--------+ |
Dynamic reconfiguration | Worker | |
and live restart support +--------+ |
---+
```
ServerEngine also provides useful options and utilities such as **logging**, **signal handlers**, **changing process names** shown by `ps` command, chuser, **stacktrace** and **heap dump on signal**.
* [Examples](#examples)
* [Simplest server](#simplest-server)
* [Multiprocess server](#multiprocess-server)
* [Multiprocess TCP server](#multiprocess-tcp-server)
* [Multiprocess server on Windows and JRuby platforms](#multiprocess-server-on-windows-and-jruby-platforms)
* [Logging](#logging)
* [Supervisor auto restart](#supervisor-auto-restart)
* [Live restart](#live-restart)
* [Dynamic config reloading](#dynamic-config-reloading)
* [Signals](#signals)
* [Utilities](#utilities)
* [BlockingFlag](#blockingflag)
* [SocketManager](#socketmanager)
* [Module API](#module-api)
* [Worker module](#worker-module)
* [Server module](#server-module)
* [List of all configurations](#list-of-all-configurations)
## Examples
### Simplest server
What you need to implement at least is a worker module which has `run` and `stop` methods.
```ruby
require 'serverengine'
module MyWorker
def run
until @stop
logger.info "Awesome work!"
sleep 1
end
end
def stop
@stop = true
end
end
se = ServerEngine.create(nil, MyWorker, {
daemonize: true,
log: 'myserver.log',
pid_path: 'myserver.pid',
})
se.run
```
Send `TERM` signal (or `KILL` on Windows) to kill the daemon. See also **Signals** section bellow for details.
### Multiprocess server
Simply set **worker_type: "process"** or **worker_type: "thread"** parameter, and set number of workers to `workers` parameter.
```ruby
se = ServerEngine.create(nil, MyWorker, {
daemonize: true,
log: 'myserver.log',
pid_path: 'myserver.pid',
worker_type: 'process',
workers: 4,
})
se.run
```
See also **Worker types** section bellow.
### Multiprocess TCP server
One of the typical implementation styles of TCP servers is that a parent process listens socket and child processes accept connections from clients.
ServerEngine allows you to optionally implement a server module to control the parent process:
```ruby
# Server module controls the parent process
module MyServer
def before_run
@sock = TCPServer.new(config[:bind], config[:port])
end
attr_reader :sock
end
# Worker module controls child processes
module MyWorker
def run
until @stop
# you should use Cool.io or EventMachine actually
c = server.sock.accept
c.write "Awesome work!"
c.close
end
end
def stop
@stop = true
end
end
se = ServerEngine.create(MyServer, MyWorker, {
daemonize: true,
log: 'myserver.log',
pid_path: 'myserver.pid',
worker_type: 'process',
workers: 4,
bind: '0.0.0.0',
port: 9071,
})
se.run
```
### Multiprocess server on Windows and JRuby platforms
Above **worker_type: "process"** depends on `fork` system call, which doesn't work on Windows or JRuby platform.
ServerEngine provides **worker_type: "spawn"** for those platforms (This is still EXPERIMENTAL). However, unfortunately, you need to implement different worker module because `worker_type: "spawn"` is not compatible with **worker_type: "process"** in terms of API.
What you need to implement at least to use worker_type: "spawn" is `def spawn(process_manager)` method. You will call `process_manager.spawn` at the method, where `spawn` is same with `Process.spawn` excepting return value.
In addition, Windows does not support signals. ServerEngine provides **command_sender: "pipe"** for Windows (and for other platforms, if you want to use it).
When using **command_sender: "pipe"**, the child process have to handle commands sent from parent process via STDIN.
On Windows, **command_sender: "pipe"** is default.
You can call `Server#stop(stop_graceful)` and `Server#restart(stop_graceful)` instead of sending signals.
```ruby
module MyWorker
def spawn(process_manager)
env = {
'SERVER_ENGINE_CONFIG' => config.to_json
}
script = %[
require 'serverengine'
require 'json'
conf = JSON.parse(ENV['SERVER_ENGINE_CONFIG'], symbolize_names: true)
logger = ServerEngine::DaemonLogger.new(conf[:log] || STDOUT, conf)
@stop = false
command_pipe = STDIN.dup
STDIN.reopen(File::NULL)
Thread.new do
until @stop
case command_pipe.gets.chomp
when "GRACEFUL_STOP"
@stop = true
when "IMMEDIATE_STOP"
@stop = true
when "GRACEFUL_RESTART", "IMMEDIATE_RESTART"
# do something...
end
end
end
until @stop
logger.info 'Awesome work!'
sleep 1
end
]
process_manager.spawn(env, "ruby", "-e", script)
end
end
se = ServerEngine.create(nil, MyWorker, {
worker_type: 'spawn',
command_sender: 'pipe',
log: 'myserver.log',
})
se.run
```
## Logging
ServerEngine logger rotates logs by 1MB and keeps 5 generations by default.
```ruby
se = ServerEngine.create(MyServer, MyWorker, {
log: 'myserver.log',
log_level: 'debug',
log_rotate_age: 5,
log_rotate_size: 1*1024*1024,
})
se.run
```
ServerEngine's default logger extends from Ruby's standard Logger library to:
* support multiprocess aware log rotation
* support reopening of log file
* support 'trace' level, which is lower level than 'debug'
See also **Configuration** section bellow.
## Supervisor auto restart
Server programs running 24x7 hours need to survive even if a process stalled because of unexpected memory swapping or network errors.
Supervisor process runs as the parent process of the server process and monitor it to restart automatically. You can enable supervisor process by setting `supervisor: true` parameter:
```ruby
se = ServerEngine.create(nil, MyWorker, {
daemonize: true,
pid_path: 'myserver.pid',
supervisor: true, # enables supervisor process
})
se.run
```
This auto restart reature will be suppressed for workers which exits with exit code specified by `unrecoverable_exit_codes`. At this case, whole process will exit without error (exit code 0).
## Live restart
You can restart a server process without waiting for completion of all workers using `INT` signal (`supervisor: true` and `enable_detach: true` parameters must be enabled).
This feature allows you to minimize downtime where workers take long time to complete a task.
```
# 1. starts server
+------------+ +----------+ +-----------+
| Supervisor |----| Server |----| Worker(s) |
+------------+ +----------+ +-----------+
# 2. receives SIGINT and waits for shutdown of the server for server_detach_wait
+------------+ +----------+ +-----------+
| Supervisor | | Server |----| Worker(s) |
+------------+ +----------+ +-----------+
# 3. starts new server if the server doesn't exit in server_detach_wait time
+------------+ +----------+ +-----------+
| Supervisor |\ | Server |----| Worker(s) |
+------------+ | +----------+ +-----------+
|
| +----------+ +-----------+
\--| Server |----| Worker(s) |
+----------+ +-----------+
# 4. old server exits eventually
+------------+
| Supervisor |\
+------------+ |
|
| +----------+ +-----------+
\--| Server |----| Worker(s) |
+----------+ +-----------+
```
Note that network servers (which listen sockets) shouldn't use live restart because it causes "Address already in use" error at the server process. Instead, simply use `worker_type: "process"` configuration and send `USR1` to restart workers instead of the server. It restarts a worker without waiting for shutdown of the other workers. This way doesn't cause downtime because server process doesn't close listening sockets and keeps accepting new clients (See also `restart_server_process` parameter if necessary).
## Dynamic config reloading
Robust servers should not restart only to update configuration parameters.
```ruby
module MyWorker
def initialize
reload
end
def reload
@message = config[:message] || "Awesome work!"
@sleep = config[:sleep] || 1
end
def run
until @stop
logger.info @message
sleep @sleep
end
end
def stop
@stop = true
end
end
se = ServerEngine.create(nil, MyWorker) do
YAML.load_file("config.yml").merge({
daemonize: true,
worker_type: 'process',
})
end
se.run
```
Send `USR2` signal to reload configuration file.
## Signals
- **TERM:** graceful shutdown
- **QUIT:** immediate shutdown (available only when `worker_type` is "process")
- **USR1:** graceful restart
- **HUP:** immediate restart (available only when `worker_type` is "process")
- **USR2:** reload config file and reopen log file
- **INT:** detach process for live restarting (available only when `supervisor` and `enable_detach` parameters are true. otherwise graceful shutdown)
- **CONT:** dump stacktrace and memory information to /tmp/sigdump-.log file. This can be
disabled by including `disable_sigdump: true` in the configuration.
Immediate shutdown and restart send SIGQUIT signal to worker processes which kills the processes.
Graceful shutdown and restart call `Worker#stop` method and wait for completion of `Worker#run` method.
Note that signals are not supported on Windows.
You have to use piped command instead of signals on Windows.
See also **Multiprocess server on Windows and JRuby platforms** section.
## Utilities
### BlockingFlag
`ServerEngine::BlockingFlag` is recommended to stop workers because `stop` method is called by a different thread from the `run` thread.
```ruby
module MyWorker
def initialize
@stop_flag = ServerEngine::BlockingFlag.new
end
def run
until @stop_flag.wait_for_set(1.0) # or @stop_flag.set?
logger.info @message
end
end
def stop
@stop_flag.set!
end
end
se = ServerEngine.create(nil, MyWorker) do
YAML.load_file(config).merge({
daemonize: true,
worker_type: 'process'
})
end
se.run
```
### SocketManager
`ServerEngine::SocketManager` is a powerful library to listen on the same port across multiple worker processes dynamically.
```ruby
module MyServer
def before_run
@socket_manager_server = ServerEngine::SocketManager::Server.open
@socket_manager_path = @socket_manager_server.path
end
def after_run
@socket_manager_server.close
end
attr_reader :socket_manager_path
end
module MyWorker
def initialize
@stop_flag = ServerEngine::BlockingFlag.new
@socket_manager = ServerEngine::SocketManager::Client.new(server.socket_manager_path)
end
def run
lsock = @socket_manager.listen_tcp('0.0.0.0', 12345)
until @stop
c = lsock.accept
c.write "Awesome work!"
c.close
end
end
def stop
@stop = true
end
end
se = ServerEngine.create(MyServer, MyWorker, {
daemonize: true,
log: 'myserver.log',
pid_path: 'myserver.pid',
worker_type: 'process',
workers: 4,
bind: '0.0.0.0',
port: 9071,
})
se.run
```
Other features:
- `socket_manager_server = SocketManager::Server.share_sockets_with_another_server(path)`
- It starts a new manager server that shares all UDP/TCP sockets with the existing manager.
- We can use this for live restart for network servers.
- The old process should stop without removing the file for the socket after the new process starts.
- Limitation: This feature would not work well if the process opens new TCP ports frequently.
See also [examples](https://github.com/fluent/serverengine/tree/master/examples).
## Module API
Available methods are different depending on `worker_type`. ServerEngine supports 3 worker types:
- **embedded**: uses a thread to run worker module (default). This type doesn't support immediate shutdown or immediate restart.
- **thread**: uses threads to run worker modules. This type doesn't support immediate shutdown or immediate restart.
- **process**: uses processes to run worker modules. This type doesn't work on Windows or JRuby platform.
- **spawn**: uses processes to run worker modules. This type works on Windows and JRuby platform but available interface of worker module is limited (See also Worker module section).
### Worker module
- interface
- `initialize` is called in the parent process (or thread) in contrast to the other methods
- `before_fork` is called before fork for each worker process [`worker_type` = "thread", "process"]
- `run` is the required method for `worker_type` = "embedded", "thread", "process"
- `spawn(process_manager)` is the required method for `worker_type` = "spawn". Should call `process_manager.spawn([env,] command... [,options])`.
- `stop` is called when TERM signal is received [`worker_type` = "embedded", "thread", "process"]
- `reload` is called when USR2 signal is received [`worker_type` = "embedded", "thread", "process"]
- `after_start` is called after starting the worker process in the parent process (or thread) [`worker_type` = "thread", "process", "spawn"]
- api
- `server` server instance
- `config` configuration
- `logger` logger
- `worker_id` serial id of workers beginning from 0
### Server module
- interface
- `initialize` is called in the parent process in contrast to the other methods
- `before_run` is called before starting workers
- `after_run` is called before shutting down
- `after_start` is called after starting the server process in the parent process (available if `supervisor` parameter is true)
- hook points (call `super` in these methods)
- `reload_config`
- `stop(stop_graceful)`
- `restart(stop_graceful)`
- api
- `config` configuration
- `logger` logger
## List of all configurations
- Daemon
- **daemonize** enables daemonize (default: false)
- **pid_path** sets the path to pid file (default: don't create pid file)
- **supervisor** enables supervisor if it's true (default: false)
- **disable_sigdump** disables the handling of the `SIGCONT` signal and dumping of the thread (default: false)
- **daemon_process_name** changes process name ($0) of server or supervisor process
- **chuser** changes execution user
- **chgroup** changes execution group
- **chumask** changes umask
- **daemonize_error_exit_code** exit code when daemonize, changing user or changing group fails (default: 1)
- Supervisor: available only when `supervisor` parameters is true
- **server_process_name** changes process name ($0) of server process
- **restart_server_process** restarts server process when it receives USR1 or HUP signal. (default: false)
- **enable_detach** enables INT signal (default: true)
- **exit_on_detach** exits supervisor after detaching server process instead of restarting it (default: false)
- **disable_reload** disables USR2 signal (default: false)
- **server_restart_wait** sets wait time before restarting server after last restarting (default: 1.0) [dynamic reloadable]
- **server_detach_wait** sets wait time before starting live restart (default: 10.0) [dynamic reloadable]
- Multithread server and multiprocess server: available only when `worker_type` is "thread" or "process" or "spawn"
- **workers** sets number of workers (default: 1) [dynamic reloadable]
- **start_worker_delay** sets the delay between each worker-start when starting/restarting multiple workers at once (default: 0) [dynamic reloadable]
- **start_worker_delay_rand** randomizes start_worker_delay at this ratio (default: 0.2) [dynamic reloadable]
- **restart_worker_interval** sets wait time before restarting a stopped worker (default: 0) [dynamic reloadable]
- Multiprocess server: available only when `worker_type` is "process"
- **worker_process_name** changes process name ($0) of workers [dynamic reloadable]
- **worker_heartbeat_interval** sets interval of heartbeats in seconds (default: 1.0) [dynamic reloadable]
- **worker_heartbeat_timeout** sets timeout of heartbeat in seconds (default: 180) [dynamic reloadable]
- **worker_graceful_kill_interval** sets the first interval of TERM signals in seconds (default: 15) [dynamic reloadable]
- **worker_graceful_kill_interval_increment** sets increment of TERM signal interval in seconds (default: 10) [dynamic reloadable]
- **worker_graceful_kill_timeout** sets promotion timeout from TERM to QUIT signal in seconds. -1 means no timeout (default: 600) [dynamic reloadable]
- **worker_immediate_kill_interval** sets the first interval of QUIT signals in seconds (default: 10) [dynamic reloadable]
- **worker_immediate_kill_interval_increment** sets increment of QUIT signal interval in seconds (default: 10) [dynamic reloadable]
- **worker_immediate_kill_timeout** sets promotion timeout from QUIT to KILL signal in seconds. -1 means no timeout (default: 600) [dynamic reloadable]
- **unrecoverable_exit_codes** handles worker processes, which exits with code included in this value, as unrecoverable (not to restart) (default: [])
- **stop_immediately_at_unrecoverable_exit** stops server immediately if a worker exits with unrecoverable exit status (default: false)
- Multiprocess spawn server: available only when `worker_type` is "spawn"
- all parameters of multiprocess server excepting worker_process_name
- **worker_reload_signal** sets the signal to notice configuration reload to a spawned process. Set nil to disable (default: nil)
- Logger
- **log** sets path to log file. Set "-" for STDOUT (default: STDERR) [dynamic reloadable]
- **log_level** log level: trace, debug, info, warn, error or fatal. (default: debug) [dynamic reloadable]
- **log_rotate_age** generations to keep rotated log files (default: 5), 0 will disable log rotate.
- **log_rotate_size** sets the size to rotate log files (default: 1048576), when `log_rotate_age` is 0, the `log_rotate_size` value will be ignored.
- **log_stdout** hooks STDOUT to log file (default: true)
- **log_stderr** hooks STDERR to log file (default: true)
- **logger_class** class of the logger instance (default: ServerEngine::DaemonLogger)
---
```
Author: Sadayuki Furuhashi
Copyright: Copyright (c) 2012-2013 Sadayuki Furuhashi
License: Apache License, Version 2.0
```