Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/acdha/mountstatus

MountStatusMonitor: paranoid monitor for POSIX filesystem mounts (Linux, OS X, FreeBSD)
https://github.com/acdha/mountstatus

Last synced: 2 days ago
JSON representation

MountStatusMonitor: paranoid monitor for POSIX filesystem mounts (Linux, OS X, FreeBSD)

Awesome Lists containing this project

README 

        
# Background



Operating systems have traditionally assumed that storage components are either

working or have failed and must be replaced. Most code has been developed using

the synchronous I/O model where operations wait until they either receive a

successful result or an error. This model is not suitable for complex storage

environments involving many devices or, especially, networks where failures are

more common and might be transient. Additionally, Linux, FreeBSD and macOS have

all had serious NFS client bugs and design flaws which significantly amplified

the damage from even a momentary failure or overload.



Unfortunately many programs have non-obvious triggers which cause them to scan

all mounted filesystems, causing them to hang as soon as any mounted filesystem

stops responding. In the case of NFS home directories the user experience is

especially bad because most desktop environments will completely hang trying to

access configuration files from the user's home directory, making the system

unusable.



Finally, very few applications have the complicated code required to report when a

storage request has blocked for a long period of time which makes it hard for a

system administrator to proactively correct the problem. In some cases a reboot

may be required but in many cases the fallout from a temporary outage can be

significantly reduced by using `umount -f` or, in certain situations on Linux,

`umount -f -l`, and remounting the filesystem so any new process will be

completely unaffected.



## What `mount_status_monitor` does



`mount_status_monitor` provides the missing notification solution for imperfect

storage. It's a simple daemon which periodically checks every mounted filesystem

using an asynchronous check with a timeout so it can report soft failures caused

by unresponsive storage as well as hard errors.



After each run is complete it will send a message to syslog:



    Checked 5 mounts; 0 are dead



Optionally, the [Prometheus push-gateway](https://prometheus.io/docs/instrumenting/pushing/)

will receive two metrics (`total_mountpoints` and `dead_mountpoints`) with the

same information for alerting and correlation purposes.



When a mount test fails the mountpoint will be sent to syslog and stderr:



    Mount failed health-check: /Volumes/TestSSHFS



There are several ways to simulate failures for testing. The easiest is to use a

user-mode filesystem such as sshfs, s3fs, etc. and use `kill -STOP` to freeze

the FUSE process long enough to trigger the unresponsive mount failure. For more

involved testing or if you are also evaluating system tuning options you can use

iptables to simulate packet loss or hard failure of an NFS server.



## Installation



Compiling the code requires a working [Rust](https://www.rust-lang.org) toolchain:



    cargo build --release



A Docker image is provided for testing basic functionality:



    docker build -t mountstatus . && docker run -it --rm mountstatus



### Running the monitor



For testing you can simply run `mount_status_monitor` directly and watch the

output. Note that while the process can run without elevated permissions it is

likely that this will generate error messages due to mountpoints which are

inaccessible.



In normal operation `mount_status_monitor` relies a supervisor such as Upstart,

systemd, or launchd to keep it running. See the `upstart` and `systemd`

directories for provided config files.



## Future Directions



A long term experiment is having `mount_status_monitor` actually attempt to run

`umount -f` (or `umount -f -l` on Linux) and remount the filesystem to reduce

the number of applications which hit a dead mountpoint. This may not be

appropriate for all users and would require testing to avoid making the problem

worse.



## Contributors & Acknowledgements



Thanks to the following Rust community members who volunteered to review &

improve this code, listed in alphabetical order:



- Pascal Hertleif (@killercup)

- Paul Daniel Faria (@Nashenas88)

- Rahul Sharma (@creativcoder)

- Zachary Dremann (@Dr-Emann)