Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ssokolow/rust-cli-boilerplate

Rust project boilerplate for CLI applications
https://github.com/ssokolow/rust-cli-boilerplate

boilerplate boilerplate-application boilerplate-template cli cli-app cli-command rust rust-lang rust-language rustlang

Last synced: about 1 month ago
JSON representation

Rust project boilerplate for CLI applications

Awesome Lists containing this project

README

        

# Rust CLI Project Template

![MIT/Apache 2.0](https://img.shields.io/badge/license-MIT%2FApache_2.0-blue.svg)
![POSIX-only build tooling](https://img.shields.io/badge/dev_platform-POSIX-lightgrey.svg)

A base project template for comfortably building small but reliable utilities in
the [Rust](https://rust-lang.org/) programming language.

## Rationale

Historically, I have used Python for quick little scripts (from which my big,
long-lived projects tend to unexpectedly develop), but maintaining large things
written in Python risks turning into a
[gumption trap](https://en.wikipedia.org/wiki/Gumption_trap) due to the need to
either manually test every code path I affect when making changes or pour a lot
of effort into writing an automated test suite to do it for me.

Rust's stronger compile-time guarantees remove most of that motivation-sapping
busywork, but, since I've spent so many years getting comfortable with Python,
working in a new language risks being a chicken-and-egg problem which kills off
the will to change tasks. (Once I start something, it's easy to keep going.)

The purpose of this repository is to lower the friction for Rust-based
development of such "scripts of unknown potential" as much as possible. (Hence
the inclusion of aliases like `just run` which serve only to remove the need to
think about whether to type `just` or `cargo`, and the wrappers for `cargo-edit`
commands which regenerate the API documentation I'll have open in a browser tab
after adding/removing/updating dependencies.)

In short, this is a "do it properly" answer to the workflow that evolved around
the
[`boiler` snippet](https://gist.github.com/ssokolow/ef59f97673693d717f5096d7987afd2c)
for Python that I keep in my [UltiSnips](https://github.com/SirVer/ultisnips).

## License

This repository is licensed under your choice of the
[MIT](http://opensource.org/licenses/MIT) or
[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) licenses with the
exception of the license texts themselves. Please replace the `LICENSE` file in
the `template` folder with your preferred license if you do not habitually start
your new projects under the GNU GPLv3.

## Features

- Uses [StructOpt](https://github.com/TeXitoi/structopt) (with colourized
`--help` output and "Did you mean...?" suggestions enabled) for argument
parsing because [gumdrop](https://github.com/murarth/gumdrop) doesn't support
`OsString`-based parsing for correct handling of
[non-UTF8 paths](https://en.wikipedia.org/wiki/Mojibake).
- Uses [anyhow](https://github.com/dtolnay/anyhow) for unified error handling.
- Presents an `app::main(opts: CliOpts) -> Result<()>` function to keep your
application logic cleanly separated from argument parsing and handling of
fatal errors.
- Exposes clap's support for generating shell completions by providing a
`--dump-completions ` option.
- Enables almost all rustc and
[clippy](https://rust-lang.github.io/rust-clippy/current/) lints without
making clippy mandatory.
- A comprehensive set of [just](https://github.com/casey/just) commands, easily
customized via variables (eg. for cross-compilation), including `install` and
`uninstall`, which also take care of shell completions and a manpage.
- `just build-dist` for a 100% static x86_64 binary that starts at roughly
`272KiB` (`248KiB` with `panic="abort"`) in new projects.
- `just install-deps` to install all but two optional dependencies on
Debian-family distros.
- `just install-cargo-deps` to install all distro-agnostic dependencies.
- A basic `.travis.yml` for use with [Travis-CI](https://travis-ci.org/) and
[Nightli.es](https://nightli.es/).
- The `just fmt` command always calls the nightly version of rustfmt to ensure
access to the excessive number of customization options which are gated away
as unstable.

## Usage

1. Clone a copy of this repository (`.git` must exist, so no archive downloads)
2. Run `apply.py path/to/new/project`
3. Edit `src/app.rs` to implement your application logic

**NOTE:** As a safety measure, `apply.py` generates new projects from the git
`HEAD` of the template repository (not the working tree), so any local changes
you make will not be picked up until you commit them.

## Supplementary Files

Metadata

LICENSE
A copy of the GNU GPLv3
as my "until I've had time to think about it"
license of choice. Make a local commit which replaces this with your preferred
default for new projects.

CONTRIBUTING
A copy of the
Developer Certificate of Origin,
which is the Linux kernel developers' more ideologically appropriate
alternative to CLAs as a means of legally armouring themselves against
bad-faith contributions

Configuration

.gitignore
Ignore /target and other generated files

clippy.toml
Whitelist for CamelCase names which trigger Clippy's "identifier needs
backticks" lint

rustfmt.toml
A custom rustfmt configuration which shows
TODO/FIXME comments and attempts to make it conform
to the style I'm willing to enforce at the expense of not using rustfmt if
necessary.

Development Automation

apply.py
Run this to generate new projects as a workaround for cargo-generate's
incompatibility with justfile syntax

justfile
Build/development-automation commands via
just (a pure-Rust make-alike)

Support Code You May Borrow

gen_justfile_reference.py
Code which is used to regenerate the reference charts for
justfile variables and commands in this README so it's easy
to keep them up to date.

test_justfile.py
A test suite for my justfile which you may want to adapt
for your own projects.

## Justfile Reference

### Variables (`just --evaluate`)

VariableDefault ValueDescription

CARGO_BUILD_TARGET
x86_64-unknown-linux-musl
The target for cargo commands to use and
install-rustup-deps to install

build_flags

An easy place to modify the build flags used

channel
stable
An easy way to override the cargo channel for just this project

features

Extra cargo features to enable

build-dist

sstrip_bin
sstrip
Set this if you need to override it for a cross-compiling sstrip

strip_bin
strip
Set this to the cross-compiler's strip when cross-compiling

strip_flags
--strip-unneeded
Flags passed to strip_bin

upx_flags
--ultra-brute
Flags passed to UPX

kcachegrind

callgrind_args

Extra arguments to pass to callgrind.

callgrind_out_file
callgrind.out.justfile
Temporary file used by just kcachegrind

kcachegrind
kcachegrind
Set this to override how kcachegrind is called

install and uninstall

bash_completion_dir
~/.bash_completion.d
Where to install bash completions. You'll need to manually
add some lines to source these files in .bashrc

fish_completion_dir
~/.config/fish/completions
Where to install fish completions. You'll probably never need to
change this.

manpage_dir
~/.cargo/share/man/man1
Where to install manpages. As long as ~/.cargo/bin is
in your PATH, man should automatically pick up this
location.

zsh_completion_dir
~/.zsh/functions
Where to install zsh completions. You'll need to add this
to your fpath manually

### Commands (`just --list`)

CommandArgumentsDescription

DEFAULT

Shorthand for just fulltest

Development

add
args (optional)
Alias for cargo-edit's cargo add which regenerates local API docs
afterwards

bloat
args (optional)
Alias for cargo bloat

check
args (optional)
Alias for cargo check

clean
args (optional)
Superset of cargo clean -v which deletes other stuff this justfile
builds

clippy
args (optional)
Alias for cargo clippy which touches src/* to work
around clippy bug

doc
args (optional)
Run rustdoc with --document-private-items and then run
cargo-deadlinks

fmt
args (optional)
Alias for cargo +nightly fmt -- {{args}}

fmt-check
args (optional)
Alias for cargo +nightly fmt -- --check {{args}} which un-bloats
TODO/FIXME warnings

fulltest

Run all installed static analysis, plus cargo test

geiger
args (optional)
Alias for cargo geiger

kcachegrind
args (optional)
Run a debug build under callgrind, then open
the profile in KCachegrind

kcov

Generate a statement coverage report in target/cov/

rm
args (optional)
Alias for cargo-edit's cargo rm which regenerates local API docs
afterwards

search
args (optional)
Convenience alias for opening a crate search on lib.rs in the browser

test
args (optional)
Alias for cargo test

update
args (optional)
Alias for cargo-edit's cargo update which regenerates local API
docs afterwards

Local Builds

build

Alias for cargo build

install

Install the un-packed binary, shell completions, and a manpage

run
args (optional)
Alias for cargo run -- {{args}}

uninstall

Remove any files installed by the install task (but leave any
parent directories created)

Release Builds

build-dist

Make a release build and then strip and compress the resulting binary

dist

Call dist-supplemental and build-dist and copy the
packed binary to dist/

dist-supplemental

Build the shell completions and a manpage, and put them in dist/

Dependencies

install-apt-deps

Use apt-get to install dependencies cargo can't
(except kcov and sstrip)

install-cargo-deps

install-rustup-deps and then cargo install tools

install-deps

Run install-apt-deps and install-cargo-deps. List what
remains.

install-rustup-deps

Install (don't update) nightly and channel toolchains, plus
CARGO_BUILD_TARGET, clippy, and rustfmt

### Tips

- Edit the `DEFAULT` command. That's what it's there for.
- You can use `just` from any subdirectory in your project. It's like `git` that
way.
- `just path/to/project/` (note the trailing slash) is equivalent to
`(cd path/to/project; just)`
- `just path/to/project/command` is equivalent to
`(cd path/to/project; just command)`

- The simplest way to activate the bash completion installed by `just install`
is to add this to your `.bashrc`:

```sh
for script in ~/.bash_completion.d/*; do
if [ -e "$script" ]; then
. "$script"
fi
done
```

foo

- The simplest way to activate the zsh completion installed by `just install` is
to add this to your `.zshrc`:

```zsh
fpath=(~/.zsh/functions(:A) $fpath)
```

- When using clap/StructOpt validators for inputs such as filesystem paths, only
use them to bail out early on bad input, not as your only check. They're
conceptually similar to raw pointers and can be invalidated between when you
check them and when you try to use them because Rust can't control what the OS
and other programs do in the interim. See
[this blog post](http://blog.ssokolow.com/archives/2016/10/17/a-more-formal-way-to-think-about-validity-of-input-data/)
for more on this idea of references versus values in command-line arguments.

## Build Behaviour

In order to be as suitable as possible for building compact, easy-to-distribute,
high-reliability replacements for shell scripts, the following build options are
defined:

### If built via `just build`:

1. The default `CARGO_BUILD_TARGET` defined in the `justfile` will specify a
32-bit x86 build, statically linked against
[musl-libc](http://www.musl-libc.org/) for portability comparable to a
[Go]() binary.
(Unless `musl-gcc` is installed, this will cause build failures if you depend
on any crates which link to C or C++ code.)

### If built via `cargo build --release`:

1. Full LTO (Link-Time Optimization) will be enabled. (`lto = true`)
2. The binary will be built with `opt-level = "z"` to further reduce file size.
3. If `panic="abort"` is uncommented in `Cargo.toml`, LTO will prune away the
unwinding machinery to save even more space, but panics will not cause `Drop`
implementations to be run and will be uncatchable.

### If built via `just build-dist`:

1. Unless otherwise noted, all optimizations listed above.
[[1]](https://lifthrasiir.github.io/rustlog/why-is-a-rust-executable-large.html)
2. The binary will be stripped with
[`--strip-unneeded`](https://www.technovelty.org/linux/stripping-shared-libraries.html)
and then with
[`sstrip`](http://www.muppetlabs.com/~breadbox/software/elfkickers.html) (a
[more aggressive](https://github.com/BR903/ELFkickers/tree/master/sstrip)
companion used in embedded development) to produce the smallest possible
pre-compression size.
3. The binary will be compressed via
[`upx --ultra-brute`](https://upx.github.io/). In my experience, this makes a
file about 1/3rd the size of the input.

**NOTE:** `--strip-unneeded` removes all symbols that `readelf --syms` sees from
the `just build` output, so it's not different from `--strip-all` in this case,
but it's a good idea to get in the habit of using the safe option that's smart
enough to just Do What I Mean™.

### If built by `just dist`:

1. A packed binary will be built via `build-dist` and copied into `dist/`
2. Shell completion files and a manpage will also be built and saved into
`dist/`

**NOTE:** Depending on who you're distributing precompiled binaries to, you may
want get an overview of how virus scanners react to your binary using
[VirusTotal](https://www.virustotal.com/).

Especially with anything involving compression, small numbers of false positives
are a fact of life in the world of virus detection. For example, when I tested
the official installer for the
[NSIS](https://en.wikipedia.org/wiki/Nullsoft_Scriptable_Install_System)
authoring tools, which is used by various major companies including McAfeee, two
or three no-name entries in the list of 60+ virus scanners they test reported it
to have a virus.

If this proves problematic, you can either uninstall UPX or modify the
`justfile` so the `dist` command always prefers the `.stripped` copy of the
binary over the `.packed` one.

## Experimental Cross-Compilation Support

I am currently in the process of extending this template to support generating
Windows binaries, though I have no immediate plans to replace the `justfile`
tasks so, for now, Windows-hosted development will have to settle for calling
`cargo` commands directly.

**NOTE:** I haven't yet used a fresh Ubuntu install under VirtualBox to verify
that I've correctly listed all the steps needed to achieve a working build
environment.

To set up an environment where setting `CARGO_BUILD_TARGET` to
`x86_64-pc-windows-gnu` will complete successfully and produce a `.exe` file
which appears to work under my preliminary testing:

1. Install a MinGW package like Ubuntu's `mingw-w64`
2. Run `rustup target add x86_64-pc-windows-gnu`
3. Add the following two lines to `~/.cargo/config`:

```toml
[target.x86_64-pc-windows-gnu]
linker = "/usr/bin/x86_64-w64-mingw32-gcc"
```

To make `cargo test` also work cross-platform:

1. Make sure your `~/.wine` is 64-bit (indicated by an `#arch=win64` comment in
`system.reg`)
2. Make sure your kernel's `binfmt_misc` support is configured to allow running
`.exe` files in the terminal via Wine as if they were native binaries.

**NOTE:** Wine is only suitable for "rapid iteration, approximate compatibility"
testing. For proper testing of Windows binaries, the only reliable solution is
to download one of the specially licensed "only for testing"
[Windows VMs](https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/)
that Microsoft offers for download from [http://modern.ie/](http://modern.ie/)
and those cannot be used to make legally redistributable builds.

If you want to set up a Continuous Deployment-style workflow with testing
against real Windows targets, the only viable option is to bypass `just` and
call `cargo` directly under real Windows. I suggest a CI service like
[AppVeyor](appveyor.com) for this. (See also
[rust-cross](https://github.com/japaric/rust-cross).)

## Dependencies

In order to use the full functionality offered by this boilerplate, the
following dependencies must be installed:

- `just add`:
- [cargo-edit](https://github.com/killercup/cargo-edit)
(`cargo install cargo-edit`)
- `just bloat`:
- [cargo-bloat](https://github.com/RazrFalcon/cargo-bloat)
(`cargo install cargo-bloat`)
- `just build-dist`:
- The toolchain specified by the channel variable.
- The target specified by the CARGO_BUILD_TARGET variable.
- `strip` (Included with binutils)
- [`sstrip`](http://www.muppetlabs.com/~breadbox/software/elfkickers.html)
**(optional)**
- [`upx`](https://upx.github.io/) (**optional**, `sudo apt-get install upx`)
- `just fmt` and `just fmt-check`:
- A nightly Rust toolchain (`rustup toolchain install nightly`)
- [rustfmt](https://github.com/rust-lang/rustfmt) for the nightly toolchain
(`rustup component add rustfmt --toolchain nightly`)
- `just dist-supplemental`:
- [help2man](https://www.gnu.org/software/help2man/)
(`sudo apt-get install help2man`)
- `just kcachegrind`:
- [Valgrind](http://valgrind.org/) (`sudo apt-get install valgrind`)
- [KCachegrind](https://kcachegrind.github.io/)
(`sudo apt-get install kcachegrind`)
- `just kcov`:
- A [Rust-compatible build](http://sunjay.ca/2016/07/25/rust-code-coverage) of
kcov
- `just rm`:
- [cargo-edit](https://github.com/killercup/cargo-edit)
(`cargo install cargo-edit`)
- `just test`:
- [clippy](https://github.com/rust-lang/rust-clippy)
(`rustup component add clippy`)
- [cargo-audit](https://github.com/RustSec/cargo-audit)
(`cargo install cargo-audit`)
- [cargo-deadlinks](https://github.com/deadlinks/cargo-deadlinks)
(`cargo install cargo-deadlinks`)
- [cargo-outdated](https://github.com/kbknapp/cargo-outdated)
(`cargo install cargo-outdated`)
- `just update`:
- [cargo-edit](https://github.com/killercup/cargo-edit)
(`cargo install cargo-edit`)

### Dependency Installation

- **Debian/Ubuntu/Mint:**

```sh
export PATH="$HOME/.cargo/bin:$PATH"
cargo install just
just install-deps

# ...and now manually install the following optional tools:
# - sstrip (from ELFkickers)
# - kcov (version 31 or higher with --verify support)
```

- **Other distros:**

```sh
export PATH="$HOME/.cargo/bin:$PATH"
cargo install just
just install-cargo-deps

# ...and now manually install the following optional tools:
# - help2man
# - kcachegrind
# - kcov (version 31 or higher with --verify support)
# - strip (from binutils)
# - sstrip (from ELFkickers)
# - upx
# - valgrind
```

## TODO

- Add a `.travis.yml` at the top level to plumb the various test suites
(template repo and generated project) into CI and then add a badge.
- Add a `#[cfg(windows)]` version of the `path_output_dir` validator and make
the `libc` dependency conditional on `not(windows)` so that cross-compiling
for Windows using the `x86_64-pc-windows-gnu` target can be a viable way to
quickly fire off alpha/beta-testing builds to Windows-using peers.
- Investigate how flexible [QuiCLI](https://github.com/killercup/quicli) and its
dependency on env_logger are and whether it'd be useful to rebase on it or
whether I'd just be reinventing most of it anyway to force the exact look and
feel I achieved with stderrlog. (eg. The `Verbosity` struct doesn't implement
"`-v` and `-q` are mirrors of each other" and I'm rather fond of stderrlog's
approach to timestamp toggling.)
- What effect does QuiCLI have on the final binary size? (not a huge concern)
- Investigate why [cargo-cov](https://github.com/kennytm/cov) isn't hiding the
components of the rust standard library and whether it can be induced to
generate coverage despite some tests failing. If so, add a command for it.
- Read the [callgrind docs](http://valgrind.org/docs/manual/cl-manual.html) and
figure out how to exclude the Rust standard library from what Kcachegrind
displays.
- I may need to filter the output.
[[1]](https://stackoverflow.com/questions/7761448/filter-calls-to-libc-from-valgrinds-callgrind-output)
- Figure out how to add a `just` task for a faster but less precise profiler
like [gprof](https://en.wikipedia.org/wiki/Gprof)
[[1]](http://www.thegeekstuff.com/2012/08/gprof-tutorial/)
[[2]](https://sourceware.org/binutils/docs/gprof/),
[OProfile](http://oprofile.sourceforge.net/)
[[1]](https://llogiq.github.io/2015/07/15/profiling.html), or
[perf](https://perf.wiki.kernel.org/index.php/Main_Page)
[[1]](http://blog.adamperry.me/rust/2016/07/24/profiling-rust-perf-flamegraph/)
to make it easy to leverage the various trade-offs. (And make sure to
provide convenient access to flame graphs and at least one perf inspector
GUI or TUI.)
- Include references to these resources on how profilers can mislead in
different ways.
[[1]](http://yosefk.com/blog/how-profilers-lie-the-cases-of-gprof-and-kcachegrind.html)
[[2]](http://www.catb.org/~esr/writings/taoup/html/optimizationchapter.html)
[[3]](http://blog.adamperry.me/rust/2016/07/24/profiling-rust-perf-flamegraph/)
- Look into options for making it as easy as possible to optimize and
regression-test runtime performance.
[[1]](https://github.com/rust-lang/rust/issues/31265https://github.com/rust-lang/rust/issues/31265)
[[2]](https://crates.io/crates/bencher)
[[3]](https://github.com/japaric/criterion.rshttps://github.com/japaric/criterion.rs)
[[4]](https://github.com/BurntSushi/cargo-benchcmp)
- Test and enhance `.travis.yml`

- Consider officially supporting Windows as a target (probably using
[cargo-make](https://crates.io/crates/cargo-make) instead of Just) and, if I
do, come up with an `appveyor.yml`... possibly the one from
[this project](https://github.com/starkat99/appveyor-rust).

- Add a `run-memstats` Just task which swaps in jemalloc and sets
`MALLOC_CONF=stats_print:true`
- Investigate commit hooks
[[1]](https://stackoverflow.com/questions/3462955/putting-git-hooks-into-repository)
[[2]](https://stackoverflow.com/questions/427207/can-git-hook-scripts-be-managed-along-with-the-repository)
[[3]](https://mpdaugherty.wordpress.com/2010/04/06/how-to-include-git-hooks-in-a-repository-and-still-personalize-your-machine/)
- Once I've cleared out these TODOs, consider using this space for a reminder
list of best practices for avoiding "higher-level footguns" noted in my pile
of assorted advice. (Things like "If you can find a way to not need path
manipulation beyond 'pass this opaque token around', then you can eliminate
entire classes of bugs")
- At least _list_ a snip of example code for something like
[rustyline](https://lib.rs/crates/rustyline) as the suggested way to do simple
user prompting.
- Gather my custom clap validators into a crate, add some more, and have this
depend on it:
- Self-Contained data:
- Boolean is `1`/`y`/`yes`/`t`/`true` or `0`/`n`/`no`/`f`/`false`
(case-insensitive, include a utility function for actual parsing)
- Integers:
- Can be parsed as a decimal integer `> 0` (eg. number of volumes)
- Can be parsed as a decimal integer `>= 0` (eg. number of bytes)
- Number of bytes, with optional SI mebi- unit suffix (eg. `16m`,
including optional `b`, case-insensitive)
- Floats:
- Can be parsed as a float in the range `0.0 <= x <= 1.0`
- Invalidatable/Referential data:
- Input files:
- Directory exists and is browsable (`+rX`)
- Path is a readable file or browsable directory (ie. read or recurse)
- Output files:
- Integers:
- Augmented "number of bytes, with optional SI mebi- unit suffix"
validator with upper limit for producing files representable by
ISO9660/FAT32 filesystems on removable media. (2GiB, since some
implementations use 32-bit signed offsets)
- Paths:
- File path is probably FAT32 writable
- If file exists, `access()` says it's probably writable
- If file does not exist, name is FAT32-valid and within a probably
writable directory.
- File path is probably FAT32 writable, with `mkdir -p`
- Nonexistent path components are FAT32-valid
- Closest existing ancestor is a probably writable directory
- Network I/O:
- Integers:
- Successfully parses into a valid listening TCP/UDP port number
(0-65535, I think)
- Successfully parses into a valid, non-root, listening TCP/UDP port
number (0 or 1024-65535, I think)
- Successfully parses into a valid connecting TCP/UDP port number
(1-65535, I think)
- Strings:
- Successfully parses into a
[`SocketAddr`](https://doc.rust-lang.org/std/net/enum.SocketAddr.html)
(IP+port, may perform DNS lookup?)
- Successfully parses into an
[`IpAddr`](https://doc.rust-lang.org/std/net/enum.IpAddr.html) (may
perform DNS lookup?)
- URLs:
- Is well-formed relative URL
([external dependency](https://docs.rs/url/1.4.0/url/) behind a cargo
feature)
- Is well-formed absolute URL
([external dependency](https://docs.rs/url/1.4.0/url/) behind a cargo
feature)