Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dimo414/bkt
a subprocess caching utility, available as a command line binary and a Rust library.
https://github.com/dimo414/bkt
bash cache caching cli fzf memoization rust shell subprocess
Last synced: 9 days ago
JSON representation
a subprocess caching utility, available as a command line binary and a Rust library.
- Host: GitHub
- URL: https://github.com/dimo414/bkt
- Owner: dimo414
- License: mit
- Created: 2021-02-02T10:38:14.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2024-01-25T17:38:52.000Z (10 months ago)
- Last Synced: 2024-04-14T12:21:49.005Z (7 months ago)
- Topics: bash, cache, caching, cli, fzf, memoization, rust, shell, subprocess
- Language: Rust
- Homepage: https://www.bkt.rs
- Size: 208 KB
- Stars: 209
- Watchers: 6
- Forks: 13
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# `bkt`
[](https://github.com/dimo414/bkt/releases)
[](https://crates.io/crates/bkt)
[](https://docs.rs/bkt)
[](https://github.com/dimo414/bkt/actions)
[](https://github.com/dimo414/bkt/issues)
[](https://github.com/dimo414/bkt/blob/master/LICENSE)
`bkt` (pronounced "bucket") is a subprocess caching utility written in Rust,
inspired by [bash-cache](https://github.com/dimo414/bash-cache).
Wrapping expensive process invocations with `bkt` allows callers to reuse recent
invocations without complicating their application logic. This can be useful in
shell prompts, interactive applications such as [`fzf`](#fzf), and long-running
programs that poll other processes.
`bkt` is available as a standalone binary as well as a
[Rust library](https://crates.io/crates/bkt). See https://docs.rs/bkt/ for
library documentation. This README covers the `bkt` binary.
## Installation
Run `cargo install bkt` to compile and install `bkt` locally. You will need to
[install `cargo`](https://doc.rust-lang.org/cargo/getting-started/installation.html)
if it's not already on your system.
Pre-compiled binaries for common platforms are attached to each
[release](https://github.com/dimo414/bkt/releases) (starting with 0.5). Please
open an issue or send a PR if you would like releases to include binaries for
additional platforms.
Package manager support is being tracked
[here](https://github.com/dimo414/bkt/issues/12); volunteers are welcome.
[](https://repology.org/project/bkt/versions)
## Usage
```
bkt --ttl=DURATION [--stale=DURATION] [--cwd] [--env=ENV ...] [--modtime=FILE ...] [--scope=SCOPE] [--discard-failures] [--warm|--force] -- ...
```
`bkt` is easy to start using - simply prefix the command you intend to cache
with `bkt --ttl=[some duration] --`, for example:
```shell
# Execute and cache an invocation of 'date +%s.%N'
$ bkt --ttl=1m -- date +%s.%N
1631992417.080884000
# A subsequent invocation reuses the same cached output
$ bkt --ttl=1m -- date +%s.%N
1631992417.080884000
```
When `bkt` is passed a command it hasn't seen before (or recently) it executes
the command synchronously and caches its stdout, stderr, and exit code. Calling
`bkt` again with the same command reads the data from the cache and outputs it
as if the command had been run again.
### Cache Lifespan
Two flags, `--ttl` and `--stale`, configure how long cached data is preserved.
The TTL (Time to Live) specifies how long cached data will be used. Once the
TTL expires the cached data will be discarded and the backing command re-run.
A TTL can also be configured by setting a `BKT_TTL` environment variable.
When the data expires `bkt` has to re-execute the command synchronously, which
can introduce unexpected slowness. To avoid this, pass `--stale` with a shorter
duration than the TTL. When the cached data is older than the stale threshold
this causes `bkt` to refresh the cache in the background while still promptly
returning the cached data.
Both flags (and `BKT_TTL`) accept duration strings such as `10s` or
`1hour 30min`. The exact syntax is defined in the
[humantime](https://docs.rs/humantime/2.1.0/humantime/fn.parse_duration.html)
library.
### Execution Environment
Some commands' behavior depends on more than just the command line arguments.
It's possible to adjust how `bkt` caches such commands so that unrelated
invocations are cached separately.
#### Working Directory
For example, attempting to cache `pwd` will not work as expected by default:
```shell
$ $ bkt --ttl=1m -- pwd
/tmp/foo
$ cd ../bar
# Cached output for 'pwd' is reused even though the directory has changed
$ bkt --ttl=1m -- pwd
/tmp/foo
```
To have `bkt` key off the current working directory in addition to the command
line arguments pass `--cwd`:
```shell
$ bkt --cwd --ttl=1m -- pwd
/tmp/foo
$ cd ../bar
$ bkt --cwd --ttl=1m -- pwd
/tmp/bar
```
#### Environment Variables
Similarly, to specify one or more environment variables as relevant for the
command being cached use `--env`, such as `--env=LANG`. This flag can be
provided multiple times to key off additional variables. Invocations with
different values for any of the given variables will be cached separately.
#### File Modifications
`bkt` can also check the last-modified time of one or more files and include
this in the cache key using `--modtime`. For instance passing
`--modtime=/etc/passwd` would cause the backing command to be re-executed any
time `/etc/passwd` is modified even if the TTL has not expired.
### Refreshing Manually
It's also possible to trigger refreshes manually using `--force` or `--warm`.
The former behaves exactly as if the cached data was not found, executing the
process and caching the result. This is useful if you know the cached data
is no longer up-to-date, e.g. because something external changed.
Alternatively, it can be useful to refresh the cache asynchronously, which
`--warm` provides. This triggers a refresh in the background but immediately
ends the current process with no output. This is useful if you expect
additional invocations in the near future and want to ensure they get a cache
hit. Note that until the warming process completes concurrent calls may still
see a cache miss and trigger their own invocation.
### Setting a Cache Scope
Cached data is persisted to disk (but see [below](#cache_dir)), and is
available to any process that invokes `bkt`. Generally this is desirable, but
certain usages may want to isolate their invocations from other potential
concurrent calls.
To do so pass `--scope=...` with a sufficiently unique argument, such as a fixed
label for the calling program, the current process ID, or a timestamp.
```shell
$ bkt --ttl=1m -- date +%s.%N
1631992417.080884000
# Changing the scope causes the command to be cached separately
$ bkt --scope=foo --ttl=1m -- date +%s.%N
1631992418.010562000
```
Alternatively, define a `BKT_SCOPE` environment variable to configure a
consistent scope across invocations. This can be useful within a script to
ensure all commands share a scope.
```shell
#!/bin/bash
# Set a unique scope for this script invocation using the PID and current time
export BKT_SCOPE="my_script_$$_$(date -Ins)"
```
### Discarding Failed Invocations
By default, all invocations are cached regardless of their output or exit code.
In situations where failures should not be cached pass `--discard-failures` to
only persist successful invocations (those that return a `0` exit code).
**WARNING:** Passing this flag can cause the backing command to be invoked more
frequently than the `--ttl` would suggest, which in turn can create unexpected
load. If the backing command is failing due to an outage or bug (such as an
overloaded website) triggering additional calls can exacerbate the issue and
effectively DDoS the hampered system. It is generally safer *not* to set this
flag and instead make the client robust to occasional failures.
### Changing the Cache Directory
By default, cached data is stored under your system's temporary directory
(typically `/tmp` on Linux).
You may want to use a different location for certain commands, for instance to
be able to easily delete the cached data as soon as it's no longer needed. You
can specify a custom cache directory via the `--cache-dir` flag or by defining
a `BKT_CACHE_DIR` environment variable.
Note that the choice of directory can affect `bkt`'s performance: if the cache
directory is on a [`tmpfs`](https://en.wikipedia.org/wiki/Tmpfs) or solid-state
partition it will be significantly faster than one using a spinning disk.
If your system's temporary directory is not a good choice for the default cache
location (e.g. it is not a `tmpfs`) you can specify a different location by
defining a `BKT_TMPDIR` environment variable (for example in your `.bashrc`).
These two environment variables, `BKT_TMPDIR` and `BKT_CACHE_DIR`, have similar
effects but `BKT_TMPDIR` should be used to configure the system-wide default,
and `--cache-dir`/`BKT_CACHE_DIR` used to override it.
`bkt` periodically prunes stale data from its cache, but it also assumes the
operating system will empty its temporary storage from time to time (for `/tmp`
this typically happens on reboot). If you opt to use a directory that the
system does not maintain, such as `~/.cache`, you may want to manually delete
the cache directory on occasion, such as when upgrading `bkt`.
## Security and Privacy
The default cache directory is potentially world-readable. On Unix the cache
directory is created with `700` permissions, meaning only the current user can
access it, but this is not foolproof.
You can customize the cache directory (see [above](#cache_dir)) to a location
you trust such as `~/.cache`, but note that your home directory may be slower than
the temporary directory selected by default.
In general, if you are not the only user of your system it's wise to configure
your `TMPDIR` to a location only you can access. If that is not possible use
`BKT_TMPDIR` to configure a custom temporary directory specifically for `bkt`.
## Patterns and Tips
**Please share how you're using `bkt` on the
[Discussion Board](https://github.com/dimo414/bkt/discussions/categories/show-and-tell)!**
### Speeding up `fzf` and other preview tools
`bkt` works well with interactive tools like
[`fzf`](https://github.com/junegunn/fzf) that execute other commands. Because
`fzf` executes the `--preview` command every time an element is selected it can
be slow and tedious to browse when the command takes a long time to run. Using
`bkt` allows each selection's preview to be cached. Compare:
```shell
$ printf '%s\n' 1 0.2 3 0.1 5 | \
fzf --preview="bash -c 'sleep {}; echo {}'"
$ printf '%s\n' 1 0.2 3 0.1 5 | \
fzf --preview="bkt --ttl=10m --stale=10s -- bash -c 'sleep {}; echo {}'"
```
You'll generally want to use a long TTL and a short stale duration so that
even if you leave `fzf` running for a while the cache remains warm and is
refreshed in the background. You may also want to set a `--scope` if it's
important to invalidate the cache on subsequent invocations.
See [this discussion](https://github.com/dimo414/bkt/discussions/29) for a more
complete example of using `bkt` with `fzf`, including warming the commands before
the user starts navigating the selector.
### Using `bkt` only if installed
You may want to distribute shell scripts that utilize `bkt` without requiring
every user also install `bkt`. By wrapping `bkt` in a shell function your script
can cleanly invoke `bkt` if available without complicating your users' workflow.
Of course if they choose to install `bkt` they'll get a faster script as a
result!
```shell
# Cache commands using bkt if installed
if command -v bkt >&/dev/null; then
bkt() { command bkt "$@"; }
else
# If bkt isn't installed skip its arguments and just execute directly.
# Optionally write a msg to stderr suggesting users install bkt.
bkt() {
while [[ "$1" == --* ]]; do shift; done
"$@"
}
fi
# Now you can call bkt (the function) just like you'd call bkt (the binary):
bkt --ttl=1m -- expensive_cmd ...
```
### Decorating commands with `bkt` in shell scripts
It is sometimes helpful to cache _all_ invocations of a command in a shell
script or in your shell environment. You can use a decorator function pattern
similar to what bash-cache does to enable caching transparently, like so:
```shell
# This is Bash syntax, but other shells support similar syntax
expensive_cmd() {
bkt [bkt args ...] -- expensive_cmd "$@"
}
```
Calls to `expensive_cmd` in your shell will now go through `bkt` behind the
scenes. This can be useful for brevity and consistency but obviously changing
behavior like this is a double-edged-sword, so use with caution. Should you
need to bypass the cache for a single invocation Bash provides the
[`command` builtin](https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html#index-command),
so `command expensive_cmd ...` will invoke `expensive_cmd` directly. Other
shells provide similar features.