{"id":13587033,"url":"https://github.com/dimo414/bkt","last_synced_at":"2025-04-04T16:16:44.538Z","repository":{"id":40793638,"uuid":"335253430","full_name":"dimo414/bkt","owner":"dimo414","description":"a subprocess caching utility, available as a command line binary and a Rust library.","archived":false,"fork":false,"pushed_at":"2024-01-25T17:38:52.000Z","size":213,"stargazers_count":209,"open_issues_count":11,"forks_count":13,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-14T12:21:49.005Z","etag":null,"topics":["bash","cache","caching","cli","fzf","memoization","rust","shell","subprocess"],"latest_commit_sha":null,"homepage":"https://www.bkt.rs","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dimo414.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-02-02T10:38:14.000Z","updated_at":"2024-04-10T07:31:29.000Z","dependencies_parsed_at":"2024-01-12T22:32:49.690Z","dependency_job_id":"c0e65e97-93fd-4a21-8cc2-01116a93dd31","html_url":"https://github.com/dimo414/bkt","commit_stats":{"total_commits":101,"total_committers":4,"mean_commits":25.25,"dds":0.04950495049504955,"last_synced_commit":"e52bd6f92f54a8ef5266cf39bfe2e12e68c64e2d"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimo414%2Fbkt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimo414%2Fbkt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimo414%2Fbkt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimo414%2Fbkt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dimo414","download_url":"https://codeload.github.com/dimo414/bkt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208183,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","cache","caching","cli","fzf","memoization","rust","shell","subprocess"],"created_at":"2024-08-01T15:05:58.718Z","updated_at":"2025-04-04T16:16:44.518Z","avatar_url":"https://github.com/dimo414.png","language":"Rust","readme":"# `bkt`\n\n[![releases](https://img.shields.io/github/v/release/dimo414/bkt?sort=semver\u0026logo=github)](https://github.com/dimo414/bkt/releases)\n[![crates.io](https://img.shields.io/crates/v/bkt?logo=rust)](https://crates.io/crates/bkt)\n[![docs.rs](https://img.shields.io/docsrs/bkt?label=docs.rs)](https://docs.rs/bkt)\n[![build status](https://img.shields.io/github/actions/workflow/status/dimo414/bkt/rust.yml?branch=master)](https://github.com/dimo414/bkt/actions)\n[![issues](https://img.shields.io/github/issues/dimo414/bkt)](https://github.com/dimo414/bkt/issues)\n[![license](https://img.shields.io/github/license/dimo414/bkt)](https://github.com/dimo414/bkt/blob/master/LICENSE)\n\n`bkt` (pronounced \"bucket\") is a subprocess caching utility written in Rust,\ninspired by [bash-cache](https://github.com/dimo414/bash-cache).\nWrapping expensive process invocations with `bkt` allows callers to reuse recent\ninvocations without complicating their application logic. This can be useful in\nshell prompts, interactive applications such as [`fzf`](#fzf), and long-running\nprograms that poll other processes.\n\n`bkt` is available as a standalone binary as well as a\n[Rust library](https://crates.io/crates/bkt). See https://docs.rs/bkt/ for\nlibrary documentation. This README covers the `bkt` binary.\n\n## Installation\n\nRun `cargo install bkt` to compile and install `bkt` locally. You will need to\n[install `cargo`](https://doc.rust-lang.org/cargo/getting-started/installation.html)\nif it's not already on your system.\n\nPre-compiled binaries for common platforms are attached to each\n[release](https://github.com/dimo414/bkt/releases) (starting with 0.5). Please\nopen an issue or send a PR if you would like releases to include binaries for\nadditional platforms.\n\nPackage manager support is being tracked\n[here](https://github.com/dimo414/bkt/issues/12); volunteers are welcome.\n\n[![Packaging status](https://repology.org/badge/vertical-allrepos/bkt.svg?columns=3)](https://repology.org/project/bkt/versions)\n\n## Usage\n\n```\nbkt --ttl=DURATION [--stale=DURATION] [--cwd] [--env=ENV ...] [--modtime=FILE ...] [--scope=SCOPE] [--discard-failures] [--warm|--force] -- \u003ccommand\u003e...\n```\n\n`bkt` is easy to start using - simply prefix the command you intend to cache\nwith `bkt --ttl=[some duration] --`, for example:\n\n```shell\n# Execute and cache an invocation of 'date +%s.%N'\n$ bkt --ttl=1m -- date +%s.%N\n1631992417.080884000\n\n# A subsequent invocation reuses the same cached output\n$ bkt --ttl=1m -- date +%s.%N\n1631992417.080884000\n```\n\nWhen `bkt` is passed a command it hasn't seen before (or recently) it executes\nthe command synchronously and caches its stdout, stderr, and exit code. Calling\n`bkt` again with the same command reads the data from the cache and outputs it\nas if the command had been run again.\n\n### Cache Lifespan\n\nTwo flags, `--ttl` and `--stale`, configure how long cached data is preserved.\nThe TTL (Time to Live) specifies how long cached data will be used. Once the\nTTL expires the cached data will be discarded and the backing command re-run.\nA TTL can also be configured by setting a `BKT_TTL` environment variable.\n\nWhen the data expires `bkt` has to re-execute the command synchronously, which\ncan introduce unexpected slowness. To avoid this, pass `--stale` with a shorter\nduration than the TTL. When the cached data is older than the stale threshold\nthis causes `bkt` to refresh the cache in the background while still promptly\nreturning the cached data.\n\nBoth flags (and `BKT_TTL`) accept duration strings such as `10s` or\n`1hour 30min`. The exact syntax is defined in the\n[humantime](https://docs.rs/humantime/2.1.0/humantime/fn.parse_duration.html)\nlibrary.\n\n### Execution Environment\n\nSome commands' behavior depends on more than just the command line arguments.\nIt's possible to adjust how `bkt` caches such commands so that unrelated\ninvocations are cached separately.\n\n#### Working Directory\n\nFor example, attempting to cache `pwd` will not work as expected by default:\n\n```shell\n$ $ bkt --ttl=1m -- pwd\n/tmp/foo\n\n$ cd ../bar\n\n# Cached output for 'pwd' is reused even though the directory has changed\n$ bkt --ttl=1m -- pwd\n/tmp/foo\n```\n\nTo have `bkt` key off the current working directory in addition to the command\nline arguments pass `--cwd`:\n\n```shell\n$ bkt --cwd --ttl=1m -- pwd\n/tmp/foo\n\n$ cd ../bar\n\n$ bkt --cwd --ttl=1m -- pwd\n/tmp/bar\n```\n\n#### Environment Variables\n\nSimilarly, to specify one or more environment variables as relevant for the\ncommand being cached use `--env`, such as `--env=LANG`. This flag can be\nprovided multiple times to key off additional variables. Invocations with\ndifferent values for any of the given variables will be cached separately.\n\n#### File Modifications\n\n`bkt` can also check the last-modified time of one or more files and include\nthis in the cache key using `--modtime`. For instance passing\n`--modtime=/etc/passwd` would cause the backing command to be re-executed any\ntime `/etc/passwd` is modified even if the TTL has not expired.\n\n### Refreshing Manually\n\nIt's also possible to trigger refreshes manually using `--force` or `--warm`.\nThe former behaves exactly as if the cached data was not found, executing the\nprocess and caching the result. This is useful if you know the cached data\nis no longer up-to-date, e.g. because something external changed.\n\nAlternatively, it can be useful to refresh the cache asynchronously, which\n`--warm` provides. This triggers a refresh in the background but immediately\nends the current process with no output. This is useful if you expect\nadditional invocations in the near future and want to ensure they get a cache\nhit. Note that until the warming process completes concurrent calls may still\nsee a cache miss and trigger their own invocation.\n\n### Setting a Cache Scope\n\nCached data is persisted to disk (but see [below](#cache_dir)), and is\navailable to any process that invokes `bkt`. Generally this is desirable, but\ncertain usages may want to isolate their invocations from other potential\nconcurrent calls.\n\nTo do so pass `--scope=...` with a sufficiently unique argument, such as a fixed\nlabel for the calling program, the current process ID, or a timestamp.\n\n```shell\n$ bkt --ttl=1m -- date +%s.%N\n1631992417.080884000\n\n# Changing the scope causes the command to be cached separately\n$ bkt --scope=foo --ttl=1m -- date +%s.%N\n1631992418.010562000\n```\n\nAlternatively, define a `BKT_SCOPE` environment variable to configure a\nconsistent scope across invocations. This can be useful within a script to\nensure all commands share a scope.\n\n```shell\n#!/bin/bash\n\n# Set a unique scope for this script invocation using the PID and current time\nexport BKT_SCOPE=\"my_script_$$_$(date -Ins)\"\n```\n\n### Discarding Failed Invocations\n\nBy default, all invocations are cached regardless of their output or exit code.\nIn situations where failures should not be cached pass `--discard-failures` to\nonly persist successful invocations (those that return a `0` exit code).\n\n**WARNING:** Passing this flag can cause the backing command to be invoked more\nfrequently than the `--ttl` would suggest, which in turn can create unexpected\nload. If the backing command is failing due to an outage or bug (such as an\noverloaded website) triggering additional calls can exacerbate the issue and\neffectively DDoS the hampered system. It is generally safer *not* to set this\nflag and instead make the client robust to occasional failures. \n\n\u003ca name=\"cache_dir\"\u003e\u003c/a\u003e\n### Changing the Cache Directory\n\nBy default, cached data is stored under your system's temporary directory\n(typically `/tmp` on Linux).\n\nYou may want to use a different location for certain commands, for instance to\nbe able to easily delete the cached data as soon as it's no longer needed. You\ncan specify a custom cache directory via the `--cache-dir` flag or by defining\na `BKT_CACHE_DIR` environment variable.\n\nNote that the choice of directory can affect `bkt`'s performance: if the cache\ndirectory is on a [`tmpfs`](https://en.wikipedia.org/wiki/Tmpfs) or solid-state\npartition it will be significantly faster than one using a spinning disk.\n\nIf your system's temporary directory is not a good choice for the default cache\nlocation (e.g. it is not a `tmpfs`) you can specify a different location by\ndefining a `BKT_TMPDIR` environment variable (for example in your `.bashrc`).\nThese two environment variables, `BKT_TMPDIR` and `BKT_CACHE_DIR`, have similar\neffects but `BKT_TMPDIR` should be used to configure the system-wide default,\nand `--cache-dir`/`BKT_CACHE_DIR` used to override it.\n\n`bkt` periodically prunes stale data from its cache, but it also assumes the\noperating system will empty its temporary storage from time to time (for `/tmp`\nthis typically happens on reboot). If you opt to use a directory that the\nsystem does not maintain, such as `~/.cache`, you may want to manually delete\nthe cache directory on occasion, such as when upgrading `bkt`.\n\n## Security and Privacy\n\nThe default cache directory is potentially world-readable. On Unix the cache\ndirectory is created with `700` permissions, meaning only the current user can\naccess it, but this is not foolproof.\n\nYou can customize the cache directory (see [above](#cache_dir)) to a location\nyou trust such as `~/.cache`, but note that your home directory may be slower than\nthe temporary directory selected by default.\n\nIn general, if you are not the only user of your system it's wise to configure\nyour `TMPDIR` to a location only you can access. If that is not possible use\n`BKT_TMPDIR` to configure a custom temporary directory specifically for `bkt`.\n\n## Patterns and Tips\n\n**Please share how you're using `bkt` on the\n[Discussion Board](https://github.com/dimo414/bkt/discussions/categories/show-and-tell)!**\n\n\u003ca name=\"fzf\"\u003e\u003c/a\u003e\n### Speeding up `fzf` and other preview tools\n\n`bkt` works well with interactive tools like\n[`fzf`](https://github.com/junegunn/fzf) that execute other commands. Because\n`fzf` executes the `--preview` command every time an element is selected it can\nbe slow and tedious to browse when the command takes a long time to run. Using\n`bkt` allows each selection's preview to be cached. Compare:\n\n```shell\n$ printf '%s\\n' 1 0.2 3 0.1 5 | \\\n  fzf --preview=\"bash -c 'sleep {}; echo {}'\"\n\n$ printf '%s\\n' 1 0.2 3 0.1 5 | \\\n  fzf --preview=\"bkt --ttl=10m --stale=10s -- bash -c 'sleep {}; echo {}'\"\n```\n\nYou'll generally want to use a long TTL and a short stale duration so that\neven if you leave `fzf` running for a while the cache remains warm and is\nrefreshed in the background. You may also want to set a `--scope` if it's\nimportant to invalidate the cache on subsequent invocations.\n\nSee [this discussion](https://github.com/dimo414/bkt/discussions/29) for a more\ncomplete example of using `bkt` with `fzf`, including warming the commands before\nthe user starts navigating the selector.\n\n### Using `bkt` only if installed\n\nYou may want to distribute shell scripts that utilize `bkt` without requiring\nevery user also install it. By falling back to a no-op shell function when `bkt`\nis not available your script can take advantage of it opportunistically without\ncomplicating your users' workflow. Of course if they choose to install `bkt`\nthey'll get a faster script as a result!\n\n```shell\n# Cache commands using bkt if installed\nif ! command -v bkt \u003e\u0026/dev/null; then\n  # If bkt isn't installed skip its arguments and just execute directly.\n  bkt() {\n    while [[ \"$1\" == --* ]]; do shift; done\n    \"$@\"\n  }\n  # Optionally, write a msg to stderr suggesting users install bkt.\n  echo \"Tip: install https://github.com/dimo414/bkt for faster performance\" \u003e\u00262\nfi\n```\n\n### Decorating commands with `bkt` in shell scripts\n\nIt is sometimes helpful to cache _all_ invocations of a command in a shell\nscript or in your shell environment. You can use a decorator function pattern\nsimilar to what bash-cache does to enable caching transparently, like so:\n\n```shell\n# This is Bash syntax, but other shells support similar syntax\nexpensive_cmd() {\n  bkt [bkt args ...] -- expensive_cmd \"$@\"\n}\n```\n\nCalls to `expensive_cmd` in your shell will now go through `bkt` behind the\nscenes. This can be useful for brevity and consistency but obviously changing\nbehavior like this is a double-edged-sword, so use with caution. Should you\nneed to bypass the cache for a single invocation Bash provides the\n[`command` builtin](https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html#index-command),\nso `command expensive_cmd ...` will invoke `expensive_cmd` directly. Other\nshells provide similar features.\n","funding_links":[],"categories":["Rust"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimo414%2Fbkt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdimo414%2Fbkt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimo414%2Fbkt/lists"}