https://github.com/dpc/fs-dir-cache
A CLI tool for CIs and build scripts, making file system based caching easy and correct (locking, eviction, etc.)
https://github.com/dpc/fs-dir-cache
Last synced: 7 months ago
JSON representation
A CLI tool for CIs and build scripts, making file system based caching easy and correct (locking, eviction, etc.)
- Host: GitHub
- URL: https://github.com/dpc/fs-dir-cache
- Owner: dpc
- Created: 2023-08-25T20:49:58.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2025-06-05T21:05:00.000Z (12 months ago)
- Last Synced: 2025-10-07T12:01:57.472Z (8 months ago)
- Language: Rust
- Size: 85.9 KB
- Stars: 17
- Watchers: 1
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# FS Dir Cache
A CLI tool for CIs and build scripts, making file system based
caching easy and correct (locking, eviction, etc.)
When working on build systems / CIs it's often a requirement to
utilize some best effort caching, with locking and eviction.
Not exactly rocket science, but non-trivial enough to not want
to implement and maintain ad-hoc.
`fs-dir-cache` aims to be a simple to use utility from inside
other scripts and programs taking care of the details.
## Example
This is an example, where a CI runner can persist files between runs,
and it's used to to reuse build artifacts between builds that are likely
building the same build to speed everything up:
```bash
#!/usr/bin/env bash
set -euo pipefail
job_name="$1"
shift 1
if [ -z "$job_name" ]; then
>&2 "error: no job name"
exit 1
fi
export FS_DIR_CACHE_LOCK_TIMEOUT_SECS="$((60 * 30))" # unlock after timeout in case our job fails misereably and/or hangs
export FS_DIR_CACHE_ROOT="$HOME/.cache/fs-dir-cache" # directory to hold all cache (sub)directories
export FS_DIR_CACHE_LOCK_ID="pid-$$-rnd-$RANDOM" # acquire lock based on the current pid and something random (just in case pid gets reused)
export FS_DIR_CACHE_KEY_NAME="$job_name" # the base name of our key
log_file="$FS_DIR_CACHE_ROOT/log"
fs-dir-cache gc unused --seconds "$((5 * 24 * 60 * 60))" # delete caches not used in more than a 5 days
export log_file # log when each job starte and ended
export job_name
src_dir=$(pwd)
export src_dir
# This bash command will be executed with a CWD set to the allocated directory
function run_in_cache() {
echo "$(date --rfc-3339=seconds) RUN job=$job_name dir=$(pwd)" >> "$log_file"
>&2 echo "$(date --rfc-3339=seconds) RUN job=$job_name dir=$(pwd)"
CARGO_BUILD_TARGET_DIR="$(pwd)"
export CARGO_BUILD_TARGET_DIR
cd "$src_dir"
function on_exit() {
local exit_code=$?
echo "$(date --rfc-3339=seconds) END job=$job_name code=$exit_code" >> "$log_file"
>&2 echo "$(date --rfc-3339=seconds) END job=$job_name code=$exit_code"
exit $exit_code
}
trap on_exit EXIT
"$@"
}
export -f run_in_cache
fs-dir-cache exec \
--key-file Cargo.lock
--key-str "${CARGO_PROFILE-:dev}" \
--key-file flake.lock \
-- \
bash -c 'run_in_cache "$@"' _ "$@"
```
Using just one tool, it's easy to get correct and practical caching including:
* locking (including fallback timeouts)
* evicition
* timeouts