{"id":22752610,"url":"https://github.com/pwinckles/rocfl","last_synced_at":"2025-04-14T14:21:44.682Z","repository":{"id":43472712,"uuid":"279862520","full_name":"pwinckles/rocfl","owner":"pwinckles","description":"A CLI for OCFL repositories","archived":false,"fork":false,"pushed_at":"2022-11-28T01:29:16.000Z","size":2759,"stargazers_count":18,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-28T03:22:32.451Z","etag":null,"topics":["command-line-tool","ocfl","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pwinckles.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-15T12:29:19.000Z","updated_at":"2025-03-13T16:19:38.000Z","dependencies_parsed_at":"2023-01-21T15:16:53.280Z","dependency_job_id":null,"html_url":"https://github.com/pwinckles/rocfl","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwinckles%2Frocfl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwinckles%2Frocfl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwinckles%2Frocfl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwinckles%2Frocfl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pwinckles","download_url":"https://codeload.github.com/pwinckles/rocfl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248894948,"owners_count":21179154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","ocfl","rust"],"created_at":"2024-12-11T05:13:06.882Z","updated_at":"2025-04-14T14:21:43.809Z","avatar_url":"https://github.com/pwinckles.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rocfl\n\n![build](https://github.com/pwinckles/rocfl/workflows/build/badge.svg)\n\n`rocfl` is a command line utility for interacting with\n[OCFL](https://ocfl.io/) repositories on the local filesystem or in\nS3. Its goal is to provide a logical view of OCFL objects and make\nthem easy to interact with in a unix-like way.\n\n`rocfl` supports OCFL spec versions 1.0 and 1.1.\n\n![gif](./media/demo.gif)\n\n## Extension Support\n\n`rocfl` supports the following extensions:\n\n- [0001-digest-algorithms](https://ocfl.github.io/extensions/0001-digest-algorithms.html)\n- [0002-flat-direct-storage-layout](https://ocfl.github.io/extensions/0002-flat-direct-storage-layout.html)\n- [0003-hash-and-id-n-tuple-storage-layout](https://ocfl.github.io/extensions/0003-hash-and-id-n-tuple-storage-layout.html)\n- [0004-hashed-n-tuple-storage-layout](https://ocfl.github.io/extensions/0004-hashed-n-tuple-storage-layout.html)\n- [0005-mutable-head](https://ocfl.github.io/extensions/0005-mutable-head.html):\n  Only read is supported; not write.\n- [0006-flat-omit-prefix-storage-layout](https://ocfl.github.io/extensions/0006-flat-omit-prefix-storage-layout.html)\n- [0007-n-tuple-omit-prefix-storage-layout](https://ocfl.github.io/extensions/0007-n-tuple-omit-prefix-storage-layout.html)\n\nAdditionally, it uses the following extensions for write support that\nhave not been specified:\n\n- `rocfl-staging`: By default, new object versions are staged in this\n  extension's directory the contents of which are OCFL repository like.\n- `rocfl-locks`: This extension contains object file locks that provide\n  limited concurrent modification protection.\n\n## Install\n\nThe [releases page](https://github.com/pwinckles/rocfl/releases) has\npre-built binaries that should work on most common OSes and\narchitectures. You do _not_ need to install Rust to use them.\n\n1. Download and unzip the appropriate binary\n2. Execute `./rocfl help` to verify it works\n\n### Local Build\n\nAlternatively, you can build `rocfl` from source as follows:\n\n1. Install [Rust](https://www.rust-lang.org/tools/install), and make\n   sure `cargo` is on your `PATH`\n2. Execute: `cargo install rocfl`\n3. Verify the install: `rocfl help`\n\nIf you want to build a binary that does not include the S3\nintegration, which adds a large number of dependencies, then you can\ndo so by running: `cargo install rocfl --no-default-features`.\n\n## Configuration\n\n`rocfl` supports optional configuration that makes it less verbose to\nuse. `rocfl` expects to find its configuration file in the following,\nOS dependent location:\n\n- **Linux**: `$HOME/.config/rocfl/config.toml`\n- **Mac**: `$HOME/Library/Application\n  Support/org.rocfl.rocfl/config.toml`\n- **Windows**:\n  `{FOLDERID_RoamingAppData}/rocfl/rocfl/config/config.toml`\n  \nYou can easily edit the configuration by executing `rocfl config`,\nwhich will create/open the config file for editing.\n\nThe config file may contain any number of sections structured as\nfollows:\n\n``` toml\n[repo-name]\n# The name to attribute new OCFL versions to\nauthor_name = \"My Name\"\n# The URI address to associate with the above name\nauthor_address = \"mailto:name@example.com\"\n# The absolute path to the OCFL storage root\nroot = \"/path/to/storage/root\"\n# The absolute path to the OCFL staging root\nstaging_root = \"/path/to/staging/root\"\n\n# The next properties only apply when using S3\n# The AWS region your bucket is in\nregion = \"aws-region\"\n# The AWS credentials profile to load credentials from. If not specified, \"default\" is used.\nprofile = \"aws-credentials-profile\"\n# The URL to the S3 endpoint. This is only needed if you are using a non-standard region\nendpoint = \"https://s3-endpoint\"\n# The S3 bucket the OCFL repository is in\nbucket = \"s3-bucket\"\n```\n\n`repo-name` is the arbitrary name assigned to the configuration. This\nis the value that you pass `rocfl` when invoking the `--name` option.\n\nA special `[global]` section may be used to provide default values\nacross all configurations. This is primarily useful for defining\n`author_name` and `author_address`.\n\nAll of these properties correspond to values that can be specified in\narguments to `rocfl`. `rocfl` resolves the configuration by first\nloading the `global` config, then overlays the repository specific\nconfig, and finally applies any values specified directly as command\nline arguments.\n\nRefer to the command line argument documentation for more information\non the configuration properties.\n\n## Usage\n\nThe following is an overview of the features that `rocfl` supports.\nFor a detailed description of all of the options available, consult\nthe builtin help by executing `rocfl help` or `rocfl help \u003cCOMMAND\u003e`.\n\n### Read Commands\n\n#### Validate\n\nThe `validate` command validates either an entire OCFL repository or\nspecific objects within the repository. Any issues that are identified\nare reported with their corresponding [validation\ncode](https://ocfl.io/1.0/spec/validation-codes.html).\n\n##### Examples\n\nValidate an entire repository:\n\n``` console\nrocfl validate\n```\n\nValidate an entire repository without performing content fixity checks\nand only reporting errors:\n\n``` console\nrocfl validate -n -l Error\n```\n\nValidate an entire repository and suppress warning `W004`:\n\n``` console\nrocfl validate -w W004\n```\n\nValidate a specific object:\n\n``` console\nrocfl validate urn:example:rocfl:object-1\n```\n\nValidate multiple objects at specific paths relative the repository\nroot:\n\n``` console\nrocfl validate -p object-1/ object-2/\n```\n\n#### List\n\nThe `ls` command either lists all of the objects in a repository or\nlist the files in an OCFL object.\n\n`ls` will list objects as soon as they're found so long as it does not\nneed to sort the objects or display them in a formatted table. Object\nlists are not sorted by default, and the formatted table can be\ndisabled with `-t`.\n\n`rocfl` must scan the repository to locate objects. This can be slow\nwhen operating on large repositories. The scan can be avoided when\nlisting an object's contents if the repository uses a supported\nstorage layout extension that is defined in the repository's\n`ocfl_layout.json`\n\nWhen listing files, only files in the most recent version are\nreturned. Previous versions can be queried with the `-v` option.\n\n##### Examples\n\n###### Listing Objects\n\nThe following command lists all of the object IDs in a repository\nthat's rooted in the current working directory:\n\n```console\nrocfl ls\n```\n\nThis lists the same objects but with additional details, current\nversion and updated date:\n\n```console\nrocfl ls -l\n```\n\nAdding the `-p` flag additionally provides the path from the storage\nroot to the object:\n\n```console\nrocfl ls -lp\n```\n\nA subset of objects can be listed by providing a glob pattern to match\non:\n\n```console\nrocfl ls -lo '*object*'\n```\n\n###### Listing Object Contents\n\nThe contents of an object's current state are displayed by invoking\n`ls` on a specific object ID:\n\n```console\nrocfl ls urn:example:rocfl:object-1\n```\n\nWith the `-l` flag, additional details are displayed. In this case,\nthe version and date indicate when the individual file was last\nupdated:\n\n```console\nrocfl ls -l urn:example:rocfl:object-1\n```\n\nThe `-p` flag can also be used here to display the paths to the\nphysical files on disk relative the storage root:\n\n```console\nrocfl ls -p urn:example:rocfl:object-1\n```\n\nThe contents of previous versions are displayed by using the `-v`\noption. The following command displays the files that were in the\nfirst version of the object:\n\n```console\nrocfl ls -v1 urn:example:rocfl:object-1\n```\n\nAn object's contents can be filtered by specifying a glob pattern to\nmatch on:\n\n```console\nrocfl ls urn:example:rocfl:object-1 '*.txt'\n```\n\nThe output is sorted by name by default, but can also be sorted\nversion or updated date:\n\n```console\nrocfl ls -l -s version urn:example:rocfl:object-1\n```\n\nPaths within in an object can be interpreted as containing logical\ndirectories by using the `-D` flag. For example, the following will\nlist the logical files and logical directories that are direct\nchildren of the logical directory `sub/dir`:\n\n``` console\nrocfl ls -D urn:example:rocfl:object-1 sub/dir\n```\n\n#### Log\n\nThe `log` command displays the version metadata for all versions of an\nobject. It can also be executed on a file within an object, in which\ncase only versions that affected the specified file are displayed.\n\n##### Examples\n\nShow all of the versions of an object in ascending order:\n\n```console\nrocfl log urn:example:rocfl:object-1\n```\n\nOnly display the five most recent versions:\n\n```console\nrocfl log -rn5 urn:example:rocfl:object-1\n```\n\nShow all of the versions, but formatted so each version is on a single\nline:\n\n```console\nrocfl log -c urn:example:rocfl:object-1\n```\n\nShow all of the versions that affected a specific file:\n\n```console\nrocfl log urn:example:rocfl:object-1 file1.txt\n```\n\n#### Show\n\nThe `show` command displays everything that changed in an object\nwithin a specific version. If no version is specified, the most recent\nchanges are shown.\n\n##### Examples\n\nShow the changes in the most recent version:\n\n```console\nrocfl show urn:example:rocfl:object-1\n```\n\nShow the changes in the first version:\n\n```console\nrocfl show urn:example:rocfl:object-1 v1\n```\n\nDon't show the version metadata; only show the files that changed:\n\n```console\nrocfl show -m urn:example:rocfl:object-1\n```\n\n#### Diff\n\nThe `diff` command displays the files that changed between two\nspecific versions.\n\n##### Example\n\nShow the changes between the second and fourth versions:\n\n```console\nrocfl diff v2 v4\n```\n\n#### Cat\n\nThe `cat` command writes the contents of a file to `stdout`.\n\n##### Examples\n\nDisplay the contents of the head version of a file:\n\n```console\nrocfl cat urn:example:rocfl:object-1 file1.txt\n```\n\nDisplay the contents of a file from a specific version of the object:\n\n```console\nrocfl cat -v1 urn:example:rocfl:object-1 file1.txt\n```\n\n#### Status\n\nThe `status` command shows objects that have staged changes pending\ncommit, as well as what an object's pending changes are.\n\n##### Examples\n\nList all of the objects with staged changes:\n\n``` console\nrocfl status\n```\n\nList all of the file level changes to an object:\n\n``` console\nrocfl status urn:example:rocfl:object-1\n```\n\nStaged changes can also be examined using the more featureful `ls`,\n`show`, and `cat` commands by adding the `-S` flag.\n\n#### Info\n\nThe `info` command displays information, such as spec version and\nconfigured extensions, for repositories and objects.\n\n##### Examples\n\nShow info for a repository:\n\n``` console\nrocfl info\n```\n\nShow info for an object:\n\n``` console\nrocfl info urn:example:rocfl:object-1\n```\n\n### Write Commands\n\n`rocfl` supports updating OCFL objects by staging changes to objects\nin a local staging repository. For filesystem based repositories, the\nstaging repository is located within the storage root's extensions\ndirectory, and for S3 it's in the user's home application data.\n\nThe intended workflow is to accumulate a collection of updates to an\nobject, and then commit all of the changes to the object a single new\nOCFL version.\n\n#### Init\n\nThe `init` command creates new OCFL repositories. By default, the\nlatest OCFL spec version is used along with the\n[0004-hashed-n-tuple-storage-layout](https://ocfl.github.io/extensions/0004-hashed-n-tuple-storage-layout.html)\nstorage layout.\n\n##### Examples\n\nCreate a new repository using the default layout,\n[0004-hashed-n-tuple-storage-layout](https://ocfl.github.io/extensions/0004-hashed-n-tuple-storage-layout.html):\n\n``` console\nrocfl -r /var/tmp/ocfl-repo-1 init\n```\n\nThe default layout configuration can be changed by passing a config\nfile that contains the desired configuration:\n\n``` console\nrocfl -r /var/tmp/ocfl-repo-2 init -l 0003-hash-and-id-n-tuple-storage-layout -c\nmy-config.json\n```\n\n#### New\n\nThe `new` command stages new OCFL objects. New objects will not exist\nin the main repository until they have been committed. When invoked\nwith no options, the object is created using all of the OCFL spec\nrecommend values.\n\n##### Examples\n\nCreate a new object with non-standard settings:\n\n``` console\nrocfl new urn:example:rocfl:object-1 -d sha256 -c data -z 6\n```\n\n#### Copy\n\nThe `cp` command copies files from the local filesystem into a\nstaged object, or copies logical paths within an object to a new\nlocation within the same object. This command attempts to mimic the\nbehavior of GNU `cp` as closely as possible.\n\n##### Examples\n\nCopy a directory into the object's root:\n\n``` console\nrocfl cp -r urn:example:rocfl:object-1 /path/to/src -- /\n```\n\nCopy several files into a logical directory within the object:\n\n``` console\nrocfl cp urn:example:rocfl:object-1 /path/to/files/* -- sub/dir\n```\n\nCopy several existing files internally to a new location:\n\n``` console\nrocfl cp -i urn:example:rocfl:object-1 'internal/*.txt' -- new-location\n```\n\nCopy an entire logical directory from an old version to a new location\nin the staged version:\n\n``` console\nrocfl cp -ir -v2 urn:example:rocfl:object-1 src/dir -- dst/dir\n```\n\n#### Move\n\nThe `mv` command moves files from the local filesystem into a staged\nobject, or moves logical paths within an object to a new location\nwithin the same object. This command attempts to mimic the behavior of\nGNU `mv` as closely as possible.\n\n##### Examples\n\nMove a directory into the object's root:\n\n``` console\nrocfl mv urn:example:rocfl:object-1 /path/to/src -- /\n```\n\nMove several files into a logical directory within the object:\n\n``` console\nrocfl mv urn:example:rocfl:object-1 /path/to/files/* -- sub/dir\n```\n\nMove an existing file internally to a new location:\n\n``` console\nrocfl mv -i urn:example:rocfl:object-1 internal/file.txt -- new/location.txt\n```\n\n#### Remove\n\nThe `rm` command removes files from an object. If the removed files\nwere new to the staged version, then they are permanently removed and\nwill not appear anywhere in the object. Otherwise, references to the\nfiles are removed from the staged version, but the files still exist\nin prior versions.\n\n##### Examples\n\nRecursively remove a logical directory:\n\n``` console\nrocfl rm -r urn:example:rocfl:object-1 path/to/dir\n```\n\nRemove several individual files:\n\n``` console\nrocfl rm urn:example:rocfl:object-1 path/to/file1.txt path/to/file2.txt\n```\n\nOr with a glob:\n\n``` console\nrocfl rm urn:example:rocfl:object-1 'path/to/*.txt'\n```\n\n#### Reset\n\nThe `reset` command unstages changes made to an object. Additions are\nremoved and files that were deleted or modified are reverted to their\nprevious state.\n\n##### Examples\n\nReset a file to its previous state:\n\n``` console\nrocfl reset urn:example:rocfl:object-1 file.txt\n```\n\nReset an entire object to its previous state, removing all staged\nchanges:\n\n``` console\nrocfl reset urn:example:rocfl:object-1\n```\n\n#### Commit\n\nThe `commit` command moves an object's staged changes into the OCFL\nobject as a new version.\n\n##### Examples\n\nCommit changes to an object:\n\n``` console\nrocfl commit urn:example:rocfl:object-1 -n \"My Name\" -a \"mailto:me@example.com\" -m \"commit\nmessage\"\n```\n\nThis can be simplified if you define your name and address in [rocfl\nconfiguration](#configuration). In which case, you can simply execute:\n\n``` console\nrocfl commit urn:example:rocfl:object-1 -m \"commit message\"\n```\n\nAnd your name and address will be automatically added to the version\nmetadata.\n\nIn order to commit an object to a repository without a defined storage\nlayout, the location to store the object with the repository must be\nmanually specified as follows:\n\n``` console\nrocfl commit urn:example:rocfl:object-1 -m \"commit message\" -r relative/path/to/object/root\n```\n\n#### Purge\n\nThe `purge` command permanently removes an object from the main OCFL\nrepository. This is **not** an operation that stages changes. However,\nit will ask for confirmation before deleting an object.\n\n#### Upgrade\n\nThe `upgrade` command upgrades an object or repository to a later OCFL\nspec version. OCFL objects are not automatically upgraded when a\nrepository is upgraded, and must be upgraded individually. Upgrading\nan OCFL object requires creating a new object version, and any changes\nthat are staged for an object are included in the upgrade version.\n\n##### Examples\n\nUpgrade a 1.0 repository to spec version 1.1:\n\n``` console\nrocfl upgrade -v 1.1\n```\n\nUpgrade an object to spec version 1.1:\n\n``` console\nrocfl upgrade urn:example:rocfl:object-1 -n \"My Name\" -a \"mailto:me@example.com\" -m \"upgrade to 1.1\"\n```\n\n## S3\n\n### S3 Configuration\n\nTo connect to an OCFL repository in S3, you first need to create an\nIAM user with access to the S3 bucket, and then setup a local\n`~/.aws/credentials` file or environment variables as [described\nhere](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).\nCredential profiles can be specified using the `--profile` option.\nThen, when you invoke `rocfl` you must specify the bucket the\nrepository is in as well as the bucket region. For example:\n\n```console\nrocfl -R us-east-2 -b example-ocfl-repo ls\n```\n\nYou can specify a sub directory, or prefix, that the repository is\nrooted in within the bucket like this:\n\n```console\nrocfl -R us-east-2 -b example-ocfl-repo -r ocfl-root ls\n```\n\nChanges to objects are staged locally and are only pushed to S3 when\nthe staged version is committed. By default, changes are staged in the\nfollowing location:\n\n- **Linux**: `$HOME/.local/share/rocfl/s3/staging/\u003csha256:BUCKET/ROOT\u003e`\n- **Mac**: `{FOLDERID_RoamingAppData}/org.rocfl.rocfl/data/s3/staging/\u003csha256:BUCKET/ROOT\u003e`\n- **Windows**: `$HOME/Library/Application Support/rocfl/rocfl/s3/staging/\u003csha256:BUCKET/ROOT\u003e`\n\nThis location can be changed by setting the `--staging-root` option.\n\nAll of these properties can define defined in `rocfl`'s [config\nfile](#configuration), and activated by invoking `rocfl` using the\nconfiguration `NAME` as follows:\n\n``` console\nrocfl -n NAME ls\n```\n\n### Important S3 Considerations\n\nWhile `rocfl` supports all of the same operations on S3 as it does the\nlocal filesystem, this does come with a couple of caveats:\n\n1. Scanning for objects in S3 is very slow, and using a defined storage\n   layout extension is key to improving performance.\n2. `rocfl` does not provide any strong concurrency guarantees when\n   modifying objects in S3. A file lock is used to guard changes in\n   the staging repository, but there is an unchecked race condition if\n   multiple processes attempt to commit changes to the same object\n   from different staging locations.\n\n## Roadmap\n\nThe following features are planned:\n\n1. Export objects to a location outside the repository\n2. Index objects when the storage layout is unknown\n3. Unsafe mutating operations such as `squash`, `revert`, and\n   `rewrite` that can be used to change an object's history\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpwinckles%2Frocfl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpwinckles%2Frocfl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpwinckles%2Frocfl/lists"}