{"id":15593440,"url":"https://github.com/dmolesuc/cos","last_synced_at":"2026-01-27T10:09:23.252Z","repository":{"id":46803662,"uuid":"161562723","full_name":"dmolesUC/cos","owner":"dmolesUC","description":"A tool for testing and validating cloud object storage","archived":false,"fork":false,"pushed_at":"2023-10-22T13:51:30.000Z","size":10910,"stargazers_count":2,"open_issues_count":12,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-19T22:46:10.514Z","etag":null,"topics":["minio","openstack-swift","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmolesUC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-13T00:40:33.000Z","updated_at":"2022-07-06T18:30:27.000Z","dependencies_parsed_at":"2022-09-26T18:20:33.294Z","dependency_job_id":"cf1ca98e-1b45-476b-bb58-42e3c147a037","html_url":"https://github.com/dmolesUC/cos","commit_stats":{"total_commits":267,"total_committers":3,"mean_commits":89.0,"dds":0.02621722846441943,"last_synced_commit":"63dfaba08da2ce1ce075cc91cd48185ae1f17efa"},"previous_names":["dmolesuc3/cos"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmolesUC%2Fcos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmolesUC%2Fcos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmolesUC%2Fcos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmolesUC%2Fcos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmolesUC","download_url":"https://codeload.github.com/dmolesUC/cos/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239634334,"owners_count":19672183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["minio","openstack-swift","s3"],"created_at":"2024-10-03T00:18:32.727Z","updated_at":"2025-11-24T18:30:15.864Z","avatar_url":"https://github.com/dmolesUC.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Docker Repository on Quay](https://quay.io/repository/denistrofimov/cos/status \"Docker Repository on Quay\")](https://quay.io/repository/denistrofimov/cos)\n\n# cos\n\nA tool for testing and validating cloud object storage.\n\n- [Invocation](#invocation)\n- [Authentication](#authentication)\n- [Commands](#commands)\n   - [cos check](#cos-check)\n   - [cos crvd](#cos-crvd)\n   - [cos keys](#cos-keys)\n   - [cos suite](#cos-suite)\n- [For developers](#for-developers)\n   - [Building](#building)\n   - [Running tests](#running-tests)\n   - [Configuring JetBrains IDEs (GoLand or IDEA)](#configuring-jetbrains-ides-goland-or-idea)\n- [Roadmap](#roadmap)\n\n## Invocation\n\nInvocation is in the form\n\n```\ncos \u003ccommand\u003e [flags] [URL]\n```\n\nwhere `\u003ccommand\u003e` is one of:\n\n- [`check`](https://github.com/dmolesUC3/cos#cos-check): \n  compute and (optionally) verify the digest of an object\n- [`crvd`](https://github.com/dmolesUC3/cos#cos-crvd): \n  create, retrieve, verify, and delete an object\n- [`keys`](https://github.com/dmolesUC3/cos#cos-keys): \n  test the keys supported by an object storage endpoint\n- [`suite`](https://github.com/dmolesUC3/cos#cos-suite): \n  run a suite of test cases investigating various possible limitations of a\n  cloud storage service\n- `help`: \n  list these commands, or get help for a subcommand\n\nand `[URL]` can be the URL of an object or of a bucket/container, depending\non the context. The protocol (`s3://` or `swift://`) of the URL is used to\ndetermine the cloud storage API to use.\n\n## Authentication\n\nFor authentication, `cos` uses the same environment variables as the [AWS\nCLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)\n(for S3 and compatible storage) or [OpenStack Swift\nCLI](https://docs.openstack.org/python-swiftclient/latest/cli/index.html)\n(for Swift storage):\n\n| Protocol | Variable                | Purpose                                       |\n| :---     | :---                    | :---                                          |\n| S3       | `AWS_ACCESS_KEY_ID`     | AWS access key                                |\n|          | `AWS_SECRET_ACCESS_KEY` | Secret key associated with the AWS access key |\n| Swift    | `ST_USER`               | Swift username                                |\n|          | `ST_KEY`                | Swift password                                |\n\nCredentials for S3 storage can also be specified [in various other\nways](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials)\nsupported by the AWS SDK for Go, such as a shared credentials file or, when\nrunning in the Amazon EC2 environment, an IAM role.\n\n### Flags\n\nAll `cos` commands support the following flags:\n\n| Short form | Flag                  | Description                     |\n| :---       | :---                  | :---                            |\n| `-e`       | `--endpoint ENDPOINT` | HTTP(S) endpoint URL (required) |\n| `-r`       | `--region REGION`     | AWS region (optional)           |\n| `-v`       | `--verbose`           | Verbose output                  |\n| `-h`       | `--help`              | Print help and exit             |\n\nFor Amazon S3 buckets, the region can usually be determined from the\nendpoint URL. If not, and if the `--region` flag is not provided, it\ndefaults to `us-west-2`.\n\nFor OpenStack Swift containers, the `--region` flag is ignored.\n\nAdditional command-specific flags are listed below.\n\n## Commands\n\n### `cos check`\n\nThe `check` command computes and (optionally) verifies the digest of an\nobject. The object is streamed in five-megabyte chunks, each chunk being\nadded to the digest computation and then discarded, thus making it possible\nto verify objects of arbitary size, not limited by local storage space.\n\nIn addition to the global flags listed above, the `check` command supports the following:\n\n| Short form | Flag                | Description                                          |\n| :---       | :---                | :---                                                 |\n| `-a`       | `--algorithm ALG`   | Digest algorithm (md5 or sha256; defaults to sha256) |\n| `-x`       | `--expected DIGEST` | Expected digest value                                |\n\nBy default, `check` outputs the digest to standard output, and exits:\n\n```\n$ cos check --endpoint https://s3.us-west-2.amazonaws.com/ s3://www.dmoles.net/images/fa/archive.svg/\nc99ad299fa53d5d9688909164cf25b386b33bea8d4247310d80f615be29978f5\n```\n\nIf given an expected value that does not match, prints a message to standard\nerror, and exits with a nonzero (unsuccessful) exit code.\n\n```\n$ cos check --endpoint https://s3.us-west-2.amazonaws.com/ s3://www.dmoles.net/images/fa/archive.svg/ \\\n  -x 5f87992eb516f08d0137424d8aeb33b683b52fc4619098869d5d35af992da99c\ndigest mismatch: \nexpected: 5f87992eb516f08d0137424d8aeb33b683b52fc4619098869d5d35af992da99c\nactual: c99ad299fa53d5d9688909164cf25b386b33bea8d4247310d80f615be29978f5\n```\n\n### `cos crvd`\n\nThe `crvd` command creates, retrieves, verifies, and deletes an object.\nThe object consists of a stream of random bytes of the specified size.\n\nThe size may be specified as an exact number of bytes, or using human-readable\nquantities such as \"5K\" (4 KiB or 4096 bytes), \"3.5M\" (3.5 MiB or 3670016 bytes),\netc. The units supported are bytes (B), binary kilobytes (K, KB, KiB), \nbinary megabytes (M, MB, MiB), binary gigabytes (G, GB, GiB), and binary \nterabytes (T, TB, TiB). If no unit is specified, bytes are assumed.\n\nRandom bytes are generated using the Go default random number generator, with\na default seed of 0, for repeatability. An alternative seed can be specified\nwith the `--random-seed` flag.\n\nIn addition to the global flags listed above, the `check` command supports the following:\n\n| Short form | Flag                 | Description                                          |\n| :---       | :---                 | :---                                                 |\n| `-s`       | `--size SIZE`        | size of object to create (default 128 bytes)         |\n| `-k`       | `--key KEY`          | key to create (defaults to `cos-crvd-TIMESTAMP.bin`) |\n|            | `--random-seed SEED` | seed for random-number generator (default 1)         |\n|            | `--keep`             | keep object after verification (default false)       |\n\n```\n$ crvd swift://distrib.stage.9001.__c5e/ -e http://cloud.sdsc.edu/auth/v1.0 \n128B object created, retrieved, verified, and deleted (swift://distrib.stage.9001.__c5e/cos-crvd-1549324512.bin)\n```\n\n### `cos keys`\n\nThe `keys` command tests the keys supported by an object storage endpoint,\ncreating, retrieving, validating, and deleting a small object for each value\nin the specified key list. \n\nIn addition to the global flags listed above, the `keys` command supports\nthe following:\n\n| Short form | Flag             | Description                                    |\n| :---       | :---             | :---                                           |\n|            | `--raw`          | write keys in raw (unquoted) format            |\n| `-o`       | `--ok FILE`      | write successful (\"OK\") keys to specified file |\n| `-b`       | `--bad FILE`     | write failed (\"bad\") keys to specified file    |\n| `-l`       | `--list LIST`    | use the specified 'standard' list of keys      |\n| `-f`       | `--file FILE`    | read keys to be tested from the specified file |\n| `-s`       | `--sample COUNT` | sample size, or 0 for all keys                 |\n\n\nBy default, `keys` outputs only failed keys, to standard output, writing\neach key as a [quoted Go string literal](https://golang.org/pkg/strconv/#Quote).\n\n```\n$ cos keys s3://uc3-s3mrt5001-stg/ -e 'https://s3-us-west-2.amazonaws.com/' --list misc \n\"../leading-double-dot-path\"\n\"../../leading-multiple-double-dot-path\"\n\"trailing-double-dot-path/..\"\n(...etc.)\n```\n\nUse the `--raw` option to write the keys without quoting or escaping; note\nthat this may produce confusing results if any of the keys contain\nnewlines.\n\n```\n$ cos keys s3://uc3-s3mrt5001-stg/ -e 'https://s3-us-west-2.amazonaws.com/' --list misc --raw\n../leading-double-dot-path\n../../leading-multiple-double-dot-path\ntrailing-double-dot-path/..\ntrailing-multiple-double-dot-path/../..\n(...etc.)\n```\n\nUse the `--ok` option to write successful keys to a file, and the `--bad`\noption (or shell redirection) to write failed keys to a file instead of\nstdout.\n\n```\n$ cos keys s3://uc3-s3mrt5001-stg/ -e 'https://s3-us-west-2.amazonaws.com/' --list misc \\\n  --ok out/keys-ok.txt --bad out/keys-bad.txt\n```\n\nSeveral \"standard\" lists are provided (though these aren't very systematic;\nsee [#10](https://github.com/dmolesUC3/cos/issues/10)). Use the `--file`\noption to specify a file containing keys to test, one key per file,\nseparated by newlines (LF, `U+000A`, `\\n`).\n\n```\n$ cos keys s3://uc3-s3mrt5001-stg/ -e 'https://s3-us-west-2.amazonaws.com/' \\\n  --file my-keys.txt\n```\n\nUse the `--sample` option to check only a random sample from a large key list:\n\n```\n$ cos keys s3://uc3-s3mrt5001-stg/ -e 'https://s3-us-west-2.amazonaws.com/' \\\n  --file my-very-long-list-of-keys.txt \\\n  --sample 500\n```\n\n### `cos suite`\n\nThe `suite` command a suite of test cases investigating various possible limitations of a\ncloud storage service:\n\n- maximum file size (`--size`)\n- maximum number of files per key prefix (`--count`)\n- Unicode key support (`--unicode`)\n\nIf none of `--size`, `--count`, etc. is specified, all test cases are run.\n\nUnicode key support tests are further divided into:\n\n- Unicode category support (--unicode-categories)\n- Unicode script support (--unicode-scripts)\n- Unicode properties support (--unicode-properties)\n- Unicode emoji support (--unicode-emoji)\n        - invalid Unicode key support (--unicode-invalid)\n\nIf `--unicode` is specified, all of these are run.\n\nNote that there is considerable overlap between the characters in the\ncategory support, script support, and properties support tests.\n\nNote also that the `--unicode-invalid` test depends somewhat on the exact\nmechanisms used to generate key strings from bytes, and results with your\nown client code may differ.\n\nIn addition to the global flags listed above, the `keys` command supports\nthe following:\n\n| Short form | Flag                   | Description                                                            |\n| :---       | :---                   | :---                                                                   |\n| `-s`       | `--size`               | test file sizes                                                        |\n|            | `--size-max SIZE`      | max file size to create (default \"256G\")                               |\n| `-c`       | `--count`              | test file counts                                                       |\n|            | `--count-max COUNT`    | max number of files to create, or -1 for no limit (default 16777216)   |\n| `-u`       | `--unicode`            | test Unicode keys                                                      |\n|            | `--unicode-categories` | test Unicode categories                                                |\n|            | `--unicode-scripts`    | test Unicode scripts                                                   |\n|            | `--unicode-properties` | test Unicode properties                                                |\n|            | `--unicode-emoji`      | test Unicode emoji                                                     |\n|            | `--unicode-invalid`    | test invalid Unicode                                                   |\n| `-n`       | `--dry-run`            | dry run; list all tests that would be run, but don't make any requests |\n\nThe maximum size may be specified as an exact number of bytes, or using\nhuman-readable quantities such as \"5K\" (4 KiB or 4096 bytes), \"3.5M\" (3.5\nMiB or 3670016 bytes), etc. The units supported are bytes (B), binary\nkilobytes (K, KB, KiB), binary megabytes (M, MB, MiB), binary gigabytes (G,\nGB, GiB), and binary terabytes (T, TB, TiB). If no unit is specified, bytes\nare assumed.\n\n\n## For developers\n\n`cos` is a [Go 1.11 module](https://github.com/golang/go/wiki/Modules). \n\nAs such, it requires Go 1.11 or later, and should be cloned _outside_\n`$GOPATH/src`.\n\n### Building\n\nThe `cos` project can be built and installed simply with `go build` and `go\ninstall`, but it also supports [Mage](https://magefile.org).\n\nTo install the latest version of Mage:\n\n1. visit their [releases page](https://github.com/magefile/mage/releases),\n   download the appropriate binary, and place it in your `$PATH`, or\n2. from _outside_ this project directory (`go get` behaves differently when\n   run in the context of a module project), execute the following:\n\n   ```\n   go get -u -d github.com/magefile/mage \\\n   \u0026\u0026 cd $GOPATH/src/github.com/magefile/mage \\\n   \u0026\u0026 go run bootstrap.go\n   ```\n\n#### Mage tasks:\n\n| Tasks        | Purpose                                                          |\n| :---         | :---                                                             |\n| `build`      | builds a binary for the current platform                         |\n| `buildAll`   | builds a binary for each target platform                         |\n| `buildLinux` | builds a linux-amd64 binary (the most common cross-compile case) |\n| `clean`      | removes compiled binaries from the current working directory     |\n| `install`    | installs in $GOPATH/bin                                          |\n| `platforms`  | lists target platforms for buildAll                              |\n\nNote that `mage build` is a thin wrapper around `go build` and supports the\nsame environment variables, e.g. `$GOOS` and `$GOARCH`.\n\n### Running tests\n\nTo run all tests in all subpackages, from the project root, use `go test ./...`.\n\nTo run all tests in all subpackages with coverage and view a coverage report, use\n\n```\ngo test -coverprofile=coverage.out ./... \\\n\u0026\u0026 go tool cover -html=coverage.out\n```\n\n### Configuring JetBrains IDEs (GoLand or IDEA)\n\nIn **Preferences \u003e Go \u003e Go Modules (vgo)** (GoLand) or **Preferences \u003e\nLanguages \u0026 Frameworks Go \u003e Go Modules (vgo)** (IDEA + Go plugin) , check\n“Enable Go Modules (vgo) integration“. The “Vgo Executable” field should\ndefault to “Project SDK” (1.11.x).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmolesuc%2Fcos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmolesuc%2Fcos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmolesuc%2Fcos/lists"}