{"id":19030222,"url":"https://github.com/guardian/matrixstore-client","last_synced_at":"2026-05-02T15:30:19.352Z","repository":{"id":39970305,"uuid":"377111142","full_name":"guardian/matrixstore-client","owner":"guardian","description":"A simple commandline client for accessing files on a MatrixStore device","archived":false,"fork":false,"pushed_at":"2024-01-20T06:21:13.000Z","size":1391,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-02-13T16:56:41.107Z","etag":null,"topics":["matrixstore","objectstorage"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guardian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-15T09:41:51.000Z","updated_at":"2021-11-12T14:28:11.000Z","dependencies_parsed_at":"2022-09-14T04:20:18.308Z","dependency_job_id":null,"html_url":"https://github.com/guardian/matrixstore-client","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fmatrixstore-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fmatrixstore-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fmatrixstore-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guardian%2Fmatrixstore-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guardian","download_url":"https://codeload.github.com/guardian/matrixstore-client/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240079638,"owners_count":19744725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["matrixstore","objectstorage"],"created_at":"2024-11-08T21:16:57.887Z","updated_at":"2026-05-02T15:30:19.277Z","avatar_url":"https://github.com/guardian.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# matrixstore-client\n\nA simple commandline client for accessing files on a MatrixStore device\n\n# Prerequisites\n\nYou'll need Java v1.8 installed. Note that _later_ Java is NOT supported, owing to trouble with the third-party libraries.\nOn a Mac, you can check your Java version like so:\n```\n/usr/libexec/java_home -V\nMatching Java Virtual Machines (5):\n    15.0.2, x86_64:\t\"OpenJDK 15.0.2\"\t/Library/Java/JavaVirtualMachines/openjdk.jdk/Contents/Home\n    11.0.11, x86_64:\t\"OpenJDK 11.0.11\"\t/Users/myself/Library/Java/JavaVirtualMachines/adopt-openjdk-11.0.11/Contents/Home\n    1.8.0_282, x86_64:\t\"OpenJDK 8\"\t/Library/Java/JavaVirtualMachines/openjdk-8.jdk/Contents/Home\n    1.6.0_65-b14-468, x86_64:\t\"Java SE 6\"\t/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home\n    1.6.0_65-b14-468, i386:\t\"Java SE 6\"\t/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home\n\n/Library/Java/JavaVirtualMachines/openjdk.jdk/Contents/Home\n```\n\nYou will see that my default is currently 15. To change this, run\n```bash\ndeclare -x JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-8.jdk/Contents/Home\n```\n(get the right path from the output of java_home)\n\n\n## How to get it\n\nGo to the Release page on github, find the release you want and download the JAR file.\nSave it onto your local machine then open a Terminal (or your Windows equivalent) and go to that location\n\nIn order to run from a built JAR, either build a JAR as above or download from Github releases then:\n```\njava -jar {path-to-downloaded-jar}\n```\n\n## How to use it\n\n**How do I download a file from Matrixstore?** - read through to the bottom of this document and don't skip :-p\n\nTL;DR: \n1. Log into the vault\n2. Use the `search` or `lookup` commands to find what you are interested in\n3. Find the OID of the file. This is the long thing that looks like a UUID with another part at the end\n4. Run `get` then the OID. The file will be downloaded to the current working directory of your machine, with a filename taken from the MXFS_FILE field.\n\n\nThe first thing you'll need to do is to make a connection to a cluster.  You\ndo this with the `connect` command:\n\n```\n\u003e connect {ip-address-list} {vault-id} {user-name}\nConnecting to ... on ... as ... \nPassword \u003e \n{vault-id}\u003e \n```\n\n- `{ip-address-list}` can be either the IP address of a single cluster node or a comma-\nseparated list of multiple nodes.\n- `{vault-id}` is the UUID of the vault you want to connect to\n- `{user-name}` is the user name to use when logging in. You are then prompted\nfor a password, which you type at the prompt and is hidden behind * characters.\n  \nOnce you have a connection, the prompt changes to show the vault id behind the '\u003e' \ncharacter.\n\n### Credentials\n\nmatrixstore-client expects MatrixStore 4 credentials, specifically a username/password\ncredential as opposed to an access key credential.\n\nEarlier version credentials (.vault files) are **not** supported.\n\nConsult the MatrixStore documentation for how to create and provision users.\n\n## What now?\n\nOnce you're logged in, use the `help` command to list out the actions that can be done:\n\n```\n{vault-id}\u003e help\nAvailable commands:\n\tconnect                - connects to an appliance\n\tdisconnect | dis       - close the current connection opened by `connect`, does not show help\n\texit                   - leaves the program, does not show help\n\tstacktrace             - show detailed information for the last error that happened\n\tsearch {query-string}  - perform a search on the MatrixStore\n\tlookup {filepath}      - perform a basic search on MXFS_FILEPATH and return full metadata. Intended to be used for single files.\n\tmd5 {oid}              - calculate appliance-side checksum for the given object. Get the OID from `search` or `lookup`.\n\tmeta {oid}             - show all metadata fields associated with the given object. Get the OID from `search` or `lookup`.\n\tget {oid}              - download the file content of {oid} to the current directory.\n\tdelete {oid}           - delete the object from the appliance. Note that if there is no Trash period configured, the file will be gone baby gone.\n\tsearchdel {query-string} - delete every object that matches the given query string. The list of objects to delete is shown first and you are prompted whether to continue or not\n\tset timeout {value}    - changes the async timeout parameter. {value} is a string that must parse to FiniteDuration, e.g. '1 minute' or '2 hours'. Default is one minute.\n\tset pagesize {value}   - changes the number of rows to be printed before pausing for QUIT/CONTINUE when performing a search\n\tshow headers {on|off}  - set whether to show the header line when searching\n\nWrite any command without arguments to see a brief summary of any options that are required\n{vault-id}\u003e \n```\n\n## Some notes on quoting and escaping\n\nThis program uses the jline3 library to provide a commandline terminal.\n\nJust like a regular unix shell, arguments are separated by space characters. Multiple spaces are compressed to one.\nIf you want to have spaces in an argument, you must enclose it within double-quotes, like this:\n\n```\nlookup \"path/to/my/file with space.ext\"\n```\n\nIf there are no spaces, then the following two are equivalent:\n\n```\nlookup path/to/my/file.ext\nlookup \"path/to/my/file.ext\"\n```\n\nIf you need to send quotes as _part_ of your argument, for example in a query string, then you must escape them with the \n`\\` character:\n\n```\nsearch \"MXFS_FILENAME:\\\"my file with spaces.ext\\\"\"\n```\n\nThis will send the query string `MXFS_FILENAME:\"my file with spaces.ext\"`.  If you were to leave out the inner quotes,\nthen the command would still be sent but the MatrixStore appliance would probably send back either an error or\nunexpected results as it has not interpreted the query how you want.\n\nIf you were to forget to escape the inner quotes, then you'd probably get a syntax error from the shell; it would consider\nthere to be four parameters on the search command being `MXFS_FILENAME:my`, `file`, `with`, `spaces.ext`.\n\n## Some notes on searching\n\nThe `search` command paginates by default, i.e. it will show you one \"page\" of results\nand then pause, asking you whether to continue or quit.\n\nDon't be afraid to run `search *` to find everything, as you won't lock your terminal up\nor be stuck waiting for it to spool through millions of results.\n\nThe default page size is 10, but you can change this with the `set pagesize` command.\n\nYou'll notice that the `search` command in the list above takes a single parameter called `query-string`.  So you're probably\nwondering what the hell this is.\n\nQueries are based upon Lucene 5.3.1 syntax - which means that they are basically the same as what you would put into Kibana.\n\nIn general, you specfy a number of terms which are then combined together.  Terms are separated by spaces, so if you want\na space in your term you must quote the term with the `\"` character.\n\nYou can optionally specify a field to search for the term on, e.g. `search \"MXFS_FILENAME:\\\"my file with spaces.ext\\\"\"`\n\nIf you were to type `search \"MXFS_FILENAME:my file with spaces.ext\"`, then it would search for `my` in the field MXFS_FILENAME\nand then for `file` in any field, `with` in any field and `spaces.ext` in any field and OR the results together.\nNet result - a ton of unexpected results.\n\n Wildcards are supported, as per section 4.1.3 of \"Basic Searching\":\n\u003e The single character wildcard search looks for terms that match that with the single character replaced. For example, to search for \"text\" or \"test\" you can use the search:\n\u003e ```\n\u003e te?t\n\u003e ```\n\u003e \n\u003e Multiple character wildcard searches looks for 0 or more characters. For example, to search for test, tests or tester, you can use the search:\n\u003e ```\n\u003e test*\n\u003e ```\n\u003e You can also use the wildcard searches in the middle of a term.\n\u003e ```\n\u003e te*t\n\u003e ```\n\nIf you want to discover a bit more about the metadata, then you can run\n```\n{vault-id}\u003e search *\n2021-06-17 10:42:47,181 [INFO] from streamcomponents.OMFastSearchSource$$anon$1 in matrixstore-client-akka.actor.default-dispatcher-11 - Connection established\n1215748_KP-31725704_poster\n0cb01940-3efc-11eb-887a-cb5c82a97316-11\n14423\n------------\n.\n.\n.\nPress Q [ENTER] to quit or [ENTER] for the next page\n\n```\nwhich will show you the first page of results as above.\nYou can see that the first result is a file with MXFS_FILENAME of `1215748_KP-31725704_poster`\nand an ObjectMatrix ID (oid) of `0cb01940-3efc-11eb-887a-cb5c82a97316-11`.\n\nCopy-pasting the OID value onto the `meta` command like so will show you all of the metadata fields for the given file:\n``` \n{vauld-id}\u003e meta 0cb01940-3efc-11eb-887a-cb5c82a97316-11\n0cb01940-3efc-11eb-887a-cb5c82a97316-11\t14423\t\t1215748_KP-31725704_poster\n\tMXFS_FILENAME_UPPER: 1215748_KP-31725704_POSTER\n\tGNM_PROJECT_ID: 12806\n\tGNM_TYPE: Metadata\n\tMXFS_PATH: 1215748_KP-31725704_poster\n\tGNM_COMMISSION_ID: 2579\n\tMXFS_FILENAME: 1215748_KP-31725704_poster\n\tGNM_WORKING_GROUP_NAME: Multimedia News\n\tGNM_PROJECT_NAME: 221216 James B AM\n\tMXFS_MIMETYPE: application/xml\n\tMXFS_DESCRIPTION: Metadata for 1215748_KP-31725704.null\n\tGNM_COMMISSION_NAME: 19 December 2016 weekly agency news\n\tMXFS_CREATIONDAY: 16\n\tMXFS_COMPATIBLE: 1\n\tMXFS_ARCHYEAR: 2020\n\tMXFS_ARCHDAY: 16\n\tMXFS_CREATIONMONTH: 12\n\tMXFS_CREATIONYEAR: 2020\n\tMXFS_CATEGORY: 4\n\tMXFS_ARCHMONTH: 12\n\tMXFS_MODIFICATION_TIME: 1608139573806\n\t__mxs__length: 14423\n\tMXFS_CREATION_TIME: 1608139573561\n\tMXFS_ACCESS_TIME: 1608139573806\n\tDPSP_SIZE: -1\n\tMXFS_ARCHIVE_TIME: 1608139573711\n\tGNM_MISSING_PROJECTINFO: false\n\tMXFS_INTRASH: false\n\tGNM_HIDDEN_FILE: false\n{vauld-id}\u003e\n```\nYou'll notice that there are MXFS_PATH and MXFS_FILENAME fields present.\nIn general, MXFS_PATH is the full path _relative_ to the Vidispine storage that \nthe item originated on (so, if something is in the Assets folder you would expect the\nfirst portion to be the working group; if something is in the Masters folder you would\nexpect it to not have a path).\nYou'll also notice a bunch of GNM fields to help you isolate where the file came\nfrom.  All of these are searchable just the same as the path/filename, e.g.\n\n```\n{vault-id}\u003e search \"GNM_PROJECT_ID:12806 AND GNM_TYPE:Metadata\"\n```\n\nSee `Basic Searching.md` for more information about boolean query terms.\n\n\n`MXFS_FILENAME` is generally the bare filename without the path.\n**NOTE that the `lookup` command is an EXACT MATCH on the `MXFS_FILENAME` field only**\nIf you don't get the results you expect with `lookup`, then use `search` to try to pin\ndown the file and `meta` to examine the metadata to see what is wrong.\nSometimes, path segments exist in `MXFS_FILENAME` which can be confusing.\n\n\nFor full details, you should read chapter four of the \"Content Search\" PDF guide that comes with the ObjectMatrix SDK.\n\nI have extracted the relevant parts from version 3.2 into the \"Basic Searching.md\" file for\nconvenience but it's always worth double-checking the appliance version and getting hold\nof the right guide directly from ObjectMatrix.\n\nThe search queries are not validated or interpreted by this software, they are just passed over blindly.\n\n## How to download something?\n\n1. Log into the vault\n2. Use the `search` or `lookup` commands to find what you are interested in\n3. Find the OID of the file. This is the long thing that looks like a UUID with another part at the end\n4. Run `get` then the OID. The file will be downloaded to the current working directory of your machine, with a filename taken from the MXFS_FILE field.\n\n# How to build\n\nOnce you have a v1.8 JDK installed and SBT (**NOTE** - the ObjectMatrix\nlibraries _require_ JDK 1.8 and nothing newer) then you can simply build by either:\n\n```\nsbt assembly\n```\nwill produce a single, runnable JAR file in the `target/scala-2.13` directory or:\n```\nsbt docker:publishLocal\n```\nwill produce a Docker image on your local Docker daemon.\n\nOr you can run \"from source\" by:\n\n```\nsbt run\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguardian%2Fmatrixstore-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguardian%2Fmatrixstore-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguardian%2Fmatrixstore-client/lists"}