{"id":19402616,"url":"https://github.com/cscfi/ida2-command-line-tools","last_synced_at":"2025-04-24T07:32:13.552Z","repository":{"id":71818889,"uuid":"139105907","full_name":"CSCfi/ida2-command-line-tools","owner":"CSCfi","description":"IDA Research Data Service v2 - Command Line Tools","archived":false,"fork":false,"pushed_at":"2024-11-06T06:33:19.000Z","size":63998,"stargazers_count":10,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-06T07:29:10.804Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-29T05:43:25.000Z","updated_at":"2024-11-06T06:33:21.000Z","dependencies_parsed_at":"2024-05-08T11:43:15.929Z","dependency_job_id":null,"html_url":"https://github.com/CSCfi/ida2-command-line-tools","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fida2-command-line-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fida2-command-line-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fida2-command-line-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fida2-command-line-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/ida2-command-line-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223945204,"owners_count":17229595,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:24:46.760Z","updated_at":"2024-11-10T11:24:47.375Z","avatar_url":"https://github.com/CSCfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nThis file is part of the IDA research data storage service\n\nCopyright (C) 2018 Ministry of Education and Culture, Finland\n\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU Affero General Public License as published\nby the Free Software Foundation, either version 3 of the License,\nor (at your option) any later version.\n\nThis program is distributed in the hope that it will be useful, but\nWITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY\nor FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public\nLicense for more details.\n\nYou should have received a copy of the GNU Affero General Public License\nalong with this program. If not, see \u003chttp://www.gnu.org/licenses/\u003e.\n\n@author CSC - IT Center for Science Ltd., Espoo Finland \u003cservicedesk@csc.fi\u003e\n@license GNU Affero General Public License, version 3\n@link https://research.csc.fi/\n--\u003e\n\n# IDA Command Line Tools\n\nThis repository provides a [bash](https://www.gnu.org/software/bash/) script `ida` for interacting with\nthe [Fairdata IDA service](https://www.fairdata.fi/en/ida/) from the command line.\n\nThe script should execute correctly on any Unix like system; including Linux, BSD, Mac OS-X, and Windows\nSubsystem for Linux (WSL).\n\nSee the installation and configuration instructions below.\n\n    Usage: ida [-h]\n           ida upload    [-v|V] [-D] [-F] [-c config] [-i ignore] [-t host] [-p project]      target_pathname local_pathname\n           ida copy      [-v|V] [-D]      [-c config]             [-t host] [-p project] [-f] target_pathname new_target_pathname\n           ida move      [-v|V] [-D]      [-c config]             [-t host] [-p project]      target_pathname new_target_pathname\n           ida delete    [-v|V] [-D]      [-c config]             [-t host] [-p project]      target_pathname\n           ida download  [-v|V]           [-c config]             [-t host] [-p project] [-f] target_pathname local_pathname\n           ida validate  [-v|V]           [-c config]             [-t host] [-p project] [-f] target_pathname local_pathname\n           ida info      [-v|V]      [-j] [-c config]             [-t host] [-p project] [-f] target_pathname\n           ida inventory [-v|V]           [-c config]             [-t host] [-p project]\n\n           -h : show this guide\n           -p : project name\n           -t : target host (default: \"https://ida.fairdata.fi\")\n           -f : target_pathname is relative to frozen area, new_target_pathname is relative to staging area\n           -c : configuration file\n           -i : ignore file\n           -v : provide verbose output\n           -V : provide both verbose and debug output with explicit details about configuration and all operations\n           -D : dry-run (does not perform any operations with changes in the IDA service)\n           -F : force upload (upload files even when the local file already exists in the service)\n           -j : format the output of the info action as JSON\n\nPathnames may correspond to either files or folders. If a folder is specified, then the action is\nperformed for all files within that folder and all subfolders. Folders are downloaded as zip files.\nActions can be performed on only one file or folder at a time.\n\nUnless the -f parameter is specified, `target_pathname` and `new_target_pathname` are relative to the\nstaging area of the specified project. If the -f parameter is specified, then the `target_pathname` is\nrelative to the frozen area; and if the `copy` action is specified, the `new_target_pathname` is relative\nto the staging area of the specified project. The -f parameter is **only** allowed for `download`, `copy`,\n`validate`, and `info` actions.\n\n`local_pathname` is the pathname of a folder or file on the local system which is to be uploaded, or the\npathname on the local system to which a file will be downloaded (either a single data file or the zip\nfile for a data folder). Existing files will not be overwritten.\n\nThe `move` action can also be used to rename a file or folder without changing its location.\n\nIf both the -v (verbose) and the -D (dry-run) parameters are specified, then verbose status messages\nfor operations which are skipped will be prefixed with an asterisk '*' indicating that they were not\nactually performed.\n\nNote that files are not officially stored persistently in the IDA service until they are frozen, which can\nonly be done using the web UI of the service (https://www.fairdata.fi/en/ida/user-guide/#project-data-storage).\n\nThe output format of the info action is plain text with indentation, unless the -j (JSON) flag is given.\nThe output format of the inventory action is always formatted as JSON.\n\nExamples are provided at the end of this guide.\n\n## Installation\n\nThe `ida` and `ida-checksum` scripts requires that `bash` is installed on your system as `/bin/bash` and that\nthe utilities `curl`, `xargs`, `awk`, and `sha256sum` (or `shasum` on Mac OS-X) are installed on your system\nand visible somewhere in your `$PATH`.\n\nDownload the `ida` and `ida-checksum` scripts, ideally placing them in a location somewhere in your `$PATH`,\nand ensure that the scripts are executable. E.g.:\n\n    curl -LJO \"https://raw.githubusercontent.com/CSCfi/ida2-command-line-tools/master/ida\"\n    curl -LJO \"https://raw.githubusercontent.com/CSCfi/ida2-command-line-tools/master/ida-checksum\"\n    chmod +x ida ida-checksum\n\n## Authentication\n\nThe `ida` script must be provided with valid IDA account credentials in order to access the project space.\n\nThe username will be your CSC account username, as managed in the CSC Customer Portal [MyCSC](https://my.csc.fi/).\n\nThe password will be an application password that you create in the IDA service, as detailed below.\n\n### Application passwords\n\nTo create an application password, log in to the IDA web UI, and open the settings view by selecting \"Personal\"\nfrom the pull down settings menu at the top right of the view, and then either select the \"Security\" section\nin the left hand navigation pane or scroll down to the security section.\n\nIn the field provided, enter a name for your new application password and click \"Create new app password\". The\nnew application password will be displayed, and must be copied and saved immediately. It will not be possible to\nview the application password again, though it will be listed by the provided name and you will be able to delete\nit and remove it from use.\n\n## Configuration\n\nThe `ida` script requires that certain parameters are defined, either via environment variables, or\nvia a configuration file, or provided via command line arguments.\n\n### Configuration file\n\nThe `ida` script will look for and load a file `$HOME/.ida-config` if it exists in your home directory.\n\nYou can copy the provided example `ida-config` file from the `templates` subdirectory to your home directory\nas `$HOME/.ida-config` and edit it accordingly. E.g.:\n\n    curl -LJ \"https://raw.githubusercontent.com/CSCfi/ida2-command-line-tools/master/templates/ida-config\" -o $HOME/.ida-config\n\nBe sure to set the permissions of your configuration file securely so its contents are not visible to others:\n\n    chmod go-rwx $HOME/.ida-config\n\nIt is also possible to explicitly specify a configuration file using the `-c` command line option:\n\n    -c /some/path/name/to/ida-config\n\nConfiguration settings will be taken from existing environment variables, followed by `$HOME/.ida-config`,\nfollowed by a configuration file specified with the `-c` command line option, followed by any other\nspecified command line options; thus, it is possible to partially or completely override configuration\nsettings defined at each subsequent level of specification.\n\n### Target Host\n\nYou may define the target host using the `IDA_HOST` environment variable, when appropriate. If undefined,\nall commands will be directed to the production service `https://ida.fairdata.fi`.\n\nFor example, if you want all commands to be directed to the IDA demo environment host\n`https://ida.demo.fairdata.fi` you would define it as follows:\n\n    IDA_HOST=\"https://ida.demo.fairdata.fi\"\n\nYou may also specify the target host using the `-t` command line option:\n\n    -t \"https://ida.demo.fairdata.fi\"\n\n### Project\n\nYou may define your project using the `IDA_PROJECT` environment variable, specifying the CSC project number.\n\nFor example, if your project's CSC project number were \"2100123\" you would define it as follows:\n\n    IDA_PROJECT=\"2100123\"\n\nYou may also specify the CSC project number using the `-p` command line option:\n\n    -p \"2100123\"\n\nIf you belong to more than one project, you can define whichever project is most frequently used and\nuse the `-p` commmand line option to explicitly specify another project when different from the default.\n\nAll target pathnames are considered to be relative to the staging or frozen area of the specified project.\n\n### Username\n\nYou may define your CSC username using the `IDA_USERNAME` environment variable.\n\nFor example, if your CSC username were \"johndoe\" you would define it as follows:\n\n    IDA_USERNAME=\"johndoe\"\n\n### Password\n\nYou may define your IDA application password using the `IDA_PASSWORD` environment variable.\n\nFor example, if your IDA application password were \"Wre-xoZyN-d4dWc-RStwi-36d2t\" you would define it as follows:\n\n    IDA_PASSWORD=\"Wre-xoZyN-d4dWc-RStwi-36d2t\"\n\n### Netrc\n\nFor added security, account credentials (username and password) can be defined for each\ntarget host using [netrc](https://ec.haxx.se/usingcurl-netrc.html) in a `$HOME/.netrc` file\nlocated in your home directory:\n\n    machine ida.fairdata.fi login johndoe password Wre-xoZyN-d4dWc-RStwi-36d2t\n\nIf account credentials are defined using netrc, it will not be necessary to define either `IDA_USERNAME` or\n`IDA_PASSWORD` elsewhere.\n\nYou can copy the provided example `netrc` file from the `templates` subdirectory to your home directory\nas `$HOME/.netrc` and edit it accordingly.\n\nBe sure to set the permissions of your configuration file securely so its contents are not visible to others:\n\n    chmod go-rwx $HOME/.netrc\n\n### Run-time authentication\n\nIf any account credentials (username and/or password) cannot be found from any of the above\nsources, you will be prompted to enter them for each command.\n\n## Ignore file\n\nThe `ida` script will look for and use a file `.ida-ignore` if it exists in your home directory, to exclude\nfiles matching certain patterns from upload.\n\nThis is useful when uploading entire folders, to exclude special system files such as `.DS_Store` on MacOSX or\nvarious temporary or log files which might exist within the local filesystem but are not part of the data to\nbe stored in IDA.\n\nYou can copy the provided example `ida-ignore` file from the `templates` subdirectory to your home directory\nas `$HOME/.ida-ignore` and edit it accordingly.\n\nThe ignore file should contain one pattern per line, and will be applied only to filenames, not to pathnames\nor portions of pathnames. Patterns should be compatible with those understood by the `-name` option of the\nPOSIX `find` command.\n\nIt is also possible to explicitly specify an ignore file using the `-i` command line option:\n\n    -i /some/path/name/to/ida-ignore\n\n## Collision Avoidance for File Operations\n\nAll users belonging to a given project have the same rights, and may interact with, add, and remove\nproject data concurrently. To help avoid one user unintentionally interfering with another user's\nactivity, the IDA service employs a number of checks and restrictions to ensure that multiple concurrent\nusers' activities do not collide in undesirable ways.\n\nOperations on files in the staging area, including uploading, deleting, renaming, and moving (but not\ndownloading), will not be allowed if they intersect with the scope of an action that is being initiated.\n\nIf an action is initiated in the web UI of the service, it takes precidence over any batch operations\nutilizing the command line tools, such that the batch operations may be blocked by the initiated action.\nIn such cases, the command line tools will exit with an error message.\n\nMore details can be found in the [online user guide](https://www.fairdata.fi/en/ida/user-guide#collision-avoidance).\n\n## Info Output\n\nThe output of the `info` action is both human friendly as well as easily parsable by script.\n\n### File Info\n\nFile info consists of a sequence of field::value pairs, one per line, with the field name\nfollowed by a colon \":\" and padded by one or more spaces. E.g.\n\n    project:    12345\n    pathname:   /2017-08/Experiment_1/test01.dat\n    area:       frozen\n    type:       file\n    pid:        5c41b5d90561a361661419f75150\n    size:       446\n    checksum:   sha256:56293a80e0394d252e995f2debccea8223e4b5b2b150bee212729b3b39ac4d46\n    encoding:   application/octet-stream\n    modified:   2017-01-03T19:27:42Z\n    frozen:     2017-01-18T11:17:15Z\n\nFields included for all files, in either the staging or frozen areas:\n\n* project\n* pathname\n* area\n* type\n* size (in bytes)\n* encoding\n* checksum (not always available, see note below)\n* modified\n\nAdditional fields provided for all files in the frozen area:\n\n* pid\n* frozen\n\nNote: The checksum for a file is not always available, either because a file in staging was\nnot uploaded using a recent version of the IDA command line tools and/or postprocessing is\nstill ongoing for the freeze action with which a frozen file is associated. If the file is\nfrozen and no checksum is included in the output of the info action, check the pending actions\nsection of the IDA service web UI to monitor the status of the relevant pending action.\n\n### Folder Info\n\nFolder info consists of a sequence of field::value pairs, one per line, with the field name followed\nby a colon \":\" and padded by one or more spaces. Field::value pair lines are\nfollowed by a heading \"contents:\" on its own line, followed\nby zero or more pathnames, one per line, corresponding to the files or folders contained within the specified\nfolder. E.g.\n\n    project:    12345\n    pathname:   /2017-08/Experiment_1\n    area:       staging\n    type:       folder\n    size:       23040\n    contents:\n      /2017-08/Experiment_1/test01.dat\n      /2017-08/Experiment_1/test02.dat\n      /2017-08/Experiment_1/test03.dat\n      /2017-08/Experiment_1/test04.dat\n      /2017-08/Experiment_1/test05.dat\n      /2017-08/Experiment_1/baseline/\n\nEach pathname is indented with whitespace, and thereby easily distinguished from lines with field::value pairs,\nwhere the field name begins at the start of each line.\n\nFolder pathnames end in a forward slash \"/\", and thereby easily distinguished from file pathnames, which never\nend in a forward slash.\n\nEmpty folders will have no output lines following the \"contents:\" heading.\n\nTo retrieve a listing of the root contents of either the staging or frozen area for a project, the target\npathname \"/\" can be specified.\n\nFields included for all folders, in either the staging or frozen areas:\n\n* project\n* pathname\n* area\n* type\n* size (in bytes)\n* contents\n\n### File Inventory\n\nThe output of the `inventory` action is encoded as a JSON object with the following example structure:\n\n    {\n        \"project\": \"12345\",\n        \"created\": \"2020-03-13T12:58:36Z\",\n        \"totalFiles\": 83,\n        \"totalStagedFiles\": 45,\n        \"totalFrozenFiles\": 38,\n        \"staging\": {\n            \"/2017-08/Experiment_2/test01.dat\": {\n                \"size\": 1234,\n                \"modified\": \"2020-02-11T14:04:26Z\"\n            },\n            ...\n        },\n        \"frozen\": {\n            \"/2017-08/Experiment_1/test02.dat\": {\n                \"size\": 4567,\n                \"modified\": \"2020-02-11T14:04:26Z\",\n                \"pid\": \"5e4ea9a829bfe465487185f441180\",\n                \"checksum\": \"sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c\",\n                \"frozen\": \"2020-02-20T15:07:32Z\"\n            },\n            ...\n        }\n    }\n\n## Validation\n\nThe `validate` action can be used to check whether local files exist in IDA and have the same size in bytes in\nboth locations.\n\nThe target and local pathnames can correspond either to an individual file or a directory scope for which all\nfiles within that scope will be checked. If the -f (frozen) parameter is given, local files will be compared\nagainst files in the frozen area; else they will be compared against files in staging.\n\nNormally, only files that are missing from IDA or have a different size are reported, but if the -v (verbose)\nparameter is given, the status of all files will be reported.\n\nFiles within a directory scope which exist in IDA but which are not present locally will be ignored.\n\n## Local File Checksum Generation\n\nThe included utility `ida-checksum` can be used to generate a checksum for a local file, of the same\nformat (SHA-256, lowercase) used by the IDA service.\n\n    Usage: ida-checksum -h\n           ida-checksum local_pathname\n\n## Special Notes\n\nFiles named `.htaccess` and files with a suffix of either `.part` or `.filepart` may not be uploaded.\nThe filenames will need to be changed in some manner, such as zipping the file or adding some other\nsuffix or name change. These restrictions are due to security and other internal management contraints\nof the underlying Nextcloud platform employed by the IDA service.\n\nNote that files are not officially stored persistently in the IDA service\nuntil they are [frozen](https://www.fairdata.fi/en/ida/user-guide/#project-data-storage),\nwhich can only be done using the web UI of the service.\n\n## Examples\n\nFor the following examples, it is assumed that the user has defined their primary project in their\n`.ida-config` file and their account credentials in their `.netrc` file.\n\nNote that when uploading, copying or moving files or folders, complete target pathnames in the project staging area must\nalways be specified, not merely the pathname of the folder into which data will be uploaded or moved.\n\n**Upload a local folder named \"run42\" to the staging area with the relative pathname \"/2017-04/Experiment_42\":**\n\n    ida upload /2017-04/Experiment_42 run42\n\n**Perform the same upload as above, with verbose output detailing all operations, and appending (saving) the output (including error output) to the local file \"./uploads.log\":**\n\n    ida upload -v /2017-04/Experiment_42 run42 2\u003e\u00261 | tee -a ./uploads.log\n\n**Upload a local file named \"run42/test66.dat\" to the staging area with the relative pathname \"/2017-04/Experiment_42/test66.dat\":**\n\n    ida upload /2017-04/Experiment_42/test66.dat run42/test66.dat\n\n**Download a folder from the staging area with the relative pathname \"/2017-04/Experiment_42\" to the local file \"run42.zip\":**\n\n    ida download /2017-04/Experiment_42 run42.zip\n\n**Download a folder from the frozen area with the relative pathname \"/2017-04/Experiment_42\" to the local file \"run42.zip\":**\n\n    ida download -f /2017-04/Experiment_42 run42.zip\n\n**Download a file from the staging area with the relative pathname \"/2017-04/Experiment_42/test66.dat\" to the local pathname \"run42/test66.dat\":**\n\n    ida download /2017-04/Experiment_42/test66.dat run42/test66.dat\n\n**Download a file from the frozen area with the relative pathname \"/2017-04/Experiment_42/test66.dat\" to the local pathname \"run42/test66.dat\":**\n\n    ida download -f /2017-04/Experiment_42/test66.dat run42/test66.dat\n\n**Rename (move) a folder in the staging area from \"/2017-04/Experiment_42\" to \"/2017-04/Experiment_42a\":**\n\n    ida move /2017-04/Experiment_42 /2017-04/Experiment_42a\n\n**Copy a file in the staging area from \"/2017-04/Experiment_42/test66.dat\" to \"/2017-04/Experiment_42/verified/test66.dat\":**\n\n    ida copy /2017-04/Experiment_42/test66.dat /2017-04/Experiment_42/verified/test66.dat\n\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;The folder path \"/2017-04/Experiment_42/verified\" will be created if it does not already exist.\n\n**Copy a file from the frozen area from \"/2017-04/Experiment_42/baseline.dat\" to the staging area as \"/2019-10/Experiment_56/baseline.dat\":**\n\n    ida copy -f /2017-04/Experiment_42/baseline.dat /2019-10/Experiment_56/baseline.dat\n\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;The folder path \"/2019-10/Experiment_56\" will be created in the staging area if it does not already exist.\n\n**Move a file in the staging area from \"/2017-04/Experiment_42/test66.dat\" to \"/2017-04/Experiment_42/verified/test66.dat\":**\n\n    ida move /2017-04/Experiment_42/test66.dat /2017-04/Experiment_42/verified/test66.dat\n\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;The folder path \"/2017-04/Experiment_42/verified\" will be created if it does not already exist.\n\n**Delete a folder from the staging area with the relative pathname \"/2017-04/Experiment_42\":**\n\n    ida delete /2017-04/Experiment_42\n\n**Delete a file from the staging area with the relative pathname \"/2017-04/Experiment_42/test66.dat\":**\n\n    ida delete /2017-04/Experiment_42/test66.dat\n\n**Retrieve info about a folder in the staging area with the relative pathname \"/2017-04/Experiment_42\":**\n\n    ida info /2017-04/Experiment_42\n\n**Retrieve info about a folder in the frozen area with the relative pathname \"/2017-04/Experiment_42\":**\n\n    ida info -f /2017-04/Experiment_42\n\n**Retrieve info about a file in the staging area with the relative pathname \"/2017-04/Experiment_42/test66.dat\":**\n\n    ida info /2017-04/Experiment_42/test66.dat\n\n**Retrieve info about a file in the frozen area with the relative pathname \"/2017-04/Experiment_42/test66.dat\":**\n\n    ida info -f /2017-04/Experiment_42/test66.dat\n\n**Retrieve info about the root of the staging area of the project:**\n\n    ida info /\n\n**Retrieve info about the root of the frozen area of the project:**\n\n    ida info -f /\n\n**Retrieve an inventory of all project files stored in the service:**\n\n    ida inventory\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fida2-command-line-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fida2-command-line-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fida2-command-line-tools/lists"}