{"id":19402681,"url":"https://github.com/cscfi/allas-cli-utils","last_synced_at":"2026-02-27T23:16:11.981Z","repository":{"id":38191473,"uuid":"180119574","full_name":"CSCfi/allas-cli-utils","owner":"CSCfi","description":"Utilities for Allas command line access","archived":false,"fork":false,"pushed_at":"2025-12-01T14:24:38.000Z","size":643,"stargazers_count":4,"open_issues_count":2,"forks_count":5,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-12-04T03:21:09.063Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-04-08T09:51:24.000Z","updated_at":"2025-12-01T14:24:42.000Z","dependencies_parsed_at":"2023-02-17T16:15:33.411Z","dependency_job_id":"9ce79aae-5b8b-4bbf-a687-d07d388720e0","html_url":"https://github.com/CSCfi/allas-cli-utils","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CSCfi/allas-cli-utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fallas-cli-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fallas-cli-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fallas-cli-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fallas-cli-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/allas-cli-utils/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fallas-cli-utils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29918976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-27T19:37:42.220Z","status":"ssl_error","status_checked_at":"2026-02-27T19:37:41.463Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:25:10.461Z","updated_at":"2026-02-27T23:16:11.966Z","avatar_url":"https://github.com/CSCfi.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# allas-cli-utils - Utilities for Allas command line access\n\nThe Allas object storage system can be used in multiple ways and for many purposes. \nIn many cases, effective usage of Allas requires that the user knows the features of \nboth Object Storage systems and the software or protocol that is used to manage data in Allas.\n\nFor those users, that just want to use Allas for storing data that is in CSC computing environment, \nCSC provides a set of commands (a-tools ) for moving data between CSC computing environment and Allas.\n\n## Opening connection with allas_conf\n\nConnection configuration tool: allas_conf can be used to configure swift or S3 connections to Allas. \nThe tool is designed for linux **bash shell**, that is the default command shell in CSC computing environment.\nThe basic syntax of this tool for swift protocol is:\n\n```text\n   source allas_conf --user your-CSC-user-account\n```\nand for S3 protocol:\n```text\n   source allas_conf --mode s3cmd --user your-CSC-user-account\n```\n\nAfter successful connection configuration you can start using tools like, _swift_, _rclone_, _restic_, __A_tools__\nor _s3cmd_ to manage data in allas.\n\n## A_ -tools for easy access to Allas\n\nFor those users, that just want to use Allas for storing data that is in CSC computing environment, CSC provides a set of commands for managing and moving data between CSC computing environment and Allas.\n\n**Note! since the update done on 1.3. 2022, a-put no longer compresses the uploaded data as a default preprocessing operation. In the future, use option -c in case you want to compress the data before upload.**\n\n\n## Four main tools for using Allas\n\n|a-command | Function |\n| :--- | :--- |\n| [a-put](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-put) | Upload a file or directory to Allas as one object |\n| [a-get](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-get) | Download a stored dataset (object) from Allas |\n| [a-list](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-list) | List buckets and objects in Allas |\n| [a-delete](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-delete) | Delete an object from Allas |\n\n## Other tools\nThe available tools are:\n  \n|a-command | Function |\n| :--- | :--- |\n| [a-access](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-access)| Manage access permissions of your buckets in Allas || [a-check](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-check) | Command to check if a-put command was successfully executed |\n| [a-check](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-check) | Command to check if a-put command was successfully executed |\n| [a-encrypt]() | Make an encrypted copy of an object to make it compatible with CSC sensitive data services | \n| [a-find](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-find)| Search and locate data that has been uploaded with a-put |\n| [a-flip](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-flip)| Upload a file to Allas into a bucket that will keep the file temporarily available to the internet |\n| [a-info](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-info)| Display information about an object in Allas |\n| [a-publish](https://docs.csc.fi/data/Allas/using_allas/a_commands/#a-publish) | Upload a file to Allas into a bucket that allows public access over the internet |\n\n\n\n   \nIn addition to the above commands, there is separate tool to create incremental backups:\n\n*    `allas-backup` : create a backup copy of a local dataset into a backup repository in Allas\n\nAnd a tool for mounting Allas buckets as directories to a local computer (not yet in production)\n\n*    `allas-mount`\n\n## Installing allas-cli-utils\n\nAllas-cli-utils is mainly designed to be used in the HPC computing environment of CSC.\nIn CSC computers (Puhti, Mahti ) these tools are installed and maintained by CSC.\n\nYou can install these tools in your local Linux and Mac environment too, but in order to use\nthese tools you should have several software components available:\n\n__allas_conf__ script requires:\n\n*   [bash shell](https://en.wikipedia.org/wiki/Bash_(Unix_shell))\n*   [OpenStack client](https://github.com/openstack/python-openstackclient)(3.19 or newer, not mandatory if you use swift and know the project name.)\n*   [OpenStack Swift](https://github.com/openstack/swift) and/or [s3cmd](https://s3tools.org/s3cmd) client\n, y\nI addition to the above, a-tools require:\n\n*   [rclone](https://rclone.org/) (1.48 or newer)\n*   [zstdmt](https://github.com/mcmilk/zstdmt)\n*   [crypt4gh](https://crypt4gh.readthedocs.io/en/latest/)\n\nIf you want to use a-commands to upload or download data from *SD Connect sensitve data storage service* you need to have *SD lock util* installed\n\n* [sd-lock-util](https://github.com/CSCfi/sd-lock-util)\n\n\nExample: Installing allas-cli-utils to a Ubuntu 16.04 server running in cPouta:\n\n```text\n#install pip and openstack client\nsudo apt install python-pip python-dev\nsudo apt-get install python-setuptools\npip install --upgrade pip\npip install python-openstackclient\npip install crypt4gh\npip install s3cmd\n\n#install swift client\nsudo apt install python3-swiftclient\ncurl https://rclone.org/install.sh | sudo bash\ngit clone https://github.com/CSCfi/allas-cli-utils\ncd  allas-cli-utils/\nexport PATH=${PATH}:$(pwd)\n```\n\nExample: Installing allas-cli-utils to Ubuntu 18.04 so that conda is used in the installation of dependencies.\n\n\n```text\nsudo apt-get update\nsudo apt-get install gcc\nsudo apt-get install restic\ncurl https://rclone.org/install.sh | sudo bash\nwget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\nbash Miniconda3-latest-Linux-x86_64.sh \nbash\ngit clone https://github.com/CSCfi/allas-cli-utils\ncd allas-cli-utils/\nexport PATH=${PATH}:$(pwd)\nconda env create -n allas --file allas-dependencies.yaml \nconda activate allas\nsource allas_conf -u csc-useraccount\na-list \n```\nExample: Installing allas-cli-utils to CentOS 7 based virtual machine in cPouta\n\n```text\nsudo yum update                          # update everything first\nsudo yum install unzip                   # install rclone\ncurl https://rclone.org/install.sh | sudo bash                \nsudo yum install yum-plugin-copr         # three commands to install restic\nsudo yum copr enable copart/restic\nsudo yum install restic\n# install miniconda\nwget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\nbash Miniconda3-latest-Linux-x86_64.sh\nbash                                      #new session to enable miniconda\ngit clone https://github.com/CSCfi/allas-cli-utils       #download allas tools\ncd allas-cli-utils/\nexport PATH=${PATH}:$(pwd)\nconda create -n allas\nconda activate allas\nconda install -c conda-forge zstd python pip s3cmd\npip install --upgrade setuptools\npip install python-openstackclient\npip install python-swiftclient\npip install crypt4gh\nsource allas_conf -u csc-useraccount                     # replace csc-useraccount with your personal CSC account\na-list\n\n```\n\n\nallas_conf and basic a- commands now support MacOS (but you still must use bash in MacOS to run them). In case md5 cheksum command still fails, there is simple replacement script (md5sum.macosx) that\nyou can take in use in macs by giving following in the command in the allas-cli-utils directory:\n```\nmv md5sum.macosx md5sum\n```\n\nExample commands in macOS to install python3 libraries in to atools-venv directory for allas-cli-utilities using bash in macOS (note that you also need binaries like rclone and allas-cli-utilities itself from github):\n```\npython3 -m venv atools-venv\nsource atools-venv/bin/activate\npip3 install --upgrade pip\npip3 install openstackclient\npip3 install s3cmd\n```\n\nThen the \"normal\" way to start using Allas in macOS in a bash session would be like (exact details would depend on how you want to manage your PATH etc):\n```\nsource atools-venv/bin/activate\nsource allas_conf\n```\n\n## Configuring a-commands\n\nYou can define the default settings that the a-tools use in two files: a_env_conf and .a_tools_conf. These settings affect mostly to the a-put data upload command.\n\n### 1. a_env_conf\n\nThe file *a_env_conf*, that locates in the installation directory of _allas-cli-utils_, defines some installation specific settings. Users are not able to modify this file in CSC servers (Puhti and Mahti), but this file includes settings that you  want to change if you do a local allas-cli-utils installation. \n\n   * _allas_conf_path_ should define the location of _allas_conf_ script in your system.\n   * You may also want to change _tmp_root_ definition if you want use some other location than /tmp for temporary tiles.\n\n### 2. .a_tools_conf\n\nA user can define her own default settings for a-commands by making a configuration file named as **.a_tools_conf** to her **home directory**. These user specific settings can be done as well at the CSC servers too. In this file you can set default values for many of the functions that are defined with a-put command options.\n\nFor example, if you are uploading files that would benefit from compression, you could use _--compress_ option with a-put. If you want this to be default setting you could create .a_tools_conf file\nthat contains setting:\n\n```text\ncompression=1\n```\nNow command:\n```text\na-put my_data.b\n```\nWill compress the data during the upload process. However, you can still skip the compression with option _--nc_.\n\n```text\na-put --nc my_data.b\n```\nYou can check most commonly used settings from this sample [.a_tools_conf](./.a_tools_conf) file. Copy the sample file to your home directory and uncomment and define the variables you wish to use.\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fallas-cli-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fallas-cli-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fallas-cli-utils/lists"}