{"id":20358118,"url":"https://github.com/irods/irods_resource_plugin_s3","last_synced_at":"2026-03-03T22:00:50.325Z","repository":{"id":12649142,"uuid":"15320749","full_name":"irods/irods_resource_plugin_s3","owner":"irods","description":"S3-compatible storage resource plugin for iRODS","archived":false,"fork":false,"pushed_at":"2026-02-03T14:49:11.000Z","size":1282,"stargazers_count":12,"open_issues_count":54,"forks_count":16,"subscribers_count":19,"default_branch":"main","last_synced_at":"2026-02-04T04:37:50.687Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/irods.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING-GPLv2","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2013-12-19T19:32:03.000Z","updated_at":"2026-02-03T14:56:06.000Z","dependencies_parsed_at":"2023-10-11T00:18:19.382Z","dependency_job_id":"c291109c-49ad-4b30-bd48-b46d31d383ec","html_url":"https://github.com/irods/irods_resource_plugin_s3","commit_stats":null,"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"purl":"pkg:github/irods/irods_resource_plugin_s3","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_resource_plugin_s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_resource_plugin_s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_resource_plugin_s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_resource_plugin_s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/irods","download_url":"https://codeload.github.com/irods/irods_resource_plugin_s3/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_resource_plugin_s3/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30063347,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-03T18:21:05.932Z","status":"ssl_error","status_checked_at":"2026-03-03T18:20:59.341Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T23:25:26.718Z","updated_at":"2026-03-03T22:00:50.319Z","avatar_url":"https://github.com/irods.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# iRODS S3 Resource Plugin\n\nThis iRODS storage resource plugin allows iRODS to use any S3-compatible storage device or service to hold iRODS Data Objects, on-premise or in the cloud.\n\nThis plugin can work as a standalone \"cacheless\" resource or as an archive resource under the iRODS compound resource.  Either configuration provides a POSIX interface to data held on an object storage device or service.\n\nInstall the plugin either via your package manager (`yum`/`apt`) and [the binary distributions](https://irods.org/download/ \"iRODS Download\") or use the instructions below to build from source.\n\nThe following S3 services and appliances (in no particular order) have been tested (but may not be in continuous integration):\n\n - Amazon (AWS) S3\n - Fujifilm Object Archive\n - MinIO S3\n - Ceph S3\n - Spectra Logic Vail\n - Spectra Logic BlackPearl\n - Google Cloud Storage (GCS)\n - Wasabi S3\n - Oracle OCI\n - Quantum ActiveScale\n - Garage S3\n\n## Build Prerequisites\n\nTo build the S3 Resource Plugin, you will need to have:\n\n- the iRODS Development Tools (irods-dev(el) and irods-runtime) from https://irods.org/download\n\n- libxml2-dev(el)\n\n- libcurl4-gnutls-dev / curl-devel\n\n- libs3 from https://github.com/irods/libs3\n\n## Build Instructions\n\n```\n$ git clone -b 4-2-stable https://github.com/irods/irods_resource_plugin_s3\n$ mkdir build\n$ cd build\n$ cmake ../irods_resource_plugin_s3\n$ make package\n```\n\nThis will result in a package (deb/rpm) for your platform suitable for installation.\n\n## Example Cacheless Configuration and Usage\n\nAfter installation is complete, the new plugin can be configured in cacheless mode, live on an iRODS Server:\n\n```\nirods@hostname $ iadmin mkresc s3resc s3 $(hostname):/\u003cs3BucketName\u003e/prefix/in/bucket \"S3_DEFAULT_HOSTNAME=s3.us-east-1.amazonaws.com;S3_AUTH_FILE=/var/lib/irods/s3.keypair;S3_REGIONNAME=us-east-1;S3_RETRY_COUNT=1;S3_WAIT_TIME_SECONDS=3;S3_PROTO=HTTP;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached\"\n```\n\nA local file can be immediately put into the S3 resource:\n```\nirods@hostname $ iput -R s3resc foo.txt\n```\n\nAn object already in S3 can be registered into the iRODS Catalog:\n```\nirods@hostname $ ireg -R s3resc /\u003cs3BucketName\u003e/full/path/in/bucket /full/logical/path/to/dataObject\n```\n\nThe S3 Keypair file `S3_AUTH_FILE` should have exactly two values (Access Key ID and Secret Access Key), one per line:\n\n```\nAKDJFH4KJHFCIOBJ5SLK\nrlgjolivb7293r928vu98n498ur92jfgsdkjfh8e\n```\n\n## Configuration Options\n\nThe `S3_DEFAULT_HOSTNAME` may be comma-separated and represent more than one `host:port`:\n\n```\nS3_DEFAULT_HOSTNAME=192.168.122.128:443,192.168.122.129:443,192.168.122.130:443\n```\n\nIf the `S3_DEFAULT_HOSTNAME` points to an AWS host, [best practice](https://docs.aws.amazon.com/general/latest/gr/s3.html) includes the bucket region (e.g. `us-east-1`):\n\n```\nS3_DEFAULT_HOSTNAME=s3.\u003cbucket-region\u003e.amazonaws.com\n```\n\nTo define S3 provider constraints and control multipart behavior:\n-   `S3_MPU_CHUNK` - This defines the minimum part size allowed (in MB).  The default is 5MB which is the minimum part size defined in AWS.\n-   `S3_ENABLE_MPU=0` disables multipart uploads.\n-   `S3_MAX_UPLOAD_SIZE_MB` - This defines the maximum upload size for non-multipart uploads (in MB), maximum part size, as well as the maximum size when using the CopyObject API.  The default is 5120MB (5GB).  This setting is ignored if MPU uploads are disabled.\n-   `S3_MPU_THREADS` is the number of parts to upload in parallel.\n-   `S3_URI_REQUEST_STYLE` - The path request style used.  This is either \"path\" or \"virtualhost\".  The default is \"path\".  See [path vs virtual hosted requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html).\n-   `S3_RESTORATION_DAYS` - The number of days an object is to be restored when restoring from Glacier.  See [RestoreObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html).\n-   `S3_RESTORATION_TIER` - The data access tier option when restoring from Glacier.  Valid values are \"Expedited\", \"Standard\", and \"Bulk\".  The default is \"Standard\".  See [RestoreObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html).\n-   `S3_ENABLE_COPYOBJECT` - Some providers (such as Fujifilm) do not implement the CopyObject S3 API.  If S3_ENABLE_COPYOBJECT=0, the copy will be performed via a read from source and write to destination rather than calling CopyObject.  (Also see the note about GCS support.)\n-   `ENABLE_DIRECT_CHECKSUM_READ` - If this is set to 1, when iRODS needs to calculate the checksum on an object, it will attempt to read the checksum directly from S3 using the `GetObjectAttributes` API.  The default is 0 (off).  See [Enabling Direct Checksum Reads](#enabling-direct-checksum-reads-from-s3-provider) for more information.\n\n\u003e Notes about virtual hosting:  When using virtual hosted request style, configure the resource path and S3_DEFAULT_HOSTNAME as you would for path request style.  Leave the bucket name in the path and do not put the bucket name in the S3_DEFAULT_HOSTNAME.  This is important to retain backward compatibility with objects already created using path request style. \n\nUse the `ARCHIVE_NAMING_POLICY` parameter to control whether the names of the files within the object storage service (S3, or similar) are kept in sync with the logical names in the iRODS Catalog.\nThe default value of `consistent` will keep the names consistent.  Setting `ARCHIVE_NAMING_POLICY=decoupled` will not keep the names of the objects in sync.\n\nS3 server-side encryption can be enabled using the parameter `S3_SERVER_ENCRYPT=[0|1]` (default=0=off).  This is not the same as HTTPS, and implies that the data will be stored on disk encrypted.\nTo encrypt during the network transport to S3, use `S3_PROTO=HTTPS` (the default)\n\nThe `S3_RETRY_COUNT` defines the number of retries for a request after a retryable failure.  The default is 3 retries.\nThe `S3_WAIT_TIME_SECONDS` defines the initial number of seconds between retries.  The default is 2s.  This wait time will double on each successive failure until the `S3_MAX_WAIT_TIME_SECONDS` is reached.  If this is set to 0, there will be no wait between retries.  (Note:  For backward compatibility with previous releases, `S3_WAIT_TIME_SEC` is also valid but `S3_WAIT_TIME_SECONDS` will take priority.)\nThe `S3_MAX_WAIT_TIME_SECONDS` is the maximum wait value during successive failures.  The default is 30s.  If this is set to 0, there will be no wait between retries.  (Note:  For backward compatibility with previous releases, `S3_MAX_WAIT_TIME_SEC` is also valid but `S3_MAX_WAIT_TIME_SECONDS` will take priority.)\nThe `S3_NON_DATA_TRANSFER_TIMEOUT_SECONDS` defines the timeout value used for S3_complete_multipart_upload and S3_delete_object.  The default is 300s.  (Note:  This has been added because in some cases with very large files these take a long time to complete and the data transfer thresholds do not apply to these.)\n\n### Modifying your resource configuration\n\n`location` is the hostname of your iRODS server.  Modify this field as `host` when using `iadmin`.\n\n`vault` contains the S3 bucket name and prefix (`/s3-irods-bucket-name/prefix/in/bucket`. Modify this field as `path` when using `iadmin`.\n\nModify the `context` field to update any of the resource-specific settings (`S3_DEFAULT_HOSTNAME`, `S3_AUTH_FILE`, etc.)\n\nConfirm `S3_REGIONNAME` matches the region for your bucket and is in the form `us-east-1`.\n\n### Using the S3 plugin in cacheless mode\n\nThe S3 plugin may be used in cacheless mode.  In this case the resource can be standalone and does not require an associated cache and compound resource.  This is still being actively developed and not all features that exist for cache mode have been implemented at this time.  The following have not been implemented or have not been tested at this time.\n\n* Multiple hosts in a comma-separated list in `S3_DEFAULT_HOSTNAME`.\n\nAn additional flag called `HOST_MODE` is used to enable cacheless mode.  The default value for this is `archive_attached` which provides the legacy functionality.  The valid settings are as follows:\n\n* `archive_attached` - Legacy functionality.  Resource must be a child of a compound resource (parent/child context of archive) and must have a cache resource associated with it.\n* `cacheless_attached` - Resource does not require a compound resource or a cache.  The resource remains tagged to the server defined in the `resc_net` property.  Any requests to this resource will be redirected to that server.\n* `cacheless_detached` - Same as above but the resource is not uniquely pinned to a specific resource server.  Any resource server may fulfill a request.  This requires that all resource servers have network access to the S3 region.  (Note:  The cacheless S3 resource's host must be resolvable to an iRODS server.)\n\nCacheless mode has a few extra configuration parameters in addition to HOST_MODE.\n\n-   `CIRCULAR_BUFFER_SIZE` - The plugin uses a circular buffer to store data while it is being streamed to S3.  The size of the circular buffer is CIRCULAR_BUFFER_SIZE * S3_MPU_CHUNK.  The default value is 4 so if the S3_MPU_CHUNK is the default of 5MB the circular buffer size will be 20MB.  CIRCULAR_BUFFER_SIZE must be at least 2.  If a size is set lower than 2 then it will default to 2.\n-   `CIRCULAR_BUFFER_TIMEOUT_SECONDS` - The number of seconds the plugin will wait when waiting to read or write data from the circular buffer.  The default is 180s.\n-   `S3_CACHE_DIR` - This is the directory where temporary cache files are located in cases where a cache file is required.  (See below.)  The default is `/tmp`.\n\nThe following is an example of how to configure a `cacheless_attached` S3 resource:\n\n```\niadmin mkresc s3resc s3 $(hostname):/s3-irods-bucket-name/prefix/in/bucket \"S3_DEFAULT_HOSTNAME=s3.us-east-1.amazonaws.com;S3_AUTH_FILE=/var/lib/irods/s3.keypair;S3_REGIONNAME=us-east-1;S3_RETRY_COUNT=1;S3_WAIT_TIME_SECONDS=3;S3_PROTO=HTTP;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached\"\n```\n\nSome configuration settings have special meaning when the resource is in cacheless mode.\n-   When S3_ENABLE_MPU = 0, a cache file will be used when the S3 plugin receives parallel uploads from iRODS.\n-   When iRODS is using parallel transfer but each transfer part is less than S3_MPU_CHUNK, a cache file will be used\n-   The S3_MPU_THREADS setting is only used when flushing a cache file to S3.  In streaming mode iRODS controls the number of transfer threads that are used.\n\n#### Cache Rules When Using Cacheless Mode\n\nCare was taken to limit the use of a cache file when cacheless mode is enabled.  However, there are scenarios where a cache file is required.  The following explains when a cache file is required or when cacheless streaming is performed.\n\n1.  All objects opened in read-only mode (including `iget`) will be cacheless as S3 allows random access reads on S3 objects.\n2.  All `iput` and `irepl` will stream without a cache except in the following two cases:\n-   iRODS is performing a parallel transfer but multipart uploads is disabled.\n-   iRODS is performing a parallel transfer but each part size \u003c S3_MPU_CHUNK size.\n\nIn the cases where a cache file must be used, the base directory for the cache files can be set using the `S3_CACHE_DIR` parameter in the context string.  If it is not set, a directory under `/tmp` will be created and used.  The cache files are transient and are removed once the data object is closed.\n\n### Expectations on clients using the s3_transport/dstream directly when the put_repl_flag is set to true\n\nClients using s3_transport/dstream must set the put_repl_flag to true to use cacheless streaming.  In this case, the s3_transport has some expectations on the behavior of the client.  If these are not followed the results are undefined and the transfers will likely fail.\n\n1.  If the number_of_client_transfer_threads is set to 1, a single thread will send all of the bytes starting from the first byte to the last byte in sequential order.\n2.  If the number_of_client_transfer_threads is greater than 1:\n-   Each thread must stream the bytes from the source at the offset of thread_number * (object_size / number_of_client_transfer_threads).\n-   The last thread will send the extra bytes.\n-   Each thread will call s3_transport_ptr-\u003eset_part_size(n) where n is the size of its part.\n-   The bytes for each thread will be sent sequentially and all bytes will be sent.\n\nThis conforms to the way iput breaks up the files when doing parallel writes.  The reason for these is so that the s3_transport object can always determine the part number by the object size and offset.\n\n### Using the S3 plugin in archive mode (under compound)\n\nThe S3 plugin may be used in archive mode. In this case the resource requires an associated cache and compound resource and configured as follows:\n\n```\nirods@hostname $ iadmin mkresc compResc compound\nirods@hostname $ iadmin mkresc cacheResc unixfilesystem \u003chostname\u003e:\u003c/full/path/to/Vault\u003e\nirods@hostname $ iadmin mkresc archiveResc s3 \u003chostname\u003e:/\u003cs3BucketName\u003e/irods/Vault \"S3_DEFAULT_HOSTNAME=s3.amazonaws.com;S3_AUTH_FILE=\u003c/full/path/to/AWS.keypair\u003e;S3_RETRY_COUNT=\u003cnum reconn tries\u003e;S3_WAIT_TIME_SECONDS=\u003cwait between retries\u003e;S3_PROTO=\u003cHTTP|HTTPS\u003e\"\nirods@hostname $ iadmin addchildtoresc compResc cacheResc cache\nirods@hostname $ iadmin addchildtoresc compResc archiveResc archive\nirods@hostname $ iput -R compResc foo.txt\nirods@hostname $ ireg -R archiveResc /\u003cs3BucketName\u003e/full/path/in/bucket /full/logical/path/to/dataObject\n```\n\nNote the use of the `cache` and `archive` contextStrings on the `addchildtoresc` commands.  These inform the parent compound resource which child is serving in which role.  The S3 resource is set to `archive` and a POSIX-capable resource must be set to `cache`.\n\n### Enabling Direct Checksum Reads from S3 Provider\n\nWhen iRODS needs to calculate a hash or checksum on an object, the default behavior is to read the full object for this calculation. Some S3 providers automatically calculate a hash or checksum on object upload. Reading this directly from the S3 provider can significantly improve performance.\n\nTo take advantage of this feature, all of the following must be true:\n\n1. The S3 resource context string must include the `ENABLE_DIRECT_CHECKSUM_READ=1` option.\n2. The requested checksum must be returned in the `GetObjectAttributes` API call.\n3. The `ChecksumType` returned from `GetObjectAttributes` must be \"FULL_OBJECT\".\n\nIf any of these conditions are not satisfied or an error occurs while attempting to read the checksum, iRODS will default to the normal behavior of reading the full object to calculate the checksum.\n\nAWS by default calculates and stores the CRC64/NVME checksum. To make sure that this is the checksum requested by iRODS, one of the following should be true:\n\n- If the client is validating the checksum (for example \"iput -K\") then the client should set the `irods_default_hash_scheme` to \"crc64nvme\".\n- If the client is not validating the checksum (for example \"iput -k\" or \"ichksum\") then the server's `default_hash_scheme` should be set to \"crc64nvme\".\n- Refer to [Checksum Configuration](https://docs.irods.org/5.0.2/system_overview/configuration/#checksum-configuration) for more detail on configuring checksums in iRODS.\n\n\n### Example of a baseline resource configuration\n```\n$ ilsresc s3resc\nresource name: s3resc\nid: 10017\nzone: tempZone\ntype: s3\nclass: cache\nlocation: irods3.example.org\nvault: /s3-irods-bucket-name/prefix/in/bucket\nfree space:\nfree space time: : Never\nstatus:\ninfo:\ncomment:\ncreate time: 01575329511: 2019-12-02.23:31:51\nmodify time: 01576687386: 2019-12-18.16:43:06\ncontext: S3_DEFAULT_HOSTNAME=s3.us-east-1.amazonaws.com;S3_AUTH_FILE=/var/lib/irods/s3.keypair;S3_REGIONNAME=us-east-1;S3_RETRY_COUNT=1;S3_WAIT_TIME_SECONDS=3;S3_PROTO=HTTP;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached\nparent:\nparent context:\n```\n\n### Increasing/Decreasing Plugin Logging\n\nTo increase or decrease the logging level for the S3 plugin, add an `s3_resource_plugin` entry into the `log_level` stanza in /etc/irods/server_config.json.  The following is an example entry that increases the logging level to include debug logging.  The default level is info.\n\n```\n\"log_level\": {\n    \"agent\": \"info\",\n    \"agent_factory\": \"info\",\n    \"api\": \"info\",\n    \"authentication\": \"info\",\n    \"database\": \"info\",\n    \"delay_server\": \"info\",\n    \"legacy\": \"info\",\n    \"microservice\": \"info\",\n    \"network\": \"info\",\n    \"resource\": \"info\",\n    \"rule_engine\": \"info\",\n    \"server\": \"info\",\n    \"s3_resource_plugin\": \"debug\"\n},\n```\n\n## Troubleshooting\n\n### Handling \"Reduce Your Request Rate\" Errors\n\nOn operations that require calculating a checksum for an S3 object at rest, iRODS must read the object's data to calculate the checksum. The default buffer size for these reads is 1 MiB. Each of these read requests map to a separate GetObject request to S3 with a range of bytes. This can exceed the request rate that the S3 provider allows.\n\nTo resolve this problem, the read buffer size can be increased by updating the `checksum_read_buffer_size_in_bytes` value in the `advanced_settings` stanza of server_config.json. This change should be made for all servers that can serve requests to that S3 resource.\n\nThe following is an example of increasing this buffer size to 10 MiB:\n\n```\n    \"advanced_settings\": {\n        ...\n        \"checksum_read_buffer_size_in_bytes\": 10485760\n        ...\n```\n\n## Using this plugin with Google Cloud Storage (GCS)\n\nThis plugin has been manually tested to work with Google Cloud Storage, with some caveats.\n\n1. GCS treats bucket names with dots as domain names.  These must be verified.  See [Domain-named Bucket Verification] (https://cloud.google.com/storage/docs/domain-name-verification).\n2. If an object is uploaded using multipart uploads, subsequent calls to CopyObject fail.  CopyObject is used with `imv` when consistent naming is used.  As a workaround, GCS resources should be configured with any one of the following options:\n-   Disable MPU upload: `S3_ENABLE_MPU=0`\n-   Set archive naming policy to decoupled: `ARCHIVE_NAMING_POLICY=decoupled`\n-   Disable CopyObject:  `S3_ENABLE_COPYOBJECT=0`\n\nMake sure to:\n\n-   Set `S3_DEFAULT_HOSTNAME=storage.googleapis.com`\n-   Set `S3_PROTO=HTTPS`\n-   Set 'S3_REGIONNAME` correctly - note that AWS has `us-east-1` and GCS has `us-east4` (without the second hyphen)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_resource_plugin_s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Firods%2Firods_resource_plugin_s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_resource_plugin_s3/lists"}