{"id":20358066,"url":"https://github.com/irods/irods_client_s3_api","last_synced_at":"2025-08-09T03:57:44.431Z","repository":{"id":66171936,"uuid":"385723398","full_name":"irods/irods_client_s3_api","owner":"irods","description":"C++ S3 API for iRODS","archived":false,"fork":false,"pushed_at":"2025-03-26T19:59:57.000Z","size":2831,"stargazers_count":0,"open_issues_count":35,"forks_count":6,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-12T03:15:04.923Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/irods.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-13T20:08:22.000Z","updated_at":"2025-03-26T20:00:02.000Z","dependencies_parsed_at":"2024-01-25T02:49:05.625Z","dependency_job_id":"4c8dd0ff-6612-4c82-811d-d6741136076d","html_url":"https://github.com/irods/irods_client_s3_api","commit_stats":null,"previous_names":["irods/irods_client_s3_cpp"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_client_s3_api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_client_s3_api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_client_s3_api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/irods%2Firods_client_s3_api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/irods","download_url":"https://codeload.github.com/irods/irods_client_s3_api/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248510001,"owners_count":21116130,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T23:25:13.255Z","updated_at":"2025-08-09T03:57:44.408Z","avatar_url":"https://github.com/irods.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# iRODS S3 API\n\nA project that presents an iRODS Zone as S3 compatible storage.\n\n![S3 API network diagram](s3_api_diagram.png)\n\nImplements a subset of the Amazon S3 API:\n  - https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html\n\nThis API currently supports:\n  - [x] AbortMultipartUpload\n  - [x] CopyObject\n  - [x] CompleteMultipartUpload\n  - [x] CreateMultipartUpload\n  - [x] DeleteObject\n  - [x] DeleteObjects\n  - [x] GetBucketLocation\n  - [x] GetObject\n  - GetObjectAcl ?\n  - [x] GetObjectLockConfiguration\n  - [x] GetObjectTagging\n  - [x] HeadBucket\n  - [x] HeadObject\n  - [x] ListBuckets\n  - ListObjects ?\n  - [x] ListObjectsV2\n  - [x] PutObject\n  - PutObjectAcl ?\n  - PutObjectTagging ?\n  - [x] UploadPart\n  - UploadPartCopy ?\n\nThe goal is to support the equivalent of:\n - ils - `aws s3 ls s3://bucketname/a/b/c/`\n - iput - `aws s3 cp localfile s3://bucketname/a/b/c/filename`\n - iget - `aws s3 cp s3://bucketname/a/b/c/filename localfile`\n - irm - `aws s3 rm s3://bucketname/a/b/c/filename`\n - imv - `aws s3 mv s3://bucketname/a/b/c/filename1 s3://bucketname/a/b/c/filename2`\n\n# Limitations / What's Missing\n\n## Multipart\n\nMultipart has not been implemented for copy operations where `x-amz-copy-source` and `x-amz-copy-source-range` are used.\nWhen performing a copy from one iRODS file to another, multipart should be disabled.\n\nSee [Disabling Multipart](#disabling-multipart) for details.\n\nMultipart uploads of a local file is supported.\n\n## Tagging\n\niRODS has its own metadata system, however it is not especially clear how it should map to S3 metadata, so it is not\nincluded at the moment.\n\n## Paging\n\nPaging requires engineering work to provide paging through lists of objects efficiently, so right now this API does not\nattempt to paginate its output for things such as listobjects.\n\n## Checksum handling\n\nAmazon S3 provides many ways to communicate checksums for the data as received by the server. iRODS provides MD5 checksums, \nhowever this API does not use that to verify data objects created through PutObject.\n\n## ETags\n\nETags are not provided for or used consistently.\n\n## Versioning\n\nVersioning is not supported at this time.\n\n# Docker\n\nThis project provides two Dockerfiles, one for building and one for running the application.\n\n**IMPORTANT: All commands in the sections that follow assume you are located in the root of the repository.**\n\n## The Builder Image\n\nThe builder image is responsible for building the iRODS S3 API package. Before you can use it, you must build the image. To do that, run the following:\n```bash\ndocker build -t irods-s3-api-builder -f irods_builder.Dockerfile .\n```\n\nWith the builder image in hand, all that's left is to compile the source code for the S3 API project. The builder image is designed to compile code sitting on your machine. This is important because it gives you the ability to build any fork or branch of the project.\n\nBuilding the package requires mounting the project into the container at the appropriate location. The command you run should look similar to the one below. Don't forget to create the directory which will hold your package!\n```bash\ndocker run -it --rm \\\n    -v /path/to/irods_client_s3_api:/s3_api_source:ro \\\n    -v /path/to/packages_directory:/packages_output \\\n    irods-s3-api-builder\n```\n\nIf everything succeeds, you will have a DEB package in the local directory you mapped to **/packages_output**.\n\n### Using a build cache\n\nIn order to keep build artifacts around for faster iteration on build times, include another volume mount for `/_build_s3_api` into which build artifacts can be stored in the host filesystem:\n```bash\ndocker run -it --rm \\\n    -v /path/to/irods_client_s3_api:/s3_api_source:ro \\\n    -v /path/to/build_directory:/_build_s3_api \\\n    -v /path/to/packages_directory:/packages_output \\\n    irods-s3-api-builder\n```\n\n## The Runner Image\n\nThe runner image is responsible for running the iRODS S3 API. Building the runner image requires the DEB package for the iRODS S3 API to exist on the local machine. See the previous section for details on generating the package.\n\nTo build the image, run the following command:\n```bash\ndocker build -t irods-s3-api-runner -f irods_runner.Dockerfile /path/to/packages/directory\n```\n\nIf all goes well, you will have a containerized iRODS S3 API server! You can verify this by checking the version information. Below is an example.\n```bash\n$ docker run -it --rm irods-s3-api-runner -v\nirods_s3_api \u003cversion\u003e-\u003cbuild_sha\u003e\n```\n\n## Launching the Container\n\nTo run the containerized server, you need to provide a configuration file at the correct location. If you do not have a configuration file already, see [Configuration](#configuration) for details.\n\nTo launch the server, run the following command:\n```bash\ndocker run -d --rm --name irods_s3_api \\\n    -v /path/to/config/file:/config.json:ro \\\n    -p 9000:9000 \\\n    irods-s3-api-runner\n```\n\nThe first thing the server will do is validate the configuration. If the configuration fails validation, the server will exit immediately. If the configuration passes validation, then congratulations, you now have a working iRODS S3 API server!\n\nYou can view the log output using `docker logs -f` or by passing `-it` to `docker run` instead of `-d`.\n\nIf for some reason the default schema file is not sufficient, you can instruct the iRODS S3 API to use a different schema file. See the following example.\n```bash\n# Generate the default JSON schema.\ndocker run -it --rm irods-s3-api-runner --dump-default-jsonschema \u003e schema.json\n\n# Tweak the schema.\nvim schema.json\n\n# Launch the server with the new schema file.\ndocker run -d --rm --name irods_s3_api \\\n    -v /path/to/config/file:/config.json:ro \\\n    -v ./schema.json:/jsonschema.json:ro \\\n    -p 9000:9000 \\\n    irods-s3-api-runner \\\n    --jsonschema-file /jsonschema.json\n```\n\n## Stopping the Container\n\nIf the container was launched with `-it`, use **CTRL-C** or `docker container stop \u003ccontainer_name\u003e` to shut it down.\n\nIf the container was launched with `-d`, use `docker container stop \u003ccontainer_name\u003e`.\n\n# Building and running without Docker\n\n## Build Dependencies\n\n- iRODS development package\n- iRODS externals package for boost\n- iRODS externals package for nlohmann-json\n- iRODS externals package for spdlog\n- Curl development package\n- OpenSSL development package\n\n## Building from source\n\nThis project relies on git submodules and Docker for building the server.\n\nBefore the server can be built, you must download the appropriate git submodules. You can do that by running the following:\n\n```bash\ngit submodule update --init --recursive\n```\n\nTo build, follow the normal CMake steps.\n\n```bash\nmkdir build # Preferably outside of the repository\ncd build\ncmake /path/to/repository\nmake package # Use -j to use more parallelism.\n```\n\nUpon success, you should have an installable package.\n\nIf you run into issues, try checking if the git submodules exist on your machine.\n\n## Running without Docker\n\nThe server has three requirements that must be satisfied before launch. They are listed as follows:\n- Python 3 must be installed\n- Python 3 jsonschema module must be installed\n- A valid configuration file for the iRODS S3 API server\n\nThe Python requirements can be satisfied by using your OS's package manager and `python3 -m pip`. We'll leave that as an exercise for the reader.\n\nNow, you need a configuration file for the iRODS S3 API. See [Configuration](#configuration) for details on how to create one.\n\nWith the requirements satisfied, run the following to launch the server:\n```bash\nirods_s3_api /path/to/config.json\n```\n\nTo stop the server, you can use **CTRL-C** or send **SIGINT** or **SIGTERM** to the process.\n\n# Configuration\n\nBefore you can run the server, you'll need to create a configuration file.\n\nYou can generate a configuration file by running the following:\n```bash\nirods_s3_api --dump-config-template \u003e config.json\n```\n\n**IMPORTANT: `--dump-config-template` does not produce a fully working configuration. It must be updated before it can be used.**\n\n## Configuration File Structure\n\nThe JSON structure below represents the default configuration.\n\nNotice how some of the configuration values are wrapped in angle brackets (e.g. `\"\u003cstring\u003e\"`). These are placeholder values that must be updated before launch.\n\n**IMPORTANT: The comments in the JSON structure are there for explanatory purposes and must not be included in your configuration. Failing to follow this requirement will result in the server failing to start up.**\n\n```js\n{\n    // Defines options that affect how the client-facing component of the\n    // server behaves.\n    \"s3_server\": {\n        // The hostname or IP address to bind.\n        // \"0.0.0.0\" instructs the server to listen on all network interfaces.\n        \"host\": \"0.0.0.0\",\n\n        // The port used to accept incoming client requests.\n        \"port\": 9000,\n\n        // The minimum log level needed before logging activity.\n        //\n        // The following values are supported:\n        // - trace\n        // - debug\n        // - info\n        // - warn\n        // - error\n        // - critical\n        \"log_level\": \"info\",\n\n        // Defines the set of plugins to load.\n        \"plugins\": {\n            //\n            // Each key corresponds to a plugin's .so file name, minus the\n            // \"lib\" prefix.\n            //\n\n            \"static_bucket_resolver\": {\n                // The internal name assigned to the plugin.\n                \"name\": \"static_bucket_resolver\",\n\n                // Defines the mapping between bucket names and iRODS\n                // collections.\n                \"mappings\": {\n                    \"\u003cbucket_name\u003e\": \"/path/to/collection\"\n                }\n            },\n\n            \"static_authentication_resolver\": {\n                // The internal name assigned to the plugin.\n                \"name\": \"static_authentication_resolver\",\n\n                // Defines information for resolving an S3 username to an\n                // iRODS username.\n                \"users\": {\n                    // Maps \u003cs3_username\u003e to a specific iRODS user.\n                    // Each iRODS user that intends to access the S3 API must\n                    // have at least one entry.\n                    \"\u003cs3_username\u003e\": {\n                        // The iRODS username to resolve to.\n                        \"username\": \"\u003cstring\u003e\",\n\n                        // The secret key used to authenticate with the S3\n                        // API for this user.\n                        \"secret_key\": \"\u003cstring\u003e\"\n                    }\n                }\n            }\n        },\n\n        // Defines the region the server will report as being a member of.\n        \"region\": \"us-east-1\",\n\n        // Defines the location where part files are temporarily stored\n        // on the irods_s3_api server before being streamed to iRODS. \n        \"multipart_upload_part_files_directory\": \"/tmp\",\n\n        // Defines options that affect various authentication schemes.\n        \"authentication\": {\n            // The amount of time that must pass before checking for expired\n            // bearer tokens.\n            \"eviction_check_interval_in_seconds\": 60,\n\n            // Defines options for the \"Basic\" authentication scheme.\n            \"basic\": {\n                // The amount of time before a user's authentication\n                // token expires.\n                \"timeout_in_seconds\": 3600\n            }\n        },\n\n        // Defines options that affect how client requests are handled.\n        \"requests\": {\n            // The number of threads dedicated to servicing client requests.\n            // When adjusting this value, consider adjusting \"background_io/threads\"\n            // and \"irods_client/connection_pool/size\" as well.\n            \"threads\": 3,\n\n            // The maximum size allowed for the body of a request.\n            \"max_size_of_request_body_in_bytes\": 8388608,\n\n            // The amount of time allowed to service a request. If the timeout\n            // is exceeded, the client's connection is terminated immediately.\n            \"timeout_in_seconds\": 30\n        },\n\n        // Defines options that affect tasks running in the background.\n        // These options are primarily related to long-running tasks.\n        \"background_io\": {\n            // The number of threads dedicated to background I/O.\n            \"threads\": 6\n        }\n    },\n\n    // Defines iRODS connection information.\n    \"irods_client\": {\n        // The hostname or IP of the target iRODS server.\n        \"host\": \"\u003cstring\u003e\",\n\n        // The port of the target iRODS server.\n        \"port\": 1247,\n\n        // The zone of the target iRODS server.\n        \"zone\": \"\u003cstring\u003e\",\n\n        // Defines options for secure communication with the target iRODS server.\n        \"tls\": {\n            // Controls whether the client and server communicate using TLS.\n            //\n            // The following values are supported:\n            // - CS_NEG_REFUSE:    Do not use secure communication.\n            // - CS_NEG_REQUIRE:   Demand secure communication.\n            // - CS_NEG_DONT_CARE: Let the server decide.\n            \"client_server_policy\": \"CS_NEG_REFUSE\",\n\n            // The file containing trusted CA certificates in PEM format.\n            //\n            // Note that the certificates in this file are used in conjunction\n            // with the system default trusted certificates.\n            \"ca_certificate_file\": \"\u003cstring\u003e\",\n\n            // The file containing the server's certificate chain.\n            //\n            // The certificates must be in PEM format and must be sorted\n            // starting with the subject's certificate (actual client or server\n            // certificate), followed by intermediate CA certificates if\n            // applicable, and ending at the highest level (root) CA.\n            \"certificate_chain_file\": \"\u003cstring\u003e\",\n\n            // The file containing Diffie-Hellman parameters.\n            \"dh_params_file\": \"\u003cstring\u003e\",\n\n            // Defines the level of server certificate authentication to\n            // perform.\n            //\n            // The following values are supported:\n            // - none:     Authentication is skipped.\n            // - cert:     The server verifies the certificate is signed by\n            //             a trusted CA.\n            // - hostname: Equivalent to \"cert\", but also verifies the FQDN\n            //             of the iRODS server matches either the common\n            //             name or one of the subjectAltNames.\n            \"verify_server\": \"cert\"\n        },\n\n        // Controls how the S3 API communicates with the iRODS server.\n        //\n        // When set to true, the following applies:\n        // - Only APIs supported by the iRODS 4.2 series will be used.\n        // - Connection pool settings are ignored.\n        // - All HTTP requests will be served using a new iRODS connection.\n        //\n        // When set to false, the S3 API will take full advantage of the\n        // iRODS server's capabilities.\n        //\n        // This option should be used when the S3 API is configured to\n        // communicate with an iRODS 4.2 server.\n        \"enable_4_2_compatibility\": false,\n\n        // The credentials for the rodsadmin user that will act as a proxy\n        // for all authenticated users.\n        \"proxy_admin_account\": {\n            \"username\": \"\u003cstring\u003e\",\n            \"password\": \"\u003cstring\u003e\"\n        },\n\n        // Defines options for the connection pool.\n        \"connection_pool\": {\n            // The number of connections in the pool.\n            \"size\": 6,\n\n            // The amount of time that must pass before a connection is\n            // renewed (i.e. replaced).\n            \"refresh_timeout_in_seconds\": 600,\n\n            // The number of times a connection can be fetched from the pool\n            // before it is refreshed.\n            \"max_retrievals_before_refresh\": 16,\n\n            // Instructs the connection pool to track changes in resources.\n            // If a change is detected, all connections will be refreshed.\n            \"refresh_when_resource_changes_detected\": true\n        },\n\n        // The resource to target for all write operations.\n        \"resource\": \"\u003cstring\u003e\",\n\n        // The buffer size used to read objects from the client\n        // and write to iRODS.\n        \"put_object_buffer_size_in_bytes\": 8192,\n\n        // The buffer size used to read objects from iRODS\n        // and send to the client.\n        \"get_object_buffer_size_in_bytes\": 8192\n    }\n}\n```\n\n# Connecting with Botocore\n\nAs a simple example, this is how you pass that in through botocore, a library from Amazon that provides S3 connectivity.\n\n```python\nimport botocore.session\n\nsession = botocore.session.get_session()\nclient = session.create_client(\"s3\",\n                               use_ssl=False,\n                               endpoint_url=\"http://127.0.0.1:8080\",\n                               aws_access_key_id=\"\u003cusername\u003e\",\n                               aws_secret_access_key=\"\u003csecret key\u003e\")\n```\n\n# Disabling Multipart\n\nMultipart copies are not supported at this time.  Therefore, multipart must be disabled in the client.\n\n## Disabling Multipart for AWS CLI\n\nFor AWS CLI, multipart can be disabled by setting an arbitrarily large multipart threshold.  Since 5 GB is the largest single part allowed by AWS, this is a good choice.\n\nTo disable multipart, set the `multipart_threshold` in the ~/.aws/credentials file for the profile in question.  For example, you could create a profile called `irods_s3_no_multipart` with the following in the credentials file.\n\n```\n[irods_s3_no_multipart]\naws_access_key_id = key1 \naws_secret_access_key = secret_key1\ns3 =\n    multipart_threshold = 5GB\n```\n\nTo use this with the AWS CLI commands, use the `--profile` flag.  Example: `aws --profile irods_s3_no_multipart`.\n\n## Example for Boto3\n\nTo set the multipart threshold with a boto3 client, do the following: \n\n```python\nconfig = TransferConfig(multipart_threshold=5*1024*1024*1024)\nself.boto3_client.upload_file(put_filename, bucket_name, key, Config=config)\n```\n\n## Example of MinIO mc client\n\nThe `mc cp` command has a `--disable-multipart` option.  Here is an example of a copy with a `myminio` alias:\n\n```bash\nmc cp --disable-multipart put_file myminio/bucket_name/put_filename\n```\n\n*Note: MinIO client uses aliases to group URL and keys. Refer to the `mc alias` command for information on setting, listing, and removing aliases.*\n\n# Running Tests\n\nRun the following commands to run the test suite.\n\n```bash\ncd tests/docker\n./build_s3_api.sh\ndocker compose build\ndocker compose run client\n```\n\n*Note: If you get an error like `'name' does not match any of the regexes: '^x-'` then you will need to upgrade your version of docker compose.*\n\nThe test output will appear in the terminal.  Once the tests complete run the following to cleanup:\n\n```bash\ndocker compose down\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_client_s3_api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Firods%2Firods_client_s3_api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firods%2Firods_client_s3_api/lists"}