{"id":18485909,"url":"https://github.com/biomapas/b.cfns3largedeployment","last_synced_at":"2026-05-20T07:36:00.618Z","repository":{"id":42080955,"uuid":"403952968","full_name":"Biomapas/B.CfnS3LargeDeployment","owner":"Biomapas","description":"Custom AWS CDK resource that enables larger than usual deployments to S3.","archived":false,"fork":false,"pushed_at":"2022-04-13T11:34:37.000Z","size":63,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-13T21:49:26.465Z","etag":null,"topics":["aws","aws-cdk","aws-cdk-python","aws-cloudformation","aws-s3","deployment","s3-bucket","s3-storage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Biomapas.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-07T11:30:58.000Z","updated_at":"2021-12-17T08:10:01.000Z","dependencies_parsed_at":"2022-08-12T04:20:46.108Z","dependency_job_id":null,"html_url":"https://github.com/Biomapas/B.CfnS3LargeDeployment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Biomapas/B.CfnS3LargeDeployment","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Biomapas%2FB.CfnS3LargeDeployment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Biomapas%2FB.CfnS3LargeDeployment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Biomapas%2FB.CfnS3LargeDeployment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Biomapas%2FB.CfnS3LargeDeployment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Biomapas","download_url":"https://codeload.github.com/Biomapas/B.CfnS3LargeDeployment/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Biomapas%2FB.CfnS3LargeDeployment/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266053884,"owners_count":23869499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-cdk","aws-cdk-python","aws-cloudformation","aws-s3","deployment","s3-bucket","s3-storage"],"created_at":"2024-11-06T12:47:06.392Z","updated_at":"2026-05-20T07:36:00.535Z","avatar_url":"https://github.com/Biomapas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Pipeline](https://github.com/Biomapas/B.CfnS3LargeDeployment/workflows/Pipeline/badge.svg?branch=main)\n\n# B.CfnS3LargeDeployment\n\n**b-cfn-s3-large-deployment** - AWS CDK custom resource that handles large files deployment to S3 bucket.\n\n### Description\n\nThis custom resource deploys local files or S3 bucket objects to a destination bucket retaining their file-system \nhierarchy.\n\nTwo types of deployment sources are available:\n\n- `BucketDeploymentSource` - uses another S3 bucket object(-s) as source for the deployment to a destination bucket. \n  Only files up to 5TB are supported due to S3 bucket limitations;\n- `AssetDeploymentSource` - uses `aws-cdk.aws-s3-assets` lib to deploy local files as .zip files to assets bucket from \n  which extracted contents are moved to the destination bucket. Asset files more than 2GB in size are not supported.\n\nSee \"Known limits\" sections below for more information on this resource limitations.\n\nThis resource implementation is based on GitHub pull-request https://github.com/aws/aws-cdk/pull/15220.\n\n### Remarks\n\n[Biomapas](https://biomapas.com) aims to modernise life-science\nindustry by sharing its IT knowledge with other companies and\nthe community.\n\n### Related technology\n\n- Python \u003e= 3.8\n- Amazon Web Services (AWS)\n\n### Assumptions\n\nThe project assumes that the person working with it have basic knowledge in python\nprogramming.\n\n### Useful sources\n\nSee code documentation for any additional sources and references. Also see `aws-cdk.s3-deployment` library for more \ninformation as this implementation is based on work done there.\n\n### Install\n\nUse the package manager pip to install this package. This project is not in the PyPi\nrepository yet. Install directly from source or PyPI.\n\n```bash\npip install .\n```\n\nOr\n\n```bash\npip install b-cfn-s3-large-deployment\n```\n\n### Usage \u0026 Examples\n\nThis AWS CloudFormation custom resource is used pretty much the same way as any other resource. Simply initialize it \nwithin any valid CDK scope giving it unique name/id, providing source(-s) and the destination for the deployment. \n\nThe deployment of files depends on AWS Lambda's `/tmp` directory and its size limits. For large files `/tmp` directory \nsize can be configured using Ephemeral storage (`DeploymentProps.ephemeral_storage_size`) supported by AWS Lambda \nfunctions. \n\nOptionally, if there's a need for even larger files deployment than what AWS Lambda's `/tmp` directory supports, \nsetting the `DeploymentPops.use_efs` and `DeploymentPops.efs_props` fields, AWS Elastic File Storage (EFS) can be \nenabled to allow such files handling.\n\nA simple example of `S3LargeDeploymentResource` usage is shown below:\n\n```python\nfrom aws_cdk.core import App, Stack, Construct\nfrom aws_cdk.aws_s3 import Bucket\n\nfrom b_cfn_s3_large_deployment.resource import S3LargeDeploymentResource\nfrom b_cfn_s3_large_deployment.deployment_props import DeploymentProps\nfrom b_cfn_s3_large_deployment.deployment_source import AssetDeploymentSource, BucketDeploymentSource\n\n\nclass ExampleStack(Stack):\n    def __init__(self, scope: Construct):\n        super().__init__(...)\n\n        S3LargeDeploymentResource(\n            scope=self,\n            name='ExampleLargeDeployment',\n            sources=[\n                AssetDeploymentSource(path='/path/to/your/local/directory'),\n                AssetDeploymentSource(path='/path/to/your/local/zip/file.zip'),\n                BucketDeploymentSource(\n                  bucket=..., \n                  zip_object_key='your-source-bucket-object-key'\n                ),\n                ...\n            ],\n            destination_bucket=Bucket(...),\n            props=DeploymentProps(...)\n        )\n        ...\n\n        \napp = App()\nExampleStack(app, 'ExampleStack')\n\napp.synth()\n```\n\nHere, three types of supported sources were used:\n\n1. whole, local directory given as a path, which is then deployed to the assets bucket as a .zip object: \n\n    ```python \n    AssetDeploymentSource(path='/path/to/your/local/directory')\n    ```\n   \n2. single .zip file given as a path, which is then deployed to the assets bucket:\n    \n    ```python \n    AssetDeploymentSource(path='/path/to/your/local/zip/file.zip')\n    ```\n\n3. Single .zip S3 object found in the source bucket, given as an object key. No further pre-processing is \n  applied in this case:\n    \n    ```python\n    BucketDeploymentSource(\n       bucket=...,\n       zip_object_key='your-source-bucket-object-key'\n    )\n    ```\n\nIn all of these cases, final, source .zip objects are extracted inside `S3LargeDeploymentResource`'s handler \nfunction storage and the available contents are then deployed to the configured destination. This is all done, while \nmaintaining original file structure of source contents.\n\n### Known limits\n\n- `aws_cdk.aws_s3_assets.Asset` supports up to 2GB/asset (limited by NodeJS implementation).\n- S3 bucket supports up to 5TB objects.\n\n### Contribution\n\nFound a bug? Want to add or suggest a new feature? Contributions of any kind are gladly\nwelcome. Contact us, create a pull-request or an issue ticket.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiomapas%2Fb.cfns3largedeployment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbiomapas%2Fb.cfns3largedeployment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiomapas%2Fb.cfns3largedeployment/lists"}