{"id":13740354,"url":"https://github.com/vt-digital-libraries-platform/aws-batch-iiif-generator","last_synced_at":"2025-05-08T20:31:12.230Z","repository":{"id":56594936,"uuid":"196035356","full_name":"vt-digital-libraries-platform/aws-batch-iiif-generator","owner":"vt-digital-libraries-platform","description":"A project that demonstrates the use of AWS Batch to create IIIF tiles and manifests. It is used for VTDLP Derivative Service ","archived":false,"fork":false,"pushed_at":"2024-09-24T17:14:33.000Z","size":15398,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-11-15T10:41:52.777Z","etag":null,"topics":["aws-batch","docker","iiif","iiif-image","iiif-s3-docker","iiif-tiles","lambda"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vt-digital-libraries-platform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-09T15:24:51.000Z","updated_at":"2024-10-12T19:40:58.000Z","dependencies_parsed_at":"2024-09-10T11:03:43.083Z","dependency_job_id":null,"html_url":"https://github.com/vt-digital-libraries-platform/aws-batch-iiif-generator","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vt-digital-libraries-platform%2Faws-batch-iiif-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vt-digital-libraries-platform%2Faws-batch-iiif-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vt-digital-libraries-platform%2Faws-batch-iiif-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vt-digital-libraries-platform%2Faws-batch-iiif-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vt-digital-libraries-platform","download_url":"https://codeload.github.com/vt-digital-libraries-platform/aws-batch-iiif-generator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253144628,"owners_count":21861092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-batch","docker","iiif","iiif-image","iiif-s3-docker","iiif-tiles","lambda"],"created_at":"2024-08-03T04:00:46.765Z","updated_at":"2025-05-08T20:31:11.261Z","avatar_url":"https://github.com/vt-digital-libraries-platform.png","language":"Python","funding_links":[],"categories":["Image Servers"],"sub_categories":["IIIF Extensions"],"readme":"# aws-batch-iiif-generator\n\n## Publication\n\n- [Code4Lib Journal - Scaling IIIF Image Tiling in the Cloud](https://journal.code4lib.org/articles/14933)\n\n## Workflow\n\n![Overview](images/overview.png \"Overview\")\n\n1. Upload task file to the batch bucket\n2. Batch bucket trigger launches an instance of a Lambda function\n3. The Lambda function reads the content in the task file and submits a batch job\n4. Each batch job generates tiles and manifests from the original image and uploads the generated derivatives to the target S3 bucket\n\n![Batch Job](images/batch_job.png \"Batch Job\")\n\n1. Pull raw original files from the S3 bucket\n2. Generate tiles and manifests\n3. Upload to target S3 bucket\n\n### Deploy aws-batch-iiif-generator using CloudFormation stack\n\n#### Step 1: Launch CloudFormation stack\n\n[![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?\u0026templateURL=https://vtlib-cf-template.s3.amazonaws.com/prod/cf-templates/aws-batch-iiif-generator/20240924/awsiiifs3batch.template)\n\nClick _Next_ to continue\n\n#### Step 2: Specify stack details\n\nNote: It's a good idea to provide a namespace for these resource names to prevent collisions (prepend resource names w/ stack name maybe? Don't prepend anything to the `DockerImage`)\n| Name | Description |\n| ------------------- | ---------------------------------------------------------- |\n| Stack name | any valid name |\n| BatchRepositoryName | any valid name for Batch process repository |\n| DockerImage | any valid Docker image. E.g. wlhunter/iiif_s3_tiling:latest |\n| JDName | any valid name for Job definition |\n| JQName | any valid name for Job queue |\n| LambdaFunctionName | any valid name for Lambda function |\n| LambdaRoleName | any valid name for Lambda role |\n| S3BucketName | any valid name for S3 bucket |\n\n#### Step 3: Configure stack options\n\nLeave it as is and click **Next**\n\n#### Step 4: Review\n\nMake sure all checkboxes under Capabilities section are **CHECKED**\n\nClick _Create stack_\n\n### Deploy aws-batch-iiif-generator using AWS CLI\n\nRun the following in your shell to deploy the application to AWS:\n\n```bash\naws cloudformation create-stack --stack-name awsiiifs3batch --template-body file://awsiiifs3batch.template --capabilities CAPABILITY_NAMED_IAM\n```\n\nSee [Cloudformation: create stack](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html) for `--parameters` option\n\n### Usage\n\n- Prepare [task.json](examples/task.json)\n- Prepare [dataset](examples/sample_dataset.zip) and upload to S3 `AWS_SRC_BUCKET` bucket\n- Edit the `jobQueue` and `jobDefinition` values in [task.json](examples/task.json) to match the resource names specified during stack creation.\n- Upload [task.json](examples/task.json) to the S3 bucket created after the deployment.\n- Go to `AWS_DEST_BUCKET` to see the end results for generated IIIF tiles and manifests.\n- Test manifests in [Mirador](https://projectmirador.org/demo/) (Note: you need to configure S3 access permission and CORS settings)\n- See our [Live Demo](https://d2fmsr62h737j1.cloudfront.net/index.html)\n\n### Cleanup\n\nTo delete the sample application that you created, use the AWS CLI. Assuming you used your project name for the stack name, you can run the following:\n\n```bash\naws cloudformation delete-stack --stack-name stackname\n```\n\n## Batch Configuration\n\n- Compute Environment: Type: `EC2`, MinvCpus: `0`, MaxvCpus: `128`, InstanceTypes: `optimal`\n- Job Definition: Type: `container`, Image: `DockerImage`, Vcpus: `2`, Memory: `2000`\n- Job Queue: Priority: `10`\n\n## S3\n\n- AWS_SRC_BUCKET: For raw images and CSV files to be processed\n  - Raw image files\n  - CSV files\n- AWS_DEST_BUCKET: For saving tiles and manifests files\n\n## Lambda function\n\n- [index.py](src/index.py): Submit a batch job when a task file is uploaded to a S3 bucket\n\n## Task File\n\n- example: [task.json](examples/task.json)\n\n| Name                  | Description                                                                                                                                                                                               |\n| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| jobName               | Batch job name                                                                                                                                                                                            |\n| jobQueue              | Batch job queue name                                                                                                                                                                                      |\n| jobDefinition         | Batch job definition name                                                                                                                                                                                 |\n| command               | \"./createiiif.sh\"                                                                                                                                                                                         |\n| AWS_REGION            | AWS region, e.g. us-east-1                                                                                                                                                                                |\n| COLLECTION_IDENTIFIER | from collection metadata csv                                                                                                                                                                              |\n| AWS_SRC_BUCKET        | S3 bucket which stores the images need to be processed. (Source S3 bucket)                                                                                                                                |\n| AWS_DEST_BUCKET       | S3 bucket which stores the generated tile images and manifests files. (Target S3 bucket)                                                                                                                  |\n| ACCESS_DIR            | Path to the image folder in `AWS_SRC_BUCKET`                                                                                                                                                              |\n| DEST_PREFIX           | path pointing to the directory that contains your collection directory in `AWS_DEST_BUCKET` (This is generally your \"collection category\" and does not include COLLECTION_IDENTIFIER at the path's end. ) |\n| DEST_URL              | Root URL for accessing the manifests e.g. https://cloudfront.amazonaws.com/...                                                                                                                            |\n| CSV_NAME              | A CSV file with title and description of the images                                                                                                                                                       |\n| CSV_PATH              | Path to the csv folder under the `AWS_SRC_BUCKET`                                                                                                                                                         |\n\n## IIIF S3 Docker image\n\n- [iiif_s3_docker](https://github.com/vt-digital-libraries-platform/iiif_s3_docker)\n- Image at Docker Hub: [wlhunter/iiif_s3_tiling:0.0.1](https://hub.docker.com/repository/docker/wlhunter/iiif_s3_tiling/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvt-digital-libraries-platform%2Faws-batch-iiif-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvt-digital-libraries-platform%2Faws-batch-iiif-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvt-digital-libraries-platform%2Faws-batch-iiif-generator/lists"}