{"id":16070441,"url":"https://github.com/danmayer/s3_image_converter","last_synced_at":"2025-06-10T17:08:25.120Z","repository":{"id":66595158,"uuid":"73630857","full_name":"danmayer/s3_image_converter","owner":"danmayer","description":"a project to learn some Clojure: it does conversion on large volumes of S3 images","archived":false,"fork":false,"pushed_at":"2018-05-17T02:46:40.000Z","size":928,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-12T22:41:29.023Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danmayer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-13T17:18:31.000Z","updated_at":"2020-05-16T22:10:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"6d989bbb-6d17-436c-bd7f-17861bb34e4f","html_url":"https://github.com/danmayer/s3_image_converter","commit_stats":{"total_commits":51,"total_committers":2,"mean_commits":25.5,"dds":"0.27450980392156865","last_synced_commit":"e3c4bd2ce1f5bb96e7229cb93d080640cc9e0970"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danmayer%2Fs3_image_converter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danmayer%2Fs3_image_converter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danmayer%2Fs3_image_converter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danmayer%2Fs3_image_converter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danmayer","download_url":"https://codeload.github.com/danmayer/s3_image_converter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danmayer%2Fs3_image_converter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259114494,"owners_count":22807246,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-09T07:03:02.284Z","updated_at":"2025-06-10T17:08:25.091Z","avatar_url":"https://github.com/danmayer.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3 Image Converter\n\nA Clojure library designed to grab all images in S3 bucket and apply a transformation to them.\n\n## Usage\n\nMake sure you have your S3 credentials in `~/.aws/credentials`\n\n`lein run`\n\nwith options:\n\n`BUCKETNAME=mybucket lein run [cleanup, convert]`\n\nclean: `BUCKETNAME=mybucket lein run --function clean --matcher _dan_test.jpg`\n\nconvert: `BUCKETNAME=mybucket lein run --function convert --matcher \\.jpg`\n\nconvert pngs in directory: `BUCKETNAME=mybucket lein run --function convert --matcher \\.png --prefix products`\n\n## Running In Docker\n\n * add your aws files to a gitignored directory\n   * copy from `~/.aws/` to `./deploy`\n * `docker build -t danmayer/clojure-image-mapper:latest .`\n * `docker run -it --rm --name running-image-mapper danmayer/clojure-image-mapper:latest`\n \n To run with arguments and env variables:\n \n `docker run -e BUCKETNAME=mybucket -it --rm --name running-image-mapper danmayer/clojure-image-mapper:latest lein run --function convert --matcher \\.jpg`\n\n### dockerhub\n\n* `docker login`\n* `docker push danmayer/clojure-image-mapper`\n\nHow to make your `Dockerfile` wrap this one with your credentials and do something useful, like say run the converter every 5min in a cron. \n\n\n```\n#~/my_project/depoy/image-cron\nPATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/5 * * * * cd /usr/src/app \u0026\u0026 BUCKETNAME=my_bucker lein run --function convert --matcher \\.jpg --prefix products \u003e\u003e /var/log/cron.log 2\u003e\u00261\n```\n\n```\n#~/my_project/Dockerfile\nFROM danmayer/clojure-image-mapper:latest\nMAINTAINER Dan Mayer \u003cdan.mayer@xyz.com\u003e\n\n# add credentials\nADD deploy/aws_credentials /root/.aws/credentials\n\n# Add crontab file in the cron directory\nADD deploy/image-cron /var/spool/cron/crontabs/root\n\n# Give execution rights on the cron job\nRUN chmod 0644 /var/spool/cron/crontabs/root\n \n# Create the log file to be able to run tail\nRUN touch /var/log/cron.log\n \n# Run the command on container startup\nCMD crond -l 2 -f\n```\n\n## Using webp-imageio\n\nThe Jar as well as macOS \u0026 Linux native library is included in the repo, if it is necessary to rebuild it, do the following:\n\n```\nbrew install hg gradle cmake\nhg clone https://bitbucket.org/luciad/webp-imageio\ncd webp-imageio\ngradle build -x test\ncp ./build/libs/webp-imageio-\u003cVERSION\u003e-SNAPSHOT.jar \u003cpath-to-this-project\u003e/resources/\n# (update `:resource-paths` in project.clj)\n\nmkdir build\ncd build\ncmake ..\ncmake --build .\ncp src/main/c/* \u003cpath-to-this-project-repo\u003e/native\n```\n\n## Todo\n\n* oneoff script to remove all webp or fix public read on existing ones?\n* better way to handle AWS timeouts / throttling\n* time based filter options\n* more efficient S3 filter query (via CLI opposed to in app)\n* don't write files work directly with IO-streams\n  * https://github.com/karls/collage/issues/8\n  * https://github.com/karls/collage/blob/master/src/fivetonine/collage/util.clj#L82\n  * https://github.com/weavejester/clj-aws-s3/blob/master/src/aws/sdk/s3.clj#L134 \n* contribute patch back to collage to make WebP work again\n  * https://github.com/karls/collage/blob/master/src/fivetonine/collage/util.clj#L35-L80\n  * https://bitbucket.org/luciad/webp-imageio/src/fde3644e6aa610f6a8d97c3d982a7c3926324ecf/src/javase/java/com/luciad/imageio/webp/WebPWriteParam.java?at=default\u0026fileviewer=file-view-default#WebPWriteParam.java-28 \n* keep a count of how many images you converted\n* see about maximizing perf\n  *  increasing the buffers a lot to maybe 500 or 1000. \n  *  I also misremembered the semantics of “pipeline-async” and I think you should try replacing them all with “pipeline-blocking” instead since all the IO is blocking. \n\n## License\n\nCopyright © 2016 Dan Mayer and Ben Brinckerhoff\n\nDistributed under the Eclipse Public License either version 1.0 or (at\nyour option) any later version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanmayer%2Fs3_image_converter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanmayer%2Fs3_image_converter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanmayer%2Fs3_image_converter/lists"}