{"id":15345455,"url":"https://github.com/jthomas/openwhisk-s3-trigger-feed","last_synced_at":"2025-09-11T04:44:22.350Z","repository":{"id":140706034,"uuid":"169767709","full_name":"jthomas/openwhisk-s3-trigger-feed","owner":"jthomas","description":"S3 Trigger Feed for Apache OpenWhisk ","archived":false,"fork":false,"pushed_at":"2019-06-05T14:43:59.000Z","size":43,"stargazers_count":3,"open_issues_count":3,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-28T15:11:18.831Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jthomas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-08T16:51:28.000Z","updated_at":"2020-04-07T14:34:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"2859d9d2-c2dc-4ff2-b498-370ff2b7c8c6","html_url":"https://github.com/jthomas/openwhisk-s3-trigger-feed","commit_stats":{"total_commits":20,"total_committers":4,"mean_commits":5.0,"dds":"0.30000000000000004","last_synced_commit":"be570ba843ea8806d588c3e7e640bbef3e6e65fa"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fopenwhisk-s3-trigger-feed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fopenwhisk-s3-trigger-feed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fopenwhisk-s3-trigger-feed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fopenwhisk-s3-trigger-feed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jthomas","download_url":"https://codeload.github.com/jthomas/openwhisk-s3-trigger-feed/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249001325,"owners_count":21196348,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T11:13:26.634Z","updated_at":"2025-04-15T03:30:45.720Z","avatar_url":"https://github.com/jthomas.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache OpenWhisk S3 Event Provider\n\nThis is an Apache OpenWhisk trigger feed for an S3-compatible Object Store. It polls the bucket using the `ListObjects` API call to retrieve bucket files on a fixed interval. Changes to bucket files, as indicated by API responses, are fired as trigger events.\n\n## usage\n\n| Entity                                   | Type | Parameters                               |\n| ---------------------------------------- | ---- | ---------------------------------------- |\n| `/\u003cPROVIDER_NS\u003e/s3-trigger-feed/changes` | feed | bucket, interval, s3_endpoint, s3_apikey |\n\n- `bucket` is the bucket name.\n- `interval` is the polling interval in minutes (minimum 1).\n- `s3_endpoint` is the object store endpoint, e.g. `s3.eu-gb.objectstorage.softlayer.net`\n- `s3_apikey` is the IAM API key for the object store service.\n\n### example\n\n```\nwsk trigger create test-s3-trigger --feed /\u003cPROVIDER_NS\u003e/s3-trigger-feed/changes --param bucket \u003cBUCKET_NAME\u003e --param interval \u003cMINS\u003e --param s3_endpoint \u003cCOS_ENDPOINT\u003e --param s3_apikey \u003cCOS_KEY\u003e\n```\n\n### trigger events\n\n```\n{\"file\":\"\u003cBUCKET_FILE_OBJECT\u003e\",\"status\":\"\u003cadded|deleted|modified\u003e\"}\n```\n\n#### example\n\n```json\n{\n  \"file\": {\n    \"ETag\": \"\\\"fb47672a6f7c34339ca9f3ed55c6e3a9\\\"\",\n    \"Key\": \"file-86.txt\",\n    \"LastModified\": \"2018-12-19T08:33:27.388Z\",\n    \"Owner\": {\n      \"DisplayName\": \"80a2054e-8d16-4a47-a46d-4edf5b516ef6\",\n      \"ID\": \"80a2054e-8d16-4a47-a46d-4edf5b516ef6\"\n    },\n    \"Size\": 25,\n    \"StorageClass\": \"STANDARD\"\n  },\n  \"status\": \"deleted\"\n}\n```\n\n## limitations\n\n- This event provider uses polling to monitor file changes to a bucket. Changes that occur between polls will not be registered, e.g. add and then remove a file within the polling interval. Changes that overlap will not be registered, multiple modifications to a file within the polling interval will only result in an event for the last modification.\n- The `ListObjects` API lists a maximum of 1000 objects per bucket. If you have more files than this in the bucket, incorrect trigger events will be fired, i.e. removal events when files are still available.\n- `ListObjects` incurs a GET request per polling interval. Polling once a minute will generate ~43830 requests per month. This is more than twice the free requests per month on the Lite tier (20,000 GET requests/month). This would cost 0.02 dollars per month on the paid `Cross Region Standard` plan.\n\n## architecture\n\nThis event provider uses the \"[Pluggable OpenWhisk Event Provider](https://github.com/jthomas/openwhisk-pluggable-event-provider)\" to handle the trigger management around trigger feeds. The implementation of the event provider polls each bucket on an interval using the `ListObjects` [API call](https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html). Results are cached in Redis to allow comparison between calls. An internal memory-based queue is used to decouple polling operations from trigger firing.\n\n### memory requirements\n\nBucket files (with ETags for comparison) need to be stored between polling requests. These values are stored as JavaScript arrays serialised to JSON and then compressed using GZIP. Binary strings are then stored directly in Redis.\n\nTesting with 1000 random files, this uses ~90KB of memory per bucket in Redis. Polling 10,000 buckets of this size (1000 users with 10 buckets) would need about 900MB.\n\nThe application uses an in-memory LRU cache to reduce the amount of network requests to Redis. Cache eviction is based upon a maximum number of keys and defaults to 1000 buckets.\n\n### resiliency\n\nIf the event provider crashes during polling it can re-read the previous bucket etags versions from Redis and trigger bucket information for Cloudant.\n\nBucket polling and trigger firing operations are de-coupled into different processes and joined by an in-memory queue. This allows the trigger firing to use exponential backoff to handle with trigger limits whilst not slowing down the polling operation. If the event provider crashes, untriggered file change events are currently lost. This could be changed to use Redis to improve the availability for this error case.\n\n### running\n\nSee the \"[Pluggable OpenWhisk Event Provider](https://github.ibm.com/thomas6/openwhisk-pluggable-provider)\" docs on how to run this event provider. The following environment parameters are needed for this feed provider.\n\n- `REDIS` - Redis URL string.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjthomas%2Fopenwhisk-s3-trigger-feed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjthomas%2Fopenwhisk-s3-trigger-feed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjthomas%2Fopenwhisk-s3-trigger-feed/lists"}