{"id":18320886,"url":"https://github.com/bazaarvoice/emo-lambda-fanout","last_synced_at":"2025-10-30T12:31:15.813Z","repository":{"id":66754589,"uuid":"84995467","full_name":"bazaarvoice/emo-lambda-fanout","owner":"bazaarvoice","description":"an add-on co-process for EmoDB to trigger AWS Lambda functions on Databus events","archived":false,"fork":false,"pushed_at":"2017-11-08T14:14:41.000Z","size":122,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-23T08:45:29.602Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bazaarvoice.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-14T20:45:36.000Z","updated_at":"2017-03-14T21:23:49.000Z","dependencies_parsed_at":"2023-03-13T20:29:37.959Z","dependency_job_id":null,"html_url":"https://github.com/bazaarvoice/emo-lambda-fanout","commit_stats":null,"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazaarvoice%2Femo-lambda-fanout","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazaarvoice%2Femo-lambda-fanout/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazaarvoice%2Femo-lambda-fanout/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bazaarvoice%2Femo-lambda-fanout/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bazaarvoice","download_url":"https://codeload.github.com/bazaarvoice/emo-lambda-fanout/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238966524,"owners_count":19560366,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T18:17:45.783Z","updated_at":"2025-10-30T12:31:10.479Z","avatar_url":"https://github.com/bazaarvoice.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"This application features a prototype endpoint that invokes lambda functions on databus events.\n\nIt only accepts pre-approved keys (all others will get a 401 Unauthorized).\nThese keys must be hashed and stored in the config file.\n\nAPI\n---\n\n### GET     `/poller`\n\nList all function subscriptions for your API key.\n\n\n### GET     `/poller?lambdaArn={arn}`\n\nGet a specific function subscription by Lambda ARN.\n\n### GET     `/poller/size?lambdaArn={arn}`\n\nGet the number of queued events for a function, specified by Lambda ARN.\n\n### POST    `/poller?lambdaArn={lambdaArn}\u0026claimTtl={claimTtl}[\u0026batchSize={batchSize}]  body:{condition}`\n\nCreate a lambda subscription.\n* `{lambdaArn}`:    The ARN of the lambda function to invoke.\n* `{claimTtl}`:     How long before the event will be handled again. This should be longer than you expect the function to ever take.\n* `[{batchSize}]`:     Optional. How many messages to handle with each function invocation. For example, the minimum execution time for a lambda function is 100ms. If the actual run time of the function is 30ms, you can set `batchSize` to 3 and effectively get 3 executions for the price of one! \n* **request body**:   An EmoDB subscription condition.\n\nSee https://bazaarvoice.github.io/emodb/databus/ \"Subscription Management\" for more info.\n\n### DELETE     `/poller?lambdaArn={arn}`\n\nDeactivates and un-registers the function, specified by Lambda ARN. Note that this does not immediately clear the databus, so deleting and recreating the function will cause it to pick up where it left off.\n\n### POST       `/poller/activate?lambdaArn={arn}`\n\nRe-activates an inactive function, specified by Lambda ARN. This works whether the function got deactivated due to a `DELETE` request or \ntoo many consecutive function errors.\n\n\nSecurity Documentation\n======================\n\nKey Handling\n------------\n\n### Api key whitelist\n\nThis application will only accept emodb api keys that have been explicitly whitelisted.\n\nThe whitelist is stored in the application config (e.g., [the production config](src/main/config/config.bazaar.yaml)) under `poller.apiKeyDigestWhitelist`.\n\nThe whitelist contains argon2 hashes of the api key.\n\nTo create a hash, run https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/deploy/hash-key.sh thusly:\n\n    bin/hash-key.sh MyPlainTextKey\n\nThe actual code that generates the digest is here: https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/tools/HashKeyCommand.java.\n\nAll API requests are handled in https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/resource/PollerResource.java, which uses `getKey()` to extract and validate the api key header (https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/resource/PollerResource.java#L237).\n\n`getKey()` requires the presence of the api key in the headers, computes the base64-encoded sha512 hash of the request key and requires that the computed digest is present in the whitelist loaded from config.\n\nBeliefs:\n* We believe this is sufficient to ensure that no key that has not been explicitly whitelisted will be able to perform any API actions. \n* And additionally we believe that the whitelist is itself safe to store in a public location, as it should not be possible to derive any useful information from the hash.\n\n### Application API key handling\n\nThis application requires an EmoDB api key for managing its own application state.\n\nThis key is stored (encrypted) in the config file under `emodb.apiKey` (https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/config/config.bazaar.yaml#L10`).\n\nFor encryption, we use AWS KMS.\n\nTo encrypt the application key, we use https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/deploy/kms/encrypt-app-key.sh. The result of this is what we store in the config file.\n\nUpon application startup, we invoke https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/busplus/LambdaSubscriptionManager.java#L94, which calls https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/busplus/kms/ApiKeyCrypto.java#L64, which in turn invokes KMS to decrypt the key.\n\nThe way that KMS works, anyone who has Decrypt permission for our Customer Master Key (CMK) will be able to use KMS to decrypt the application key.\n\nSo the application needs permission to decrypt the api key by virtue of the role assigned to its instance profile.\n\n### Delegate API key handling\n\nThis application provides an additional feature of being able to register an AWS Lambda function to an EmoDB subscription. The way this works is that you create a lambda function and a policy that can invoke the function. Currently, an administrator (me) has to add that policy to the app's role. \n\nThen, you can call https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/resource/PollerResource.java#L150 and to hook the Lambda function up to a subscription.\n\nSince the poller will execute EmoDB `poll` operations on your behalf, it has to store the api key you provide. This storage mechanism is similar to the application API key handling procedure.\n\nWe encrypt the key using our CMK here: https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/busplus/LambdaSubscriptionManager.java#L194.\n\nThen, when the time comes to poll the subscription, we decrypt it here: https://github.com/bazaarvoice/emo-lambda-fanout/blob/master/src/main/java/com/bazaarvoice/emopoller/busplus/LambdaSubscriptionManager.java#L315.\n\n\n### Beliefs about API key handling\n\n* Both the lambda-fanout application's key and the subscription keys are stored encrypted in public locations (the config file and in an EmoDB table). We believe it is safe to do so because:\n** Only the key administrator (jroesler) and the application role will be able to decrypt the key.\n** No outsider will be able to assume the role that can decrypt the key.\n** No outsider will be able to gain access to the lambda-fanout instances (any decrypt request originating from the machine will be able to decrypt the key)\n\nDevelopment\n-----------\n\nIDEA\n====\n\n```\nMain class:               com.bazaarvoice.emopoller.emopollerApp\nProgram arguments:        server src/main/config/config.local.yaml\nWorking directory:        /home/YOU/repos/baz/emo-lambda-fanout (or wherever you keep it)\nUser classpath of module: emo-lambda-fanout\nJRE:                      1.8\n```\n\nAdding support for a new Emo API key\n====================================\n\nThe app will refuse to work with keys that are not whitelisted. This is to control who\ngets to use our service.\n\nEmo keys are secret!!! There's no way around handling them in plain text when we want\nto support a new key, but you *must exercise caution* not to leak the key with a careless\nchat message, email, code checkin, etc., etc. \n\n1. Have the key's owner send it to you over an encrypted channel, such as Box.\n2. Hash the key thusly: `bin/hash-key.sh MyKeyGoesHere`.\n    * this gives you something like: `JkKamLdE4pR+g1KgaM+iC6e0YmvG8+CjxsOGvRAlxeF1MvlFTMyrwpvgDmjPuaDLzqRHghp/WNjGetioS1foDA==`.\n3. Add it to `poller.apiKeyDigestWhitelist` in `src/main/config/config/config.bazaar.yaml`\n4. Follow the release/deploy instructions to deploy a new generation of proxies that accept this key.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbazaarvoice%2Femo-lambda-fanout","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbazaarvoice%2Femo-lambda-fanout","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbazaarvoice%2Femo-lambda-fanout/lists"}