{"id":13626097,"url":"https://github.com/pawarrchetan/serverless-logging","last_synced_at":"2025-04-16T11:31:14.410Z","repository":{"id":114526925,"uuid":"259705442","full_name":"pawarrchetan/serverless-logging","owner":"pawarrchetan","description":"This repo defines serverless log forwarders using elastic functionbeat and a helper subscriber function.","archived":false,"fork":false,"pushed_at":"2020-12-03T21:36:52.000Z","size":7,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-08T15:47:09.357Z","etag":null,"topics":["functionbeat","lambda-functions","serverless","serverless-functions"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pawarrchetan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-04-28T17:26:17.000Z","updated_at":"2022-03-29T19:53:16.000Z","dependencies_parsed_at":"2023-06-08T17:15:26.715Z","dependency_job_id":null,"html_url":"https://github.com/pawarrchetan/serverless-logging","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pawarrchetan%2Fserverless-logging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pawarrchetan%2Fserverless-logging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pawarrchetan%2Fserverless-logging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pawarrchetan%2Fserverless-logging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pawarrchetan","download_url":"https://codeload.github.com/pawarrchetan/serverless-logging/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249235048,"owners_count":21235137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functionbeat","lambda-functions","serverless","serverless-functions"],"created_at":"2024-08-01T21:02:09.989Z","updated_at":"2025-04-16T11:31:14.012Z","avatar_url":"https://github.com/pawarrchetan.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Overview\nThe combination of the 2 functions deployed by this repository allow centralised logging of serverless components in Elasticsearch.\nThis package is used to deploy 2 serverless function to centralise the serverless logging.\n1. create a log-shipper function to ship CloudWatch logs generated after the invocation of Lambda Functions.\n2. create a log-subscriber function to subscribe newly created CloudWatch log groups with PREFIX \"/aws/lambda/\" to the log-shipper function.\n\nWhenever the `log-shipper` lambda is invoked :\n* logs are generated in the CloudWatch log group for the respective lambda function.\n* logs are matched by the shipper function\n* logs are shipped to the Elasticsearch Cloud used for log centralization.\n\nWhenever there is any serverless component deployed in the AWS accounts, the `log-subscriber` function will detect the `CreateLogGroup` event from CloudWatch logs and create a subscription filter for the CloudWatch log group to allow the `log-shipper` function to ship the logs to the Elasticsearch.\n\n\n# Instructions\nThe `functionbeat.yml` file mentions the log group `/aws/lambda/sample` for subscription to the `log-shipper` function. This log group is only added for initialization of the `log-shipper` function.\n\n* functionbeat binary does not automatically handle updates to the function if the configuration changes. To mitigate this, we have a boolean variable in CI / CD pipeline job which takes care of the DEPLOY or UPDATE. \n\nA value of DEPLOY = \"TRUE\" will deploy the function as a new cloudformation stack at first deployment.\nA value of DEPLOY = \"FALSE\" will update the existing cloudformation stack if there are any changes to the file `functionbeat.yml`\n\n# Deploy lambda\n\n## Package\nThe 2 functions described above use 2 different methods to be deployed to the AWS accounts.\n* `log-shipper` uses functionbeat.\n* `log-subscriber` uses serverless.\n\nGetting Started : https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-getting-started.html\n\n## Deploying the functions\nThe function can be deployed using functionbeat and serverless binary separately.\n\n### Functionbeat\nYou can find the Elastic Cloud password in AWS Secrets Manager. This secret is required for HTTP Basic Authentication to send logs via HTTPS to Elastic Cloud.\n\n#### Step 1: Download Functionbeat\n```\ncurl -L -O https://artifacts.elastic.co/downloads/beats/functionbeat/functionbeat-7.6.1-linux-x86_64.tar.gz\ntar xzvf functionbeat-7.6.1-linux-x86_64.tar.gz\n```\n\n#### Step 2: Configure Functionbeat\n Guide : https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-configuration.html\n\n#### Step 3: Load Index template in Elasticsearch\n Guide : https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-template.html\n\n#### Step 4: Deploy Functionbeat\n1. Make sure you have the credentials required to authenticate with AWS. You can set environment variables that contain your credentials:\n```\nexport AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOPUSER\nexport AWS_SECRET_ACCESS_KEY=EXAMPLE567890devgHIJKMLOPNQRSTUVZ1234KEY\nexport AWS_DEFAULT_REGION=us-east-1\n```\n\n2. Make sure the user has the permissions required to deploy and run the function. For more information, See https://www.elastic.co/guide/en/beats/functionbeat/current/iam-permissions.html  for more details.\n\n3. Deploy the cloud function.\nFor example, the following command deploys a function called cloudwatch:\n\nlinux and mac:\n```\n./functionbeat -v -e -d \"*\" deploy log-shipper\n```\n\n#### Step 5: View your data in Kibana\nTo learn how to view and explore your data, see the Kibana User Guide - https://www.elastic.co/guide/en/kibana/7.6/index.html\n\n### Serverless\n\n#### Deploying the function\nYou can find the RDS password in AWS Secrets Manager. When we provision the DB with terraform we also write that secret there so that it can integrate with the automated deployment process securely.\n\nDeploy command (**staging** example):\n```bash\naws lambda create-function --function-name test-results-customer --runtime python3.6 \\\n--zip-file fileb://results-customer-consumer.zip --handler handler.handler \\\n--role arn:aws:iam::882235782134:role/temp-lambda-results-consumer \\\n--environment \"Variables={API_ENDPOINT=https://api.epilot.cloud,API_USERNAME=staging_results_consumer,API_PASSWORD=\u003credacted\u003e,WIDGET_API_ENDPOINT=https://api.staging.epilot.io/widget}\"\n```\n\n#### Map lambda function to sqs\n```bash\naws lambda create-event-source-mapping --function-name test-results-customer --batch-size 5 \\\n--event-source-arn arn:aws:sqs:eu-central-1:882235782134:customer\n```\n\n#### Update function code\n```bash\naws lambda update-function-code --function-name test-results-customer  --zip-file fileb://results-customer-consumer.zip\n```\n\n## Subscriber Function\nOnce both the serverless functions are deployed, we also have the capability to auto-subcribe newly created CloudWatch log groups to the log-shipper function.\nThe log subscriber function will look for prefix /aws/lambda/ to auto-subscribe the newly created CloudWatch log groups.\n\n# References\n\n* https://www.elastic.co/guide/en/beats/functionbeat/current/index.html\n\n* https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-getting-started.html\n\n* https://serverless.com/framework/docs/getting-started/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpawarrchetan%2Fserverless-logging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpawarrchetan%2Fserverless-logging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpawarrchetan%2Fserverless-logging/lists"}