https://github.com/pawarrchetan/serverless-logging
This repo defines serverless log forwarders using elastic functionbeat and a helper subscriber function.
https://github.com/pawarrchetan/serverless-logging
functionbeat lambda-functions serverless serverless-functions
Last synced: 8 months ago
JSON representation
This repo defines serverless log forwarders using elastic functionbeat and a helper subscriber function.
- Host: GitHub
- URL: https://github.com/pawarrchetan/serverless-logging
- Owner: pawarrchetan
- Created: 2020-04-28T17:26:17.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2020-12-03T21:36:52.000Z (about 5 years ago)
- Last Synced: 2024-11-08T15:47:09.357Z (about 1 year ago)
- Topics: functionbeat, lambda-functions, serverless, serverless-functions
- Language: Python
- Homepage:
- Size: 6.84 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Overview
The combination of the 2 functions deployed by this repository allow centralised logging of serverless components in Elasticsearch.
This package is used to deploy 2 serverless function to centralise the serverless logging.
1. create a log-shipper function to ship CloudWatch logs generated after the invocation of Lambda Functions.
2. create a log-subscriber function to subscribe newly created CloudWatch log groups with PREFIX "/aws/lambda/" to the log-shipper function.
Whenever the `log-shipper` lambda is invoked :
* logs are generated in the CloudWatch log group for the respective lambda function.
* logs are matched by the shipper function
* logs are shipped to the Elasticsearch Cloud used for log centralization.
Whenever there is any serverless component deployed in the AWS accounts, the `log-subscriber` function will detect the `CreateLogGroup` event from CloudWatch logs and create a subscription filter for the CloudWatch log group to allow the `log-shipper` function to ship the logs to the Elasticsearch.
# Instructions
The `functionbeat.yml` file mentions the log group `/aws/lambda/sample` for subscription to the `log-shipper` function. This log group is only added for initialization of the `log-shipper` function.
* functionbeat binary does not automatically handle updates to the function if the configuration changes. To mitigate this, we have a boolean variable in CI / CD pipeline job which takes care of the DEPLOY or UPDATE.
A value of DEPLOY = "TRUE" will deploy the function as a new cloudformation stack at first deployment.
A value of DEPLOY = "FALSE" will update the existing cloudformation stack if there are any changes to the file `functionbeat.yml`
# Deploy lambda
## Package
The 2 functions described above use 2 different methods to be deployed to the AWS accounts.
* `log-shipper` uses functionbeat.
* `log-subscriber` uses serverless.
Getting Started : https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-getting-started.html
## Deploying the functions
The function can be deployed using functionbeat and serverless binary separately.
### Functionbeat
You can find the Elastic Cloud password in AWS Secrets Manager. This secret is required for HTTP Basic Authentication to send logs via HTTPS to Elastic Cloud.
#### Step 1: Download Functionbeat
```
curl -L -O https://artifacts.elastic.co/downloads/beats/functionbeat/functionbeat-7.6.1-linux-x86_64.tar.gz
tar xzvf functionbeat-7.6.1-linux-x86_64.tar.gz
```
#### Step 2: Configure Functionbeat
Guide : https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-configuration.html
#### Step 3: Load Index template in Elasticsearch
Guide : https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-template.html
#### Step 4: Deploy Functionbeat
1. Make sure you have the credentials required to authenticate with AWS. You can set environment variables that contain your credentials:
```
export AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOPUSER
export AWS_SECRET_ACCESS_KEY=EXAMPLE567890devgHIJKMLOPNQRSTUVZ1234KEY
export AWS_DEFAULT_REGION=us-east-1
```
2. Make sure the user has the permissions required to deploy and run the function. For more information, See https://www.elastic.co/guide/en/beats/functionbeat/current/iam-permissions.html for more details.
3. Deploy the cloud function.
For example, the following command deploys a function called cloudwatch:
linux and mac:
```
./functionbeat -v -e -d "*" deploy log-shipper
```
#### Step 5: View your data in Kibana
To learn how to view and explore your data, see the Kibana User Guide - https://www.elastic.co/guide/en/kibana/7.6/index.html
### Serverless
#### Deploying the function
You can find the RDS password in AWS Secrets Manager. When we provision the DB with terraform we also write that secret there so that it can integrate with the automated deployment process securely.
Deploy command (**staging** example):
```bash
aws lambda create-function --function-name test-results-customer --runtime python3.6 \
--zip-file fileb://results-customer-consumer.zip --handler handler.handler \
--role arn:aws:iam::882235782134:role/temp-lambda-results-consumer \
--environment "Variables={API_ENDPOINT=https://api.epilot.cloud,API_USERNAME=staging_results_consumer,API_PASSWORD=,WIDGET_API_ENDPOINT=https://api.staging.epilot.io/widget}"
```
#### Map lambda function to sqs
```bash
aws lambda create-event-source-mapping --function-name test-results-customer --batch-size 5 \
--event-source-arn arn:aws:sqs:eu-central-1:882235782134:customer
```
#### Update function code
```bash
aws lambda update-function-code --function-name test-results-customer --zip-file fileb://results-customer-consumer.zip
```
## Subscriber Function
Once both the serverless functions are deployed, we also have the capability to auto-subcribe newly created CloudWatch log groups to the log-shipper function.
The log subscriber function will look for prefix /aws/lambda/ to auto-subscribe the newly created CloudWatch log groups.
# References
* https://www.elastic.co/guide/en/beats/functionbeat/current/index.html
* https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-getting-started.html
* https://serverless.com/framework/docs/getting-started/