{"id":24576578,"url":"https://github.com/dmrhimali/functionbeatdemo","last_synced_at":"2026-01-04T13:01:55.761Z","repository":{"id":270609800,"uuid":"910693552","full_name":"dmrhimali/functionbeatDemo","owner":"dmrhimali","description":"deploy fuctionbeat to read cloudwatch logs from a aws lambda function and ship them to an elasticsearch instance.","archived":false,"fork":false,"pushed_at":"2025-01-01T19:05:21.000Z","size":4494,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-23T22:41:21.240Z","etag":null,"topics":["functionbeat","gitlab-ci"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmrhimali.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-01T05:04:46.000Z","updated_at":"2025-01-01T20:50:11.000Z","dependencies_parsed_at":"2025-01-01T19:37:41.050Z","dependency_job_id":"99c981f7-ae60-4747-a454-37477607d147","html_url":"https://github.com/dmrhimali/functionbeatDemo","commit_stats":null,"previous_names":["dmrhimali/functionbeatdemo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FfunctionbeatDemo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FfunctionbeatDemo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FfunctionbeatDemo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmrhimali%2FfunctionbeatDemo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmrhimali","download_url":"https://codeload.github.com/dmrhimali/functionbeatDemo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244039241,"owners_count":20387835,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functionbeat","gitlab-ci"],"created_at":"2025-01-23T22:40:30.771Z","updated_at":"2026-01-04T13:01:55.690Z","avatar_url":"https://github.com/dmrhimali.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Functionbeat\n\nFunctionbeat is defined as a “serverless” shipper that can be deployed as a function to collect and ship data into the ELK Stack. Designed for monitoring cloud environments, Functionbeat is currently tailored for Amazon setups and can be deployed as an Amazon Lambda function to collect data from Amazon CloudWatch, Kinesis and SQS.\n\nWe will create and deploy fuctionbeat to read cloudwatch logs generated from a aws lambda function and ship them to an elasticsearch instance.\n\n\n## setup Elasticsearch\n\n## Creat key pair\nhttps://console.aws.amazon.com/ec2/ \u003e\nNETWORK \u0026 SECURITY \u003e\nKey Pairs.\nCreate Key Pair\n\nThe private key file is automatically downloaded by your browser\n\nset permissions so only you can read it:\n`$ chmod 400 my-key-pair.pem`\n\n## Create Ec2 instsance\n\nI used t2.small  linux (but t2.large would be better)\n\n### Add vpc: default:\n![out](img/out1.png)\n![out](img/out2.png)\n\n\n### configure security group to open SSH(22), Elasticsearch(9200, 9300) and Kibana(5601) ports\n\n![out](img/out3.png)\n\nInbound open ports:\nPorts\tProtocol\tSource\tEC2-SSH\n22\t    tcp\t        0.0.0.0/0\t✔\n5601\ttcp\t        0.0.0.0/0\t✔\n9300\ttcp\t        0.0.0.0/0\t✔\n5400\ttcp\t        0.0.0.0/0\t✔\n9200\ttcp\t        0.0.0.0/0\t✔\n\nOutbound rules: all\n\nPorts\tProtocol\tDestination\tEC2-SSH\nAll\t    All\t        0.0.0.0/0\t    ✔\n\n### Add elastic IP\n\nEC2\u003e Elastic IP \u003e select instance\u003e associate elastic ip to instance\n\n![out](img/out4.png)\n\n### specify the previously created keypair.\n\n\n## Login to ec2 instance\n\nec2 console\u003e instance name\u003e click connect \u003e run ssh command given in terminal\n\ncd into location you downloaded the keypair and execute ssh to connect:\n\n```sh\n(base)  ✘ rdissanayakam@RBH12855  ~/vp/aws_access_keys  pwd                                                                            \n/Users/rdissanayakam/vp/aws_access_keys\n(base)  rdissanayakam@RBH12855  ~/vp/aws_access_keys  ls\nlogstash_keypair.pem  \n(base)  rdissanayakam@RBH12855  ~/vp/aws_access_keys  ssh -i \"logstash_keypair.pem\" ec2-user@ec2-3-213-192-20.compute-1.amazonaws.com\nLast login: Wed Jul 10 15:00:21 2019 from 204.26.112.13\n\n       __|  __|_  )\n       _|  (     /   Amazon Linux 2 AMI\n      ___|\\___|___|\n\nhttps://aws.amazon.com/amazon-linux-2/\nNo packages needed for security; 6 packages available\nRun \"sudo yum update\" to apply all updates.\n```\n\nnow login as root:\n\n```sh\n[ec2-user@ip-172-31-62-80 ~]$ sudo su -\nLast login: Wed Jul 10 15:00:23 UTC 2019 on pts/0\n[root@ip-172-31-62-80 ~]# \n```\n\n## install java\n`$ sudo yum install java-1.8.0-openjdk-devel`\n\ncheck version;\n`$ java -version`\n\noutput:\n```sh\nopenjdk version \"1.8.0_201\"\nOpenJDK Runtime Environment (build 1.8.0_201-b09)\nOpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode)\n```\n\n##  add the Elasticsearch repository\n\nImport the repository’s GPG key using the following command:\n`$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch`\n\nOpen your text editor and create the following repo file:\n`$ sudo vi /etc/yum.repos.d/elasticsearch.repo`\n\ncopy and save following, then close test editor:\n```sh\n[elasticsearch-6.x]\nname=Elasticsearch repository for 6.x packages\nbaseurl=https://artifacts.elastic.co/packages/6.x/yum\ngpgcheck=1\ngpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch\nenabled=1\nautorefresh=1\ntype=rpm-md\n```\n\nor:\n\ncat \u003e /etc/yum.repos.d/elasticsearch.repo \u003c\u003c \"EOF\"\n[elasticsearch-6.x]\nname=Elasticsearch repository for 6.x packages\nbaseurl=https://artifacts.elastic.co/packages/6.x/yum\ngpgcheck=1\ngpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch\nenabled=1\nautorefresh=1\ntype=rpm-md\nEOF\n\n\n\n## install the Elasticsearch package\n`$ sudo yum install elasticsearch`\n\n## modify elasticsearch.yml:\n\nNow we need to change networkhost so it is not locahost anymore:\n\n[root@ip-172-31-62-80 ~]# `netstat -nptl`\nActive Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name          \ntcp6       0      0 `127.0.0.1:9200  `     :::*                    LISTEN      5460/java           \ntcp6       0      0 `127.0.0.1:9300 `      :::*                    LISTEN      5460/java           \n\nNotice it is set to loopback address. lets change it throufh elasticsearch.yml\n\n```sh\nvi /etc/elasticsearch/elasticsearch.yml\n```\n\nchange network host to private ip of the ec2 instance so it can be accessed from outside(lambda):\n\n```json\nnetwork.host: 172.31.62.80\n```\n\n\n\n## Run elasticsearch as a service\n\n```sh\n$ sudo systemctl enable elasticsearch.service\n$ sudo systemctl start elasticsearch.service\n```\n\nStart service and set it to auto run on boot up\n\n```sh\n/etc/init.d/elasticsearch status\nsudo chkconfig --levels 235 elasticsearch on\n\n\n$ sudo service elasticsearch stop\n$ sudo service elasticsearch start\n$ sudo service elasticsearch status\n```\n\nNow ip is corrected so that elasticsearch service is accessible from outside through ip:\n[root@ip-172-31-62-80 ~]# `netstat -nptl`\nActive Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name           \ntcp6       0      0 `172.31.62.80:9200 `      :::*                    LISTEN      5460/java           \ntcp6       0      0 `172.31.62.80:9300 `      :::*                    LISTEN      5460/java                 \n[root@ip-172-31-62-80 ~]# \n\n## check if runnig\n`$ sudo systemctl status elasticsearch.service`\n\nor better yet (both public ip and private ip in ec2 instace in url works)\n`sudo curl -XGET 'http://3.213.192.20:9200/_cat/indices?v\u0026pretty'`\n\n```sh\n[root@ip-172-31-62-80 ~]# sudo curl -XGET 'http://3.213.192.20:9200/_cat/indices?v\u0026pretty'\nhealth status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size\ngreen  open   .kibana_1                     kY8iQBMsT1iGBJXj1FiIcg   1   0          5            1     40.4kb         40.4kb\ngreen  open   .kibana_task_manager          yKn_5v5wSvm489h7HshNgQ   1   0          2            0     12.6kb         12.6kb\nyellow open   accounts                      oDNcG3EYScitVEKMCKq6ZQ   5   1          2            0     10.5kb         10.5kb\nyellow open   functionbeat-7.2.0-2019.07.10 9yxPz0MkR96IrR-kkgjcEQ   1   1         13            0     37.9kb         37.9kb\n```\n```sh\nsudo curl -XGET 'http://172.31.62.80:9200/_cat/indices?v\u0026pretty'\nhealth status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size\ngreen  open   .kibana_1                     kY8iQBMsT1iGBJXj1FiIcg   1   0          5            1     40.4kb         40.4kb\ngreen  open   .kibana_task_manager          yKn_5v5wSvm489h7HshNgQ   1   0          2            0     12.6kb         12.6kb\nyellow open   accounts                      oDNcG3EYScitVEKMCKq6ZQ   5   1          2            0     10.5kb         10.5kb\nyellow open   functionbeat-7.2.0-2019.07.10 9yxPz0MkR96IrR-kkgjcEQ   1   1         13            0     37.9kb         37.9kb\n```\n\n```json\n{\n  \"name\" : \"fLVNqN_\",\n  \"cluster_name\" : \"elasticsearch\",\n  \"cluster_uuid\" : \"6zKcQppYREaRH0tyfJ9j7Q\",\n  \"version\" : {\n    \"number\" : \"6.7.0\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"rpm\",\n    \"build_hash\" : \"8453f77\",\n    \"build_date\" : \"2019-03-21T15:32:29.844721Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"7.7.0\",\n    \"minimum_wire_compatibility_version\" : \"5.6.0\",\n    \"minimum_index_compatibility_version\" : \"5.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n```\n\nTo view the messages logged by the Elasticsearch service you can use the command below:\n\n`$  sudo journalctl -u elasticsearch`\n\n\n## Create test index\n`curl -X PUT \"http://127.0.0.1:9200/mytest_index\"\n{\"acknowledged\":true,\"shards_acknowledged\":true,\"index\":\"mytest_index\"}`\n\n\n## check indices\n`sudo curl -XGET 'http://172.31.62.80:9200/_cat/indices?v\u0026pretty'`\n\n\n## post:\n curl -X POST 172.31.62.80:9200/accounts/person/2 -curl -H 'Content-Type: application/json' -d '\n {\n     \"name\" : \"John\",\n     \"lastname\" : \"Doe\",\n     \"job_description\" : \"Systems administrator and Linux specialit\"\n }\n '\n\n## get:\ncurl -X GET 172.31.62.80:9200/accounts/person/1\n\n\n# Install kibana in ec2 instance -linux\n\n## Download and install the Public Signing  Key (Already done from above)\n`sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch`\n\nNext create a file called “kibana.repo” in “/etc/yum.repos.d/” directory\n`sudo vi /etc/yum.repos.d/kibana.repo`\n\nThen add the following config into the repository created above\n[kibana-5.x]\nname=Kibana repository for 5.x packages\nbaseurl=https://artifacts.elastic.co/packages/5.x/yum\ngpgcheck=1\ngpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch\nenabled=1\nautorefresh=1\ntype=rpm-md\n\nopen port 5601 in security group of ec2 instance\n\n##  install kibana\n`sudo yum install kibana`\n\n## change kibana.yml configuration\n\nNotice kibana points to loopback/localhost:\n\n```sh\n[root@ip-172-31-62-80 ~]# `netstat -nptl`\nActive Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name          \ntcp        0      0 `127.0.0.1:5601`       0.0.0.0:*               LISTEN      3468/node           \n```\n\n\n```sh\nvi /etc/kibana/kibana.yml\n```\nchange server host to private ip of ec2 instance:\n\n```json\nserver.host: \"172.31.62.80\"\n```\n\nchange elasticsearch.hosts to use ec2 private ip:\n\n```json\nelasticsearch.hosts: [\"http://172.31.62.80:9200\"]\n```\n\n## Start Kibana\n`sudo -i service kibana stop`\n`sudo -i service kibana start`\n\n```sh\n[root@ip-172-31-62-80 ~]# `netstat -nptl`\nActive Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name          \ntcp     0      0    172.31.62.80:5601     0.0.0.0:*               LISTEN      3468/node    \n```\nCheck if you are running Systemd or SysV init\n`ps -p 1`\n\n—– For SysV init —–\n`sudo chkconfig --add kibana`\n`sudo -i service kibana stop`\n`sudo -i service kibana start`\n\n—– For systemd —–\n`sudo /bin/systemctl daemon-reload`\n`sudo /bin/systemctl enable kibana.service`\n`sudo systemctl stop kibana.service`\n`sudo systemctl start kibana.service`\n\n## access your kibana page with following URL:\nhttp://3.213.192.20:5601/app/kibana\n\n\n# FunctionBeat\n\nhttps://logz.io/blog/beats-tutorial/\n\n## What are Beats?\nBeats are a collection of lightweight (resource efficient, no dependencies, small) and open source log shippers that act as agents installed on the different servers in your infrastructure for collecting logs or metrics.\n\nThese can be \n- log files (Filebeat), \n- network data (Packetbeat), \n- server metrics (Metricbeat), \n\nor any other type of data that can be collected by the growing number of Beats being developed by both Elastic and the community. Once collected, the data is sent either directly into Elasticsearch or to Logstash for additional processing.\n\nBeats are built on top of a Go framework called libbeat — a library for data forwarding which means that new beats are being developed and contributed by the community all the time.\n\n### Functionbeat\nFunctionbeat is defined as a “serverless” shipper that can be deployed as a function to collect and ship data into the ELK Stack. Designed for monitoring cloud environments, Functionbeat is currently tailored for Amazon setups and can be `deployed as an Amazon Lambda function` to collect data from Amazon CloudWatch, Kinesis and SQS.\n\n## Install functionbeat\n\n1. bring up linux ec2 instance\n\n2. ssh into it\n\n    ssh -i \"logstash_keypair.pem\" ec2-user@ec2-18-234-167-255.compute-1.amazonaws.com\n\n    ```bash\n    (base)  rdissanayakam@RBH12855  ~/vp/aws_access_keys  pwd\n    /Users/rdissanayakam/vp/aws_access_keys\n    (base)  rdissanayakam@RBH12855  ~/vp/aws_access_keys  ls\n    logstash_keypair.pem        logstash_keypair_public.pem tmp.json\n    (base)  rdissanayakam@RBH12855  ~/vp/aws_access_keys  ssh -i \"logstash_keypair.pem\" ec2-user@ec2-18-234-167-255.compute-1.amazonaws.com\n    The authenticity of host 'ec2-18-234-167-255.compute-1.amazonaws.com (127.0.0.1)' can't be established.\n    ECDSA key fingerprint is SHA256:BT44tRbfvTKE3q5z1NsM/imn2rk9JvcuzWZXepZoLyI.\n    Are you sure you want to continue connecting (yes/no)? yes\n    Warning: Permanently added 'ec2-18-234-167-255.compute-1.amazonaws.com' (ECDSA) to the list of known hosts.\n\n        __|  __|_  )\n        _|  (     /   Amazon Linux 2 AMI\n        ___|\\___|___|\n\n    https://aws.amazon.com/amazon-linux-2/\n    No packages needed for security; 6 packages available\n    Run \"sudo yum update\" to apply all updates.\n    [ec2-user@ip-172-31-63-184 ~]$ \n    ```\n\n3. login as root\n\n    $ sudo su -\n\n    ```bash\n    [ec2-user@ip-172-31-63-184 ~]$ sudo su -\n    [root@ip-172-31-63-184 ~]\n    ```\n4. install functionbeat\n\n    curl -L -O https://artifacts.elastic.co/downloads/beats/functionbeat/functionbeat-7.2.0-linux-x86_64.tar.gz\ntar xzvf functionbeat-7.2.0-linux-x86_64.tar.gz\n\n    ```bash\n    [root@ip-172-31-63-184 ~]# curl -L -O https://artifacts.elastic.co/downloads/beats/functionbeat/functionbeat-7.2.0-linux-x86_64.tar.gz\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n    100 47.2M  100 47.2M    0     0  48.7M      0 --:--:-- --:--:-- --:--:-- 48.6M\n    [root@ip-172-31-63-184 ~]#\n\n    [root@ip-172-31-63-184 ~]# ls\nf   unctionbeat-7.2.0-linux-x86_64.tar.gz\n    ```\n\n\n5. extract functionbeat-7.2.0-linux-x86_64.tar.gz\n\n    ```bash\n    [root@ip-172-31-63-184 ~]# tar xvzf functionbeat-7.2.0-linux-x86_64.tar.gz\n    functionbeat-7.2.0-linux-x86_64/NOTICE.txt\n    functionbeat-7.2.0-linux-x86_64/fields.yml\n    functionbeat-7.2.0-linux-x86_64/LICENSE.txt\n    functionbeat-7.2.0-linux-x86_64/kibana/\n    functionbeat-7.2.0-linux-x86_64/functionbeat\n    functionbeat-7.2.0-linux-x86_64/.build_hash.txt\n    functionbeat-7.2.0-linux-x86_64/functionbeat.reference.yml\n    functionbeat-7.2.0-linux-x86_64/pkg/functionbeat\n    functionbeat-7.2.0-linux-x86_64/README.md\n    functionbeat-7.2.0-linux-x86_64/functionbeat.yml\n    ```\n6. veirfy extract has functionbeat.yml\n\n    ```bash\n    [root@ip-172-31-63-184 ~]# ls\n    functionbeat-7.2.0-linux-x86_64  functionbeat-7.2.0-linux-x86_64.tar.gz\n    [root@ip-172-31-63-184 ~]# \n    [root@ip-172-31-63-184 ~]# cd functionbeat-7.2.0-linux-x86_64/\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]# ls\n    fields.yml  functionbeat  functionbeat.reference.yml  `functionbeat.yml`  kibana  LICENSE.txt  NOTICE.txt  pkg  README.md\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]#\n    ```\n\n7. Create aws credentials file in root\n\n    ```sh\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]# cat ~/credentials.md\n    export AWS_ACCESS_KEY_ID=AKIAIIMCMBUCW3IT4SIQ\n    export AWS_SECRET_ACCESS_KEY=xxxx\n    export AWS_DEFAULT_REGION=us-east-1\n\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]#\n    ```\n\n    source file:\n    $ source ~/credentials.md\n\n    verify:\n    $ printenv\n\n8. configure functionbeat\n\n    https://www.elastic.co/guide/en/beats/functionbeat/current/elasticsearch-output.html\n\n    $ vi functionbeat-7.2.0-linux-x86_64/functionbeat.yml\n\n#============================  Provider ===============================\nfunctionbeat.provider.aws.deploy_bucket: \"functionbeat-lambdalog-deploy\"\nfunctionbeat.provider.aws.functions:\n  - name: cloudwatch\n    enabled: true\n    type: cloudwatch_logs\n    description: \"lambda function for cloudwatch logs\"\n    triggers:\n      - log_group_name: /aws/lambda/activityStreamLambda\n#==================== Elasticsearch template setting ==========================\nsetup.template.settings:\n  index.number_of_shards: 1\n#================================ Outputs =====================================\noutput.elasticsearch:\n  hosts: [\"3.213.192.20:9200\"]\n  enabled: true\n#================================ Processors =====================================\nprocessors:\n  - add_host_metadata: ~\n  - add_cloud_metadata: ~\n  - add_fields:\n      target: project\n      fields:\n        name: insights-lambda\n\n#================================ Logging =====================================\nlogging.level: debug\n\n\n## full file:\n\n\n```bash    \n###################### Functionbeat Configuration Example #######################\n\n# This file is an example configuration file highlighting only the most common\n# options. The functionbeat.reference.yml file from the same directory contains all the\n# supported options with more comments. You can use it as a reference.\n#\n# You can find the full configuration reference here:\n# https://www.elastic.co/guide/en/beats/functionbeat/index.html\n#\n\n#============================  Provider ===============================\n# Configure functions to run on AWS Lambda, currently we assume that the credentials\n# are present in the environment to correctly create the function when using the CLI.\n#\n# Configure which S3 bucket we should upload the lambda artifact.\nfunctionbeat.provider.aws.deploy_bucket: \"functionbeat-lambdalog-deploy\"\n\nfunctionbeat.provider.aws.functions:\n  # Define the list of function availables, each function required to have a unique name.\n  # Create a function that accepts events coming from cloudwatchlogs.\n  - name: cloudwatch\n    enabled: true\n    type: cloudwatch_logs\n\n    # Description of the method to help identify them when you run multiples functions.\n    description: \"lambda function for cloudwatch logs\"\n\n    # Concurrency, is the reserved number of instances for that function.\n    # Default is 5.\n    #\n    # Note: There is a hard limit of 1000 functions of any kind per account.\n    #concurrency: 5\n\n    # The maximum memory allocated for this function, the configured size must be a factor of 64.\n    # There is a hard limit of 3008MiB for each function. Default is 128MiB.\n    #memory_size: 128MiB\n\n    # Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.\n    #dead_letter_config.target_arn:\n\n    # Execution role of the function.\n    # role: arn:aws:iam::219708553334:role/functionbeat-role\n\n    # Connect to private resources in an Amazon VPC.\n    # virtual_private_cloud:\n      # security_group_ids: [\"sg-1838b463\"]\n      # subnet_ids: [\"subnet-154d1c28\", \"subnet-956304cd\"]\n\n    # Optional fields that you can specify to add additional information to the\n    # output. Fields can be scalar values, arrays, dictionaries, or any nested\n    # combination of these.\n    #fields:\n    #  env: staging\n\n    # List of cloudwatch log group registered to that function.\n    triggers:\n      - log_group_name: /aws/lambda/activityStreamLambda\n    #    filter_pattern: mylog_\n\n    # Define custom processors for this function.\n    #processors:\n    #  - dissect:\n    #      tokenizer: \"%{key1} %{key2}\"\n\n\n#==================== Elasticsearch template setting ==========================\n\nsetup.template.settings:\n  index.number_of_shards: 1\n  #index.codec: best_compression\n  #_source.enabled: false\n\n\n# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.\n# This requires a Kibana endpoint configuration.\nsetup.kibana:\n\n  # Kibana Host\n  # Scheme and port can be left out and will be set to the default (http and 5601)\n  # In case you specify and additional path, the scheme is required: http://localhost:5601/path\n  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601\n  #host: \"localhost:5601\"\n\n  # Kibana Space ID\n  # ID of the Kibana Space into which the dashboards should be loaded. By default,\n  # the Default Space will be used.\n  #space.id:\n\n#============================= Elastic Cloud ==================================\n\n# These settings simplify using functionbeat with the Elastic Cloud (https://cloud.elastic.co/).\n\n# The cloud.id setting overwrites the `output.elasticsearch.hosts` and\n# `setup.kibana.host` options.\n# You can find the `cloud.id` in the Elastic Cloud web UI.\n#cloud.id:\n\n# The cloud.auth setting overwrites the `output.elasticsearch.username` and\n# `output.elasticsearch.password` settings. The format is `\u003cuser\u003e:\u003cpass\u003e`.\n#cloud.auth:\n\n#================================ Outputs =====================================\n\n# Configure what output to use when sending the data collected by the beat.\n\n#-------------------------- Elasticsearch output ------------------------------\noutput.elasticsearch:\n  # Array of hosts to connect to.\n  hosts: [\"3.213.192.20:9200\"]\n  enabled: true\n  # Optional protocol and basic auth credentials.\n  #protocol: \"https\"\n  #username: \"elastic\"\n  #password: \"changeme\"\n\n#----------------------------- Logstash output --------------------------------\n#output.logstash:\n  # The Logstash hosts\n  #hosts: [\"localhost:5044\"]\n\n  # Optional SSL. By default is off.\n  # List of root certificates for HTTPS server verifications\n  #ssl.certificate_authorities: [\"/etc/pki/root/ca.pem\"]\n\n  # Certificate for SSL client authentication\n  #ssl.certificate: \"/etc/pki/client/cert.pem\"\n\n  # Client Certificate Key\n  #ssl.key: \"/etc/pki/client/cert.key\"\n\n#================================ Processors =====================================\n\n# Configure processors to enhance or manipulate events generated by the beat.\n\nprocessors:\n  - add_host_metadata: ~\n  - add_cloud_metadata: ~\n  - add_fields:\n      target: project\n      fields:\n        name: insights-lambda\n\n#================================ Logging =====================================\n\n# Sets log level. The default log level is info.\n# Available log levels are: error, warning, info, debug\nlogging.level: debug\n\n# At debug level, you can selectively enable logging only for some components.\n# To enable all selectors use [\"*\"]. Examples of other selectors are \"beat\",\n# \"publish\", \"service\".\n#logging.selectors: [\"*\"]\n\n#============================== Xpack Monitoring ===============================\n# functionbeat can export internal metrics to a central Elasticsearch monitoring\n# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The\n# reporting is disabled by default.\n\n# Set to true to enable the monitoring reporter.\n#monitoring.enabled: false\n\n# Uncomment to send the metrics to Elasticsearch. Most settings from the\n# Elasticsearch output are accepted here as well.\n# Note that the settings should point to your Elasticsearch *monitoring* cluster.\n# Any setting that is not set is automatically inherited from the Elasticsearch\n# output configuration, so if you have the Elasticsearch output configured such\n# that it is pointing to your Elasticsearch monitoring cluster, you can simply\n# uncomment the following line.\n#monitoring.elasticsearch:\n\n#================================= Migration ==================================\n\n# This allows to enable 6.7 migration aliases\n#migration.6_to_7.enabled: true\n```\n\n9. deploy functionbeat\n\n    ```bash\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]# pwd\n    /root/functionbeat-7.2.0-linux-x86_64\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]# ./functionbeat deploy cloudwatch\n    ```\n\n    DEBUG: if cloud formation stack that automatically deplys this fails just delete stack in console\n        https://console.aws.amazon.com/cloudformation/\u003e select stack\u003e delete stack\n        ```sh\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]# ./functionbeat deploy cloudwatch\n    Function: cloudwatch, could not deploy, error: AlreadyExistsException: Stack [fnb-cloudwatch-stack] already exists\n        status code: 400, request id: 0dc334b7-a1b5-11e9-9ff2-2d5b755700ca\n    Fail to deploy 1 function(s)\n        ```\n\n    in ec2 sshed instance see if s3 bucket exist: \n    $ aws s3 ls\n\n    DEBUG: \n    ```sh\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]# ./functionbeat deploy cloudwatch\n    Function: cloudwatch, could not deploy, error: failed to create the stack\n    Fail to deploy 1 function(s)\n    [root@ip-172-31-63-184 functionbeat-7.2.0-linux-x86_64]#\n    ```\n\n    Now in https://console.aws.amazon.com/cloudformation/ select stack and check events tab to see which event failed to debug.\n    `AWSLogs Resource Limit Exceeded - serverless `cloudwatchLog` event\n    Reason: cloudwatch logs allow only 1 subscription filter per group https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html\n\n    I just deleted manually existing subscription.\n\n\n    You can check stack console events for debug:\n\n    ![out](./img/out8.png)\n\n10.  Now you should see in cloudewatch logs console the subscription :\n\n    Log Groups                              Insights    Expire Events After     Metric Filters      Subscriptions\n    /aws/lambda/activityStreamLambda        Explore     2 weeks (14 days)       0 filters           Lambda (cloudwatch)\n\n`Now in lambda functions you should see a lambda called cloudwatch `\nhttps://console.aws.amazon.com/lambda/home?region=us-east-1#/functions/cloudwatch?tab=graph\n\n![out](./img/out5.png)\n11. To stop deployment and remove subscription:\n    $ ./functionbeat remove cloudwatch\n\n\n12. for updates to functionbeat.yml:\n\nmake and save change to functionbeat.yml\nthen:\n`$ ./functionbeat update cloudwatch`\n\n13. verify event come to kibana:\n\nin playground sns \u003e pubilsh message\u003e \n\n```json\n{\n    \"memberId\": 1,\n    \"activityDate\": \"2019-04-25T17:52:30.459   Z\",\n    \"activityDescription\": \"Completed enrollment form\",\n    \"activityType\": \"CompletedEnrollementForm\",\n    \"deviceName\": null,\n    \"activityUrl\": null,\n    \"calories\": 0,\n    \"steps\": 0,\n    \"stepType\": null,\n    \"cumulativeStepsForDay\": null,\n    \"dataCollectionMethod\": null,\n    \"duration\": null,\n    \"durationInSeconds\": 0,\n    \"distance\": null,\n    \"activeMinutes\": 0,\n    \"rowId\": null,\n    \"badgeUrl\": null,\n    \"weight\": null,\n    \"mood\": null,\n    \"accomplished\": true,\n    \"rewardsCurrency\": null,\n    \"rewardsEarned\": 0,\n    \"manuallyEntered\": false,\n    \"createdDate\": null,\n    \"amount\": 0,\n    \"entityId\": null,\n    \"height\": null,\n    \"bmi\": null,\n    \"bloodPressureSystolic\": null,\n    \"bloodPressureDiastolic\": null,\n    \"bloodPressurePulse\": null,\n    \"cholesterolTotal\": null,\n    \"glucoseFasting\": null,\n    \"bodyFat\": null,\n    \"a1c\": null,\n    \"waistCircumference\": null,\n    \"hipCircumference\": null,\n    \"memberDate\": null,\n    \"localStartDate\": null,\n    \"localEndDate\": null,\n    \"cholesterolHDL\": null,\n    \"cholesterolLDL\": null,\n    \"cholesterolTriglyceride\": null,\n    \"glucoseNonFasting\": null,\n    \"rewardable\": true,\n    \"energyBurned\": null,\n    \"energyBurnedGoal\": null,\n    \"exerciseTime\": null,\n    \"exerciseTimeGoal\": null,\n    \"standMinutes\": null,\n    \"standMinutesGoal\": null,\n    \"manualAsValidated\": false\n}\n```\n\nNow activityStreamLambda kicks in and places json in dynamodb. veirfy its there. \n\nNow logs created in activityStreamLambda should trigger the new cloudwatch lambda created.  so there should be logs for the new cloudwatch lambda:\n\n![out](./img/out6.png)\n\nNow you should see recor in elastic search:\n```sh\n[root@ip-172-31-62-80 ~]# sudo curl -XGET 'http://3.213.192.20:9200/_cat/indices?v\u0026pretty'\nhealth status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size\n\nyellow open   `functionbeat-7.2.0-2019.07.10 9yxPz0MkR96IrR-kkgjcEQ`   1   1         13            0     37.9kb         37.9kb\n```\n\n\nand kibana:\n\n(create index func*)\n![out](./img/out7.png)\n\n\n#### successful post to elasticsearch:\n![out](./img/out9.png)\n\n\n# Dockerfile for functionbeat\n\ncreate local dir structure;\nfunctionbeat \n    -| config\n        |- functionbeat.yml\n    -| Dockerfile\n\n\n### functionbeat.yml\n```sh\n\n#============================  Provider ===============================\nfunctionbeat.provider.aws.deploy_bucket: \"functionbeat-lambdalog-deploy\"\nfunctionbeat.provider.aws.functions:\n  - name: cloudwatchlogs\n    enabled: true\n    type: cloudwatch_logs\n    description: \"lambda function for cloudwatch logs\"\n    triggers:\n      - log_group_name: /aws/lambda/activityStreamLambda\n\n#==================== Elasticsearch template setting ==========================\nsetup.template.settings:\n  index.number_of_shards: 1\n\n#================================ Outputs =====================================\noutput.elasticsearch:\n  hosts: [$HOSTS]\n  enabled: true\n\n#================================ Processors =====================================\nprocessors:\n  - add_host_metadata: ~\n  - add_cloud_metadata: ~\n  - add_fields:\n      target: project\n      fields:\n        name: insights-lambda\n\n\n#================================ Logging =====================================\nlogging.level: debug\n```\n\n\nRef: https://www.elastic.co/guide/en/beats/functionbeat/6.7/using-environ-vars.html\n\nYou can specify complex objects, such as lists or dictionaries, in environment variables by using a JSON-like syntax.\n\nAs with JSON, dictionaries and lists are constructed using {} and []. But unlike JSON, the syntax allows for trailing commas and slightly different string quotation rules. Strings can be unquoted, single-quoted, or double-quoted, as a convenience for simple settings and to make it easier for you to mix quotation usage in the shell. \n\nArrays at the top-level do not require brackets ([]).\n\nFor example, the following environment variable is set to a `list`:\n\n```json\nES_HOSTS=\"10.45.3.2:9220,10.45.3.1:9230\"\n```\n\n\nYou can reference this variable in the config file:\n\n```json\noutput.elasticsearch:\n  hosts: '${ES_HOSTS}'\n```\n\nWhen Functionbeat loads the config file, it resolves the environment variable and replaces it with the specified list before reading the hosts settin\n\n\n### Dockerfile\n```sh\n#FROM alpine:3.7\nFROM ubuntu:14.04\n\n# source aws configuration\nARG aws_access_key_id\nENV AWS_ACCESS_KEY_ID=$aws_access_key_id\nARG aws_secret_access_key\nENV AWS_SECRET_ACCESS_KEY=$aws_secret_access_key\nARG aws_default_region\nENV AWS_DEFAULT_REGION=$aws_default_region\nARG elasticsearch_hosts\nENV HOSTS=['$elasticsearch_hosts']\n\n#RUN export AWS_ACCESS_KEY_ID=$aws_access_key_id\n#RUN export AWS_SECRET_ACCESS_KEY=$aws_secret_access_key\n#RUN export AWS_DEFAULT_REGION=$aws_default_region\n#RUN export HOSTS=$elasticsearch_hosts\n\n# install curl\nRUN apt-get update\nRUN apt-get -y install curl\n\n\n# download and extract functionbeat\nRUN apt install curl\nRUN curl -L -O https://artifacts.elastic.co/downloads/beats/functionbeat/functionbeat-7.2.0-linux-x86_64.tar.gz\nRUN tar xvzf functionbeat-7.2.0-linux-x86_64.tar.gz\n\n# copy configuration\nCOPY config/functionbeat.yml functionbeat-7.2.0-linux-x86_64/functionbeat.yml\n\n# run functionbeat\nWORKDIR \"functionbeat-7.2.0-linux-x86_64\"\nCMD ./functionbeat deploy cloudwatch\n\n```\n\n### build image:\n\ncd into directory containing dockerfile and run\n\n```sh\ndocker build --build-arg aws_access_key_id=xxxx --build-arg aws_secret_access_key=xxxx --build-arg aws_default_region=\"us-east-1\" --build-arg elasticsearch_hosts=3.213.192.20:9200  --build-arg elasticsearch_index=\"playground-applications\" --build-arg bucket_name=\"functionbeat-lambdalog-deploy-playground\" --build-arg aws_account_id=\"219708553334\" --build-arg security_group_ids=\"sg-1838b463\"  --build-arg subnet_ids=\"subnet-154d1c28,subnet-956304cd\" -t functionbeat .\n```\n\n\n### create and run image container\n\ndocker stop functionbeat; docker rm functionbeat\ndocker rmi functionbeat\n\n`$ docker run --name functionbeat functionbeat`\n\nNow you should see deployed cloudwatch lambda and the subscription to activityStream log stream\n\n### run functionbeat command\nlist help:\n`$ docker run functionbeat ./functionbeat --help `\n\nremove :\n`$ docker run functionbeat ./functionbeat remove cloudwatch`\n\n\n### debug docker run.sh\nrun.sh\n```sh\n#!/usr/bin/env bash\n#./functionbeat update cloudwatch || ./functionbeat deploy cloudwatch\n\nsucess=false\n\noutput=$(./functionbeat update cloudwatch)\n\nif echo output | grep -q \"ValidationError: No updates are to be performed\"; then\n    sucess=true\n    echo \"cloudwatch functionbeat already up-to-date\"\nelif echo output | grep -q \"ValidationError: Stack [fnb-cloudwatch-stack] does not exist\"; then\n    output = $(./functionbeat deploy cloudwatch)\n    if echo output | grep -q \"deploy successful\"; then\n        sucess=true\n        output \"cloudwatch functionbeat deployed\"\n    fi\nelse\n     echo \"something went wrong: \", $output\nfi\n\necho \"success:\" $suces\n```\n\n\n(base)  rdissanayakam@RBH12855  ~/vp/gitlab/erewhon/functionbeat   functionbeat ●✚  `docker run functionbeat  `                       \nfail to deploy 1 function(s)\nsomething went wrong: , Function: cloudwatch, could not update, error: ValidationError: No updates are to be performed. status code: 400, request id: 09de30b8-a4d0-11e9-be17-73c8d11b0605\nsuccess: false\n(base)  rdissanayakam@RBH12855  ~/vp/gitlab/erewhon/functionbeat   functionbeat ●✚  docker ps -a | grep func                        \n986dcce26736        functionbeat                                          \"/bin/sh -c 'sh ./ru…\"   2 minutes ago       Exited (0) About a minute ago                                                                                                           gallant_sinoussi\n(base)  rdissanayakam@RBH12855  ~/vp/gitlab/erewhon/functionbeat   functionbeat ●✚  `docker start gallant_sinoussi`\ngallant_sinoussi\n(base)  ✘ rdissanayakam@RBH12855  ~/vp/gitlab/erewhon/functionbeat   functionbeat ●✚  d`ocker exec -it gallant_sinoussi bash    `\nroot@986dcce26736:/functionbeat-7.2.0-linux-x86_64# `ls`\nLICENSE.txt  NOTICE.txt  README.md  data  fields.yml  functionbeat  functionbeat.reference.yml  functionbeat.yml  kibana  logs  pkg  run.sh\nroot@986dcce26736:/functionbeat-7.2.0-linux-x86_64# output=$(./functionbeat update cloudwatch)\n\n\n\n## Errors\n\n\n### try to deploy when /aws/lambda/activityStreamlambda (input logstream) does not exist\n\nFail to deploy 1 function(s)\nFunction: cloudwatch, could not deploy, error: failed to create the stack\ncomplete deploying cloudwatch lambda\n\n### try to deploy when cloudwatch stack already exist (deployed before)\nFunction: cloudwatch, could not deploy, error: AlreadyExistsException: Stack [fnb-cloudwatch-stack] already exists\n\tstatus code: 400, request id: 2bc27a50-a73d-11e9-bd32-a5f4266302d2\nFail to deploy 1 function(s)\ncomplete deploying cloudwatch lambda\n\n\n### try yo update when not cloudwatch stack does not exist (not deployed before)\n\nfail to deploy 1 function(s)\nFunction: cloudwatch, could not update, error: ValidationError: Stack [fnb-cloudwatch-stack] does not exist\n\tstatus code: 400, request id: 034a2d8a-a73c-11e9-80ea-391c7d95c316\nFail to deploy 1 function(s)\n\n### try to update when there are no updates\nfail to deploy 1 function(s)\nFunction: cloudwatch, could not update, error: ValidationError: No updates are to be performed.\n\tstatus code: 400, request id: 2bc7408f-a7d2-11e9-8353-c32466e03c9b  \n\n### try to update but cloudwatch stack in rolledback state\nFunction: cloudwatch, could not update, error: ValidationError: Stack:arn:aws:cloudformation:us-east-1:219708553334:stack/fnb-cloudwatch-stack/05a8b690-a73c-11e9-83f3-12d99c723360 is in ROLLBACK_COMPLETE state and can not be updated.\n\tstatus code: 400, request id: 2951efbd-a73d-11e9-b392-45c325678c6a\nfail to deploy 1 function(s)\n\nreason: look at the events in cloudformation stack:\n\nThe role defined for the function cannot be assumed by Lambda. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: ec2aa631-08ad-4cc4-8586-bd54b8bbdc2c)\n\nA delete of stack deletes the lambda role\n\n\n## Dockerfile you can install aws cli and run aws commands:\n\n\n`$ aws logs describe-log-groups --log-group-name-prefix \"/aws/lambda/activityStreamLambda\"`\n{\n    \"logGroups\": [\n        {\n            \"arn\": \"arn:aws:logs:us-east-1:219708553334:log-group:/aws/lambda/activityStreamLambda:*\", \n            \"creationTime\": 1563221619494, \n            \"metricFilterCount\": 0, \n            \"logGroupName\": \"/aws/lambda/activityStreamLambda\", \n            \"storedBytes\": 0\n        }\n    ]\n}\n\n`$ aws logs describe-log-groups --log-group-name-prefix \"/aws/lambda/activityStreamLambda2\"`\n\n{\n    \"logGroups\": []\n}\n\n## cloudformation stack:\n\n`aws cloudformation describe-stacks --stack-name fnb-cloudwatch-stack`\n{\n    \"Stacks\": [\n        {\n            \"StackId\": \"arn:aws:cloudformation:us-east-1:219708553334:stack/fnb-cloudwatch-stack/e191b690-a7d0-11e9-a3b7-121b5b00f9e8\",\n            \"StackName\": \"fnb-cloudwatch-stack\",\n            \"CreationTime\": \"2019-07-16T13:51:59.186Z\",\n            \"RollbackConfiguration\": {},\n            \"StackStatus\": \"CREATE_COMPLETE\",\n            \"DisableRollback\": false,\n            \"NotificationARNs\": [],\n            \"Capabilities\": [\n                \"CAPABILITY_NAMED_IAM\"\n            ],\n            \"Tags\": [],\n            \"EnableTerminationProtection\": false\n        }\n    ]\n}\n\n`aws cloudformation describe-stacks --stack-name fnb-cloudwatch-stack2-non-existing`\n\nAn error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id fnb-cloudwatch-stack2 does not exist\n\n\n# use aws cli\n\nif you have a profile for a aws account in ~/.aws/credentials:\n(base)  rdissanayakam@RBH12855  ~/vp  cat ~/.aws/credentials \n[default]\naws_access_key_id = xxxx\naws_secret_access_key = xxxx\n\n[playground]\naws_access_key_id = xxxx\naws_secret_access_key = xxxx\n\n[qagitlab]\n`aws_access_key_id = xxxx\n`aws_secret_access_key = xxxx`\n\nNow you can specify profile and region to run commands in that aws account without having to export any keys:\n\n`aws logs describe-log-groups --profile qagitlab --region \"us-east-1\"`\n\n```json\n{\n    \"logGroups\": [\n        {\n            \"logGroupName\": \"/aws/lambda/activityStreamLambda\",\n            \"creationTime\": 1559574028442,\n            \"metricFilterCount\": 0,\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/activityStreamLambda:*\",\n            \"storedBytes\": 54485919789\n        },\n        {\n            \"logGroupName\": \"/aws/lambda/cloudwatch\",\n            \"creationTime\": 1563457691444,\n            \"metricFilterCount\": 0,\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:*\",\n            \"storedBytes\": 0\n        }\n    ]\n}\n\n```\n\nview log streams in aws/lambda/cloudwatch logstreams:\n\n`aws logs describe-log-streams --log-group-name /aws/lambda/cloudwatch  --profile qagitlab --region \"us-east-1\"`\n\n```json\n{\n    \"logStreams\": [\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]1c7eb80068b04e4785ee36c2c52107a6\",\n            \"creationTime\": 1563463290187,\n            \"firstEventTimestamp\": 1563463292304,\n            \"lastEventTimestamp\": 1563463292304,\n            \"lastIngestionTime\": 1563464119149,\n            \"uploadSequenceToken\": \"49597108459857708445888670258489252953444985484909881474\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]1c7eb80068b04e4785ee36c2c52107a6\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]20bcb3c0ccdd4c9eae0d0fbe39d2c1ce\",\n            \"creationTime\": 1563463290212,\n            \"firstEventTimestamp\": 1563463292162,\n            \"lastEventTimestamp\": 1563463292162,\n            \"lastIngestionTime\": 1563464118961,\n            \"uploadSequenceToken\": \"49597124621207753821046875876469688693968748011181179458\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]20bcb3c0ccdd4c9eae0d0fbe39d2c1ce\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]2be30f4ab10f4e7eb9dad0153c2fe103\",\n            \"creationTime\": 1563457727989,\n            \"firstEventTimestamp\": 1563457728915,\n            \"lastEventTimestamp\": 1563463361814,\n            \"lastIngestionTime\": 1563463361824,\n            \"uploadSequenceToken\": \"49597464168384614002961182721547714094624459578952562210\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]2be30f4ab10f4e7eb9dad0153c2fe103\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]3ee6d8a95ac8411498a55cd8bab80e40\",\n            \"creationTime\": 1563457728128,\n            \"firstEventTimestamp\": 1563457729096,\n            \"lastEventTimestamp\": 1563462944647,\n            \"lastIngestionTime\": 1563463353843,\n            \"uploadSequenceToken\": \"49596895203451162687568998200904080691176482277761017282\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]3ee6d8a95ac8411498a55cd8bab80e40\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]4a1af33c001949ee90bde0fcfd1b9d13\",\n            \"creationTime\": 1563463290144,\n            \"firstEventTimestamp\": 1563463292690,\n            \"lastEventTimestamp\": 1563463711297,\n            \"lastIngestionTime\": 1563464117404,\n            \"uploadSequenceToken\": \"49596804279544162339679056254223931068498803953192141234\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]4a1af33c001949ee90bde0fcfd1b9d13\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]775728d5d0bc457e88b987e114a7cf96\",\n            \"creationTime\": 1563457727990,\n            \"firstEventTimestamp\": 1563457728947,\n            \"lastEventTimestamp\": 1563463351594,\n            \"lastIngestionTime\": 1563463351603,\n            \"uploadSequenceToken\": \"49597164569402357011681385699256019577984614593156502258\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]775728d5d0bc457e88b987e114a7cf96\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]7f01ac20f9ab4bdd94843bddf687994b\",\n            \"creationTime\": 1563457728075,\n            \"firstEventTimestamp\": 1563457729058,\n            \"lastEventTimestamp\": 1563462915576,\n            \"lastIngestionTime\": 1563463352085,\n            \"uploadSequenceToken\": \"49597148102799711359049418256589390946717411633353069746\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]7f01ac20f9ab4bdd94843bddf687994b\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]d3f7a8d29bd84cad8dac1c998eb0e535\",\n            \"creationTime\": 1563463290042,\n            \"firstEventTimestamp\": 1563463292099,\n            \"lastEventTimestamp\": 1563463490813,\n            \"lastIngestionTime\": 1563464117591,\n            \"uploadSequenceToken\": \"49596779053587915372591227432812526796506709955879093122\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]d3f7a8d29bd84cad8dac1c998eb0e535\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]d6c15e0a7d5345cb940cdec41fff282f\",\n            \"creationTime\": 1563457728137,\n            \"firstEventTimestamp\": 1563457729100,\n            \"lastEventTimestamp\": 1563462938672,\n            \"lastIngestionTime\": 1563463360744,\n            \"uploadSequenceToken\": \"49597583964285896806512608661348517577311451405582002114\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]d6c15e0a7d5345cb940cdec41fff282f\",\n            \"storedBytes\": 0\n        },\n        {\n            \"logStreamName\": \"2019/07/18/[$LATEST]e1ccb90b8f5a40898813a244cf3f9d63\",\n            \"creationTime\": 1563463290200,\n            \"firstEventTimestamp\": 1563463292545,\n            \"lastEventTimestamp\": 1563463292545,\n            \"lastIngestionTime\": 1563464118633,\n            \"uploadSequenceToken\": \"49597147888868662669544800575863167953362365475052729122\",\n            \"arn\": \"arn:aws:logs:us-east-1:1234:log-group:/aws/lambda/cloudwatch:log-stream:2019/07/18/[$LATEST]e1ccb90b8f5a40898813a244cf3f9d63\",\n            \"storedBytes\": 0\n        }\n    ]\n}\n\n```\n\n## view logstrem event :\n\n`aws logs get-log-events --log-group-name '/aws/lambda/cloudwatch' --log-stream-name '2019/07/18/[$LATEST]1c7eb80068b04e4785ee36c2c52107a6'  --profile qagitlab --region \"us-east-1\" \u003e1c7eb80068b04e4785ee36c2c52107a6.json`\n\n\n## describe load balancers (list subnet id, security group)\n\n`aws elb describe-load-balancers --profile qagitlab --region us-east-1`\n\n\nfind ip resolution: 10.3.x.x\n`$ nslookup internal-ops-elasticsearch-1755344086.us-east-1.elb.amazonaws.com`\nServer:\t\t192.168.86.1\nAddress:\t192.168.86.1#53\n\nNon-authoritative answer:\nName:\tinternal-ops-elasticsearch-1755344086.us-east-1.elb.amazonaws.com\nAddress: 10.3.6.192\nName:\tinternal-ops-elasticsearch-1755344086.us-east-1.elb.amazonaws.com\nAddress: 10.3.7.184\n\n\nfind subnets of a vpc:\n\n`aws ec2 describe-subnets --filters \"Name=vpc-id,Values=vpc-bccb26d8\" --profile qagitlab --region us-east-1`\n\nfiltered by tags:\n`rdissanayakam@rbh12855  ~/vp/aws_access_keys  aws ec2 describe-subnets  --filters Name=tag:group,Values=functionbeat --profile stagegitlab --region us-east-1`\n\n```json\n{\n    \"Subnets\": [\n        {\n            \"AvailabilityZone\": \"us-east-1a\",\n            \"AvailableIpAddressCount\": 11,\n            \"CidrBlock\": \"10.1.53.0/28\",\n            \"DefaultForAz\": false,\n            \"MapPublicIpOnLaunch\": false,\n            \"State\": \"available\",\n            \"SubnetId\": \"subnet-0d226e854c363d497\",\n            \"VpcId\": \"vpc-29f3ce4c\",\n            \"AssignIpv6AddressOnCreation\": false,\n            \"Ipv6CidrBlockAssociationSet\": [],\n            \"Tags\": [\n                {\n                    \"Key\": \"Name\",\n                    \"Value\": \"stage-functionbeat-lambda-a-sub\"\n                },\n                {\n                    \"Key\": \"group\",\n                    \"Value\": \"functionbeat\"\n                },\n                {\n                    \"Key\": \"env\",\n                    \"Value\": \"stage\"\n                }\n            ]\n        },\n        {\n            \"AvailabilityZone\": \"us-east-1b\",\n            \"AvailableIpAddressCount\": 11,\n            \"CidrBlock\": \"10.1.53.16/28\",\n            \"DefaultForAz\": false,\n            \"MapPublicIpOnLaunch\": false,\n            \"State\": \"available\",\n            \"SubnetId\": \"subnet-004f094a358f275cd\",\n            \"VpcId\": \"vpc-29f3ce4c\",\n            \"AssignIpv6AddressOnCreation\": false,\n            \"Ipv6CidrBlockAssociationSet\": [],\n            \"Tags\": [\n                {\n                    \"Key\": \"Name\",\n                    \"Value\": \"stage-functionbeat-lambda-b-sub\"\n                },\n                {\n                    \"Key\": \"env\",\n                    \"Value\": \"stage\"\n                },\n                {\n                    \"Key\": \"group\",\n                    \"Value\": \"functionbeat\"\n                }\n            ]\n        }\n    ]\n}\n```\n(base)  rdissanayakam@rbh12855  ~/vp/aws_access_keys  \n\nview security groups:\n\n`aws ec2 describe-security-groups --filters Name=vpc-id,Values=vpc-29f3ce4c --profile stagegitlab --region us-east-1 `\n\n`aws ec2 describe-security-groups --filters Name=group-name,Values=allow-functionbeat-sg --profile qagitlab --region=us-east-1`\n\n```json\n{\n    \"SecurityGroups\": [\n        {\n            \"Description\": \"Security group for lambda to move logs from cloudwatch to elasticsearch\",\n            \"GroupName\": \"allow-functionbeat-sg\",\n            \"IpPermissions\": [],\n            \"OwnerId\": \"1234\",\n            \"GroupId\": \"sg-0a0eaa390b0292fb4\",\n            \"IpPermissionsEgress\": [\n                {\n                    \"IpProtocol\": \"-1\",\n                    \"IpRanges\": [\n                        {\n                            \"CidrIp\": \"0.0.0.0/0\"\n                        }\n                    ],\n                    \"Ipv6Ranges\": [],\n                    \"PrefixListIds\": [],\n                    \"UserIdGroupPairs\": []\n                }\n            ],\n            \"Tags\": [\n                {\n                    \"Key\": \"Name\",\n                    \"Value\": \"cloudwatch-functionbeat-lambda-sg\"\n                }\n            ],\n            \"VpcId\": \"vpc-bccb26d8\"\n        }\n    ]\n}```\n(base)  rdissanayakam@RBH12855  ~/vp/aws_access_keys  \n\n\n`aws ec2 describe-security-groups --profile qagitlab --group-names allow-functionbeat-sg --region us-east-1`           \n{\n    \"SecurityGroups\": [\n        {\n            \"Description\": \"Security group for lambda to move logs from cloudwatch to elasticsearch\",\n            \"GroupName\": \"allow-functionbeat-sg\",\n            \"IpPermissions\": [],\n            \"OwnerId\": \"1234\",\n            \"GroupId\": \"sg-0ac480210436baaa1\",\n            \"IpPermissionsEgress\": [\n                {\n                    \"IpProtocol\": \"-1\",\n                    \"IpRanges\": [\n                        {\n                            \"CidrIp\": \"0.0.0.0/0\"\n                        }\n                    ],\n                    \"Ipv6Ranges\": [],\n                    \"PrefixListIds\": [],\n                    \"UserIdGroupPairs\": []\n                }\n            ],\n            \"Tags\": [\n                {\n                    \"Key\": \"Name\",\n                    \"Value\": \"cloudwatch-functionbeat-lambda-sg\"\n                }\n            ],\n            \"VpcId\": \"vpc-7884611d\"\n        }\n    ]\n}\n\ncheck if member id record exist in dynamodb table:\n\n`aws dynamodb get-item --table-name healthy_habit_training --key '{\"member_id\": {\"N\": \"1\"}}'  --profile playgroundgitlab  --region us-east-1`\n{\n    \"Item\": {\n        \"member_id_mod1k\": {\n            \"N\": \"1\"\n        },\n        \"member_id\": {\n            \"N\": \"1\"\n        },\n        \"CompletedEnrollementForm\": {\n            \"N\": \"5\"\n        }\n    }\n}\n(base)  rdissanayakam@rbh12855  ~/vp  \n\n\n# Elasticsearch behind a elastic load balancer in different vpc than cloudwatch lambda (SAME AWS ACCONT, DIFFERENT VPCs)\n\n## bring up ec2 instance and install elasticsearch \n\n- Use readme above to do it.\n\n- Create elastic ip \u003e associate with instance to get a public ip.\n\n- note down the vpc-id of elasticsearch ec2 instance\n\n![out](./img/out10.png)\n\n## Create elasticload balancer\n\nServices \u003e EC2 \u003e Load Balancers in left pane \n\n# Purge queue\n\n`aws sqs purge-queue --queue-url https://sqs.us-east-1.amazonaws.com/028037478894/STAGE_hh_recommender_training_activity_queue --profile stagegitlab --region=us-east-1`\n\nGet started with New Relic Infrastructure\n\nBegin your trial by following these simple steps to install the agent on an unlimited number of hosts!\n\nSelect platform:\n\n\nLinux\n\nWindows\nSelect installer:\n\nyum\napt\nzypper\nyumAmazon Linux, CentOS, RHELaptDebian, Ubuntu\nzypperSLES\nCreate a configuration file and add your license key\n\necho \"license_key: xxxx\" | sudo tee -a /etc/newrelic-infra.yml\nYou can add extra configurations that fine-tune your agent’s behavior and collect custom attributes.\nCreate the agent’s yum repository\n\nAmazon Linux, CentOS 6, RHEL 6\n\nsudo curl -o /etc/yum.repos.d/newrelic-infra.repo https://download.newrelic.com/infrastructure_agent/linux/yum/el/6/x86_64/newrelic-infra.repo\nAmazon Linux 2, CentOS 7, RHEL 7\n\nsudo curl -o /etc/yum.repos.d/newrelic-infra.repo https://download.newrelic.com/infrastructure_agent/linux/yum/el/7/x86_64/newrelic-infra.repo\nCentOS 8, RHEL 8\n\nsudo curl -o /etc/yum.repos.d/newrelic-infra.repo https://download.newrelic.com/infrastructure_agent/linux/yum/el/8/x86_64/newrelic-infra.repo\nUpdate your yum cache\n\nsudo yum -q makecache -y --disablerepo='*' --enablerepo='newrelic-infra'\nRun the installation script\n\nsudo yum install newrelic-infra -y\nGet started\n\nWait a few seconds, and your data should begin appearing!\n\nYou can integrate with your Amazon Elastic Compute Cloud (Amazon EC2) account for dynamic-tagging and autoscaling properties\n\n\n## run.py (Not used)\n\n```python\n#!/usr/bin/python\nimport time\nimport boto3\nimport subprocess\nfrom botocore.exceptions import ClientError\n\ncloudformation_client = boto3.client(\"cloudformation\")\nlogs_client = boto3.client(\"logs\")\n\nLOG_GROUP_NAME=\"/aws/lambda/activityStreamLambda\"\nCLOUDFORMATION_STACK_NAME=\"fnb-cloudwatch-stack\"\n\n\n\ndef update_functionbeat():\n    subprocess_output=''\n    try:\n        #subprocess_output = subprocess.run([\"./functionbeat\", \"-v\", \"-e\",  \"update\", \"cloudwatch\", \"capture_output=True\"])\n        #print(\"update output = \"+str(subprocess_output))\n        pipe = subprocess.Popen( './functionbeat -v -e -d \"*\" update cloudwatch', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE )\n        res = pipe.communicate()\n        print(\"retcode =\", pipe.returncode)\n        print(\"res =\", res)\n        print(\"stderr =\", res[1])\n    except subprocess.CalledProcessError as e:\n        print(\"Unexpected error in update_functionbeat return code: \" + e.returncode + \", output:\\n\" + e.output)\n    print(\"update completed!\")\n\n\ndef deploy_functionbeat():\n    subprocess_output=''\n    try:\n        subprocess_output = subprocess.run([\"./functionbeat\", \"-v\", \"-e\",  \"deploy\", \"cloudwatch\", \"capture_output=True\"])\n        print(\"deployment output = \"+str(subprocess_output))\n    except subprocess.CalledProcessError as e:\n        print(\"Unexpected error in update_functionbeat return code: \" + e.returncode + \", output:\\n\" + e.output)\n    print(\"deploy completed!\")\n\n\ndef stack_exists_with_status(name, required_status = [\"CREATE_COMPLETE\",]):\n    try:\n        data = cloudformation_client.describe_stacks(StackName = name)\n    except ClientError as e:\n        print(\"Unexpected error in stack_exists_with_status: %s\" % e)\n        # if e.response['Error']['Code'] == 'EntityAlreadyExists':\n        #     print(\"User already exists\")\n        # else:\n        #     print(\"Unexpected error: %s\" % e)\n        return False\n    return data[\"Stacks\"][0][\"StackStatus\"] in required_status\n\n\ndef stack_in_failed_state(name):\n    required_status = [\"CREATE_FAILED\", \"DELETE_FAILED\", \"ROLLBACK_FAILED\", \"UPDATE_ROLLBACK_FAILED\" ]\n    return stack_exists_with_status(name, required_status)\n\n\ndef stack_in_inprogress_state(name):\n    required_status = [\"CREATE_IN_PROGRESS\", \"DELETE_IN_PROGRESS\", \"REVIEW_IN_PROGRESS\", \"ROLLBACK_IN_PROGRESS\", \"UPDATE_COMPLETE_CLEANUP_IN_PROGRESS\", \"UPDATE_IN_PROGRESS\", \"UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS\", \"UPDATE_ROLLBACK_IN_PROGRESS\"]\n    return stack_exists_with_status(name, required_status)\n\n\ndef stack_in_completed_state(name):\n    required_status = [\"CREATE_COMPLETE\", \"DELETE_COMPLETE\", \"ROLLBACK_COMPLETE\", \"UPDATE_COMPLETE\", \"UPDATE_ROLLBACK_COMPLETE\"]\n    return stack_exists_with_status(name, required_status)\n\n\ndef check_stack_exists(name):\n    try:\n        data = cloudformation_client.describe_stacks(StackName = name)\n    except ClientError as e:\n        print(\"Unexpected error in check_stack_exists: %s\" % e)\n        return False\n    return True\n\n\ndef get_stack_status(name):\n    try:\n        data = cloudformation_client.describe_stacks(StackName = name)\n    except ClientError as e:\n        print(\"Unexpected error in get_stack_status: %s\" % e)\n        return \"Stack Does Not Exist\"\n    return data[\"Stacks\"][0][\"StackStatus\"]\n\n\ndef delete_stack(name):\n    try:\n        data = cloudformation_client.delete_stack(StackName = name)\n    except ClientError as e:\n        print(\"Unexpected error in delete_stack: %s\" % e)\n        return False\n    return True\n\n\ndef wait_till_stack_deletion_completes(name):\n    stack_delete_complete=False\n    while True:\n        exists = stack_exists(name)\n        if(exists == False):\n            stack_delete_complete=True\n            break\n        else:\n            print(\"wating for delete completion...\")\n            time.sleep( 5 )\n    return stack_delete_complete\n\n\ndef wait_till_stack_progress_completes(name):\n    while True:\n        exists = stack_exists(name)\n        status  = get_stack_status(name)\n        if (stack_exists_with_status(\"CREATE_IN_PROGRESS\", \"DELETE_IN_PROGRESS\", \"REVIEW_IN_PROGRESS\", \"ROLLBACK_IN_PROGRESS\", \"UPDATE_IN_PROGRESS\", \"UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS\", \"UPDATE_ROLLBACK_IN_PROGRESS\")):\n            print(\"wating for progrss state completion...\")\n            time.sleep( 5 )\n        else:\n            break\n\n\ndef check_log_group_exists(name):\n    try:\n        data = logs_client.describe_log_groups(logGroupNamePrefix=name)\n    except ClientError as e:\n        rint(\"Unexpected error in check_log_group_exists: %s\" % e)\n        return False\n    return True\n\ndef main():\n    # check cloudformation stack fnb-cloudwatch-stack exist\n    stack_exists = check_stack_exists(CLOUDFORMATION_STACK_NAME)\n    # check cloudformation stack fnb-cloudwatch-stack status\n    stack_status =  get_stack_status(CLOUDFORMATION_STACK_NAME) if stack_exists == True else \"\"\n\n\n    # if stack exist\n    if (stack_exists) :\n        print(\"checking if cloudformation stack in in_progress or failed state....\")\n        # if stack is in one of IN_PROGRESS status\n        if (stack_in_inprogress_state(CLOUDFORMATION_STACK_NAME)):\n            print(\"stack in IN_PROGRSS stage. waiting for completion.....\")\n            wait_till_stack_progress_completes(CLOUDFORMATION_STACK_NAME)   # wait for IN_PROGRESS to change status\n        # if stack status is in one of  FAILED status\n        if(stack_in_failed_state(CLOUDFORMATION_STACK_NAME) ):\n            print(\"stack in IN_PROGRSS stage. waiting for deletion.....\")\n            delete_stack(CLOUDFORMATION_STACK_NAME)         # call delete stack\n            wait_till_stack_deletion_completes(CLOUDFORMATION_STACK_NAME)         # verify deleted\n            print(\"stack delete complete.\")\n        # else in COMPLETE state\n            # continue\n    else:\n        print(\"no functionbeat cloudformation stacks found\")\n\n\n    # ckeck log group for activityStreamLambda exist\n    log_group_exists = check_log_group_exists(LOG_GROUP_NAME)\n    # check if stack exist\n    stack_exists = check_stack_exists(CLOUDFORMATION_STACK_NAME)\n\n    # if log group exists\n    if (log_group_exists):\n        print(\"log group exist!\")\n        # if stack exist\n        if (stack_exists):\n            print(\"stack exist!\")\n            # if stack is not in one in a failed or in progress state\n            if (stack_in_completed_state(CLOUDFORMATION_STACK_NAME)):\n                print(\"stack is in non-fail state. Attempting updating functionbeat...\")\n                # call update on functionbeat\n                update_functionbeat()\n        else:\n            # call deploy on functionbeat\n            print(\"no stack exist. Attempting deploying functionbeat...\")\n            deploy_functionbeat()\n    else:\n        print(\"log group does not exist\")\n\n\nif __name__ == '__main__':\n    main()\n\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmrhimali%2Ffunctionbeatdemo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmrhimali%2Ffunctionbeatdemo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmrhimali%2Ffunctionbeatdemo/lists"}