https://github.com/ev2900/logstash_example

Logstash log collection, parsing example
https://github.com/ev2900/logstash_example

aws cloud9 cloudformation logstash opensearch

Last synced: 10 days ago
JSON representation

Logstash log collection, parsing example

• Host: GitHub
• URL: https://github.com/ev2900/logstash_example
• Owner: ev2900
• Created: 2022-09-27T14:36:07.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-11-24T15:38:58.000Z (about 1 month ago)
• Last Synced: 2024-11-24T16:34:00.408Z (about 1 month ago)
• Topics: aws, cloud9, cloudformation, logstash, opensearch
• Homepage:
• Size: 176 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Send logs from Logstash on Cloud9 to OpenSearch

  

Follow the instructions below

1. Run the CloudFormation stack below. It will create the required resources required for this example

[![Launch CloudFormation Stack](https://sharkech-public.s3.amazonaws.com/misc-public/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=logstash-opensearch&templateURL=https://sharkech-public.s3.amazonaws.com/misc-public/logstash_cloud9_s3.yaml)

The resources created by the CloudFormation stack are documented in the architecture below



2. Open the Cloud9 environment and install Logstash. Complete all of the subsequent steps in the Cloud9 terminal

Download logstash

```curl https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-7.16.2-linux-x64.tar.gz -o logstash-oss-with-opensearch-output-plugin-7.16.2-linux-x64.tar.gz```

Uncompress download

```tar -zxvf logstash-oss-with-opensearch-output-plugin-7.16.2-linux-x64.tar.gz```

3. Configure Logstash

Create configuration file

```sudo vim logstash-config.conf```

Copy / paste the following into the the ```logstash-config.conf``` file. Replace the *path*, *hosts*, *user*, *password* parts of the config

```

input {

    file {

        path => ""

        start_position => "beginning"

    }

}

output {

    opensearch {

        hosts       => [":443"]

        user        => ""

        password    => ""

        index       => "logstash-logs-%{+YYYY.MM.dd}"

    }

}

```

4. Run Logstash

```/home/ec2-user/environment/logstash-7.16.2/bin/logstash -f /home/ec2-user/environment/logstash-config.conf```

5. Add logs to the log file / folder specified by the path. If you need sample log data you can use the following

*Note* you made need to modify the premisions on the log file via. ```chmod 777 logstash-config.conf```

```

{"timestamp":1661869220203, "CPU": 90, "Message": "High CPU"}

{"timestamp":1661869220203, "Memory": 90, "Message": "High Memory"}

{"timestamp":1661869220203, "Disk": 80, "Message": "High Disk"}

{"timestamp":1661869220203, "Network": 100, "Message": "High Network"}

```