https://github.com/barrettotte/elastic-markov
Learning Elastic stack fundamentals by ingesting sentences generated by a Markov chain.
https://github.com/barrettotte/elastic-markov
docker-elk elasticstack markov-chain
Last synced: 3 months ago
JSON representation
Learning Elastic stack fundamentals by ingesting sentences generated by a Markov chain.
- Host: GitHub
- URL: https://github.com/barrettotte/elastic-markov
- Owner: barrettotte
- License: mit
- Created: 2022-11-06T15:10:31.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2022-11-11T19:29:29.000Z (over 2 years ago)
- Last Synced: 2025-02-03T13:13:22.220Z (5 months ago)
- Topics: docker-elk, elasticstack, markov-chain
- Language: Python
- Homepage:
- Size: 23.4 KB
- Stars: 0
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# elastic-markov
Learning Elastic stack fundamentals by ingesting sentences generated by a Markov chain.
## Summary
markov-gen.stdout => FileBeat => LogStash => ElasticSearch <= Kibana
- **Markov** outputs random sentences, usernames, and IPv4 addresses to stdout.
The random sentences are generated from a Markov chain trained on all sentences from Moby Dick
- **FileBeat** ingests logs from `markov-gen` container and sends to LogStash
- **LogStash** accepts FileBeat input, processes/enriches it, and pushes to ElasticSearch.
Specifically, the random IPv4 addresses are used to enrich logs with GeoIP data.
- **ElasticSearch** stores and indexes data
- **Kibana** provides visualization over ElasticSearch data and other things
## Run
`docker compose -f docker-compose.yml up --build` or `./start.sh`
Note: Logstash takes way longer to start than you'd think...just keep an eye on the logstash logs
## Logs
Format: `{timestamp} {log level} {username} {ipv4} {sentence}`
Sample logs from `markov-gen`, generated from a Markov chain trained on Moby Dick.
```txt
2022-11-11 19:25:18,226 INFO gloomyLlama9 218.93.249.20 I might proceed with several more examples, one way or other, has this Siamese connexion with a plurality of other mortals.
2022-11-11 19:25:19,244 INFO peskyBurritos4 12.171.13.9 For, spite of her leak, and spite of our hootings, for a long time.
2022-11-11 19:25:22,261 INFO awedThrush5 239.5.175.122 But by those clanking links, the vast corpse itself, not the ship, is to be found, the aliment of the right whale.
2022-11-11 19:25:24,277 INFO cruelCordial2 246.153.8.47 Now when I looked about the quarter-deck, for some one lone whale on the outskirts of the shoal.
2022-11-11 19:25:25,309 INFO debonairLard4 9.193.18.31 As many know, he wrote the history of his own sovereign will and pleasure.
2022-11-11 19:25:27,325 INFO cruelBagels2 147.186.63.244 But the time of capture.
2022-11-11 19:25:30,328 INFO needfulLizard5 89.48.243.197 Was it that this old carpenter had been thrusting his shavings into the furnace throughout the passage.
2022-11-11 19:25:31,333 INFO amazedShads5 52.1.91.250 Almost universally, a lone whale as a civilized dish, is his exceeding richness.
2022-11-11 19:25:33,342 INFO unhappyCrackers3 90.116.157.38 Then it darted a thousand feet straight up into the mizzen rigging, like three quarters of the stuff of the whale's head?
2022-11-11 19:25:35,354 INFO spiritedCheese5 224.255.241.75 Nor can any son of mortal woman, for the first time eye thee, no matter how many in the ship's run for salvation.
```
## Kibana
I won't remember where some of this stuff is...
- Home - http://localhost:5601
- Index Management - http://localhost:5601/app/management/data/index_management/indices
- Dev tools - http://localhost:5601/app/dev_tools#/console
- Grok debugger - http://localhost:5601/app/dev_tools#/grokdebugger
## References
- Elastic stack
- https://www.youtube.com/watch?v=Hqn5p67uev4
- https://medium.com/@sece.cosmin/docker-logs-with-elastic-stack-elk-filebeat-50e2b20a27c6
- https://www.bogotobogo.com/DevOps/Docker/Docker_ELK_7_6_Elastic_Stack_Docker_Compose.php
- https://github.com/deviantony/docker-elk
- https://www.youtube.com/playlist?list=PL_mJOmq4zsHZYAyK606y7wjQtC0aoE6Es
- https://www.javainuse.com/elasticsearch/filebeat-elk
- https://github.com/elastic/elasticsearch/blob/main/libs/grok/src/main/resources/patterns/legacy/grok-patterns
- Markov chains
- https://www.kdnuggets.com/2019/11/markov-chains-train-text-generation.html
- https://www.nltk.org/book/ch02.html
- https://towardsdatascience.com/text-generation-with-markov-chains-an-introduction-to-using-markovify-742e6680dc33
- https://www.gutenberg.org/files/2701/2701-h/2701-h.htm