{"id":20619743,"url":"https://github.com/twtrubiks/docker-elk-tutorial","last_synced_at":"2025-09-20T03:31:09.774Z","repository":{"id":41394833,"uuid":"119839899","full_name":"twtrubiks/docker-elk-tutorial","owner":"twtrubiks","description":"docker-elk-tutorial + django + logging","archived":false,"fork":false,"pushed_at":"2024-07-16T15:00:34.000Z","size":54,"stargazers_count":105,"open_issues_count":1,"forks_count":45,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-03T00:25:12.549Z","etag":null,"topics":["django","docker-elk","docker-logging","elasticsearch","kibana","logstash","python","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twtrubiks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-01T13:32:25.000Z","updated_at":"2025-01-21T05:43:17.000Z","dependencies_parsed_at":"2024-11-20T11:02:45.198Z","dependency_job_id":null,"html_url":"https://github.com/twtrubiks/docker-elk-tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/twtrubiks/docker-elk-tutorial","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2Fdocker-elk-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2Fdocker-elk-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2Fdocker-elk-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2Fdocker-elk-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twtrubiks","download_url":"https://codeload.github.com/twtrubiks/docker-elk-tutorial/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twtrubiks%2Fdocker-elk-tutorial/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276040561,"owners_count":25574825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-20T02:00:10.207Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","docker-elk","docker-logging","elasticsearch","kibana","logstash","python","tutorial"],"created_at":"2024-11-16T12:12:24.428Z","updated_at":"2025-09-20T03:31:09.500Z","avatar_url":"https://github.com/twtrubiks.png","language":"Python","readme":"# docker-elk-tutorial\n\ndocker-elk-tutorial📝\n\n* [Youtube Tutorial PART 1 - ELK 簡介](https://youtu.be/T_sLKn3vXa4)\n* [Youtube Tutorial PART 2 - docker ELK 環境建立](https://youtu.be/4JybtoFgC8g)\n* [Youtube Tutorial PART 3 - 透過 python 送 log 到 ELK](https://youtu.be/EpEJGLzIK6A)\n* [Youtube Tutorial PART 4 - logging for Django + ELK](https://youtu.be/_bkx0FfNRpQ)\n* [Youtube Tutorial PART 5 - docker logging + ELK](https://youtu.be/gTqAjea4Ncg)\n\n* 進階 [Youtube Tutorial - Linux 教學 - docker-elk-tutorial 7.6.0](https://youtu.be/iWFasUQ1tNQ) - [docker-elk-tutorial 7.6.0](https://github.com/twtrubiks/docker-elk-tutorial/tree/elk-7.6.0)\n\n## 簡介\n\n* [Youtube Tutorial PART 1 - ELK 簡介](https://youtu.be/T_sLKn3vXa4)\n\ndocker-elk :question: 這是什麼:question:  他可以吃嗎:confused:\n\n重點在 **ELK** ，他是由三個東西所組成的。\n\n[Elasticsearch](https://www.elastic.co/) ( E )\n\n![img](https://i.imgur.com/qSbJRSv.png)\n\n[Logstash](https://www.elastic.co/products/logstash) ( L )\n\n![img](https://i.imgur.com/7sQUVqy.png)\n\n[Kibana](https://www.elastic.co/products/kibana) ( K )\n\n![img](https://i.imgur.com/eajQh99.png)\n\n基本上，整個工作流程是這樣\n\n![img](https://i.imgur.com/ZTDCjnD.png)\n\n步驟一\n\nLogstash 蒐集從 docker or 其他地方的 log 資訊，這個步驟主要是因為我們可以透過 [logstash.conf](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/docker-elk/logstash/pipeline/logstash.conf) 過濾\n\n以及解析我們需要的資訊。\n\n步驟二\n\nLogstash 將處理完後的 log 資訊轉發到 Elasticsearch 進行 index。\n\n步驟三\n\n最後使用者可以透過 Kibana 分析以及視覺化所要的資料。\n\n以上就是整個工作的流程，那他有什麼用呢 :confused:\n\n像是分散式系統好了，之前介紹的 [docker swawm](https://github.com/twtrubiks/docker-swarm-tutorial)，每個容器的 log都進去一個一個看一定會累死，\n\n所以這時候就可以統一把 log 送到 docker-elk 中，方便統一管理以及分析。\n\n使用者的 log 非常重要，如果可以從 log 中分析出使用者愛好以及習慣，就可以推薦他類似的東西或\n\n進行改善，當然，有一點很重要，就是這些 log 必須 **處理** 過，你可能會和我說可以用 AI（ AI 正夯 :expressionless: ）\n\n但這不是這次的重點:relaxed:\n\n由於這篇文章我會採用 Docker 建立 docker-elk，所以建議對 Docker 要有一定的認識，如果你不了解\n\nDocker ，可參考我之前的 Docker 教學文章\n\n* [Docker 基本教學 - 從無到有 Docker-Beginners-Guide](https://github.com/twtrubiks/docker-tutorial)\n\n透過這篇文章，你將會學會\n\n* [docker ELK 環境建立](https://github.com/twtrubiks/docker-elk-tutorial#docker-elk-%E7%92%B0%E5%A2%83%E5%BB%BA%E7%AB%8B)\n\n* [透過 python 送 log 到 ELK](https://github.com/twtrubiks/docker-elk-tutorial#%E9%80%8F%E9%81%8E-python-%E9%80%81-log-%E5%88%B0-elk)\n\n* [logging for Django + ELK](https://github.com/twtrubiks/docker-elk-tutorial#logging-for-django--elk) - Django 如何設定 logging 以及發送 logging 到 ELK 中\n\n* [docker logging + ELK](https://github.com/twtrubiks/docker-elk-tutorial#docker-logging--elk) - 將 docker logs 發送到 docker ELK 中\n\n## docker ELK 環境建立\n\n* [Youtube Tutorial PART 2 - docker ELK 環境建立](https://youtu.be/4JybtoFgC8g)\n\n我們直接使用 [docker-elk](https://github.com/deviantony/docker-elk) 這邊的 docker-compose.yml 即可，但因為我擔心版本會\n\n更新（ 導致怪問題 ），所以我放一份到我自己的目錄下，建議閱讀一下 [docker-elk](https://github.com/deviantony/docker-elk)\n\n中的 README.md，先到 [docker-elk](https://github.com/twtrubiks/docker-elk-tutorial/tree/master/docker-elk) 目錄底下\n\n\u003e cd  docker-elk\n\n直接執行以下指令\n\n\u003e docker-compose up\n\n第一次會比較慢，因為要 pull image 而且還要初始化 :sleeping:\n\n這時候可以起來運動一下拉拉筋 :relaxed:\n\n也可以用 `docker ps` 確認 docker-elk 都有正常運行\n\n![img](https://i.imgur.com/OrprV0K.png)\n\n[docker-compose.yml](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/docker-elk/docker-compose.yml) 如果沒有特別修改，默認的 port 可參考下方\n\n```conf\n5000: Logstash TCP input\n9200: Elasticsearch HTTP\n9300: Elasticsearch TCP transport\n5601: Kibana\n```\n\n以上是預設的，這邊我多加上一個 UDP 的 port\n\n```conf\n12201: Logstash UDP input\n```\n\n那要如何加，首先，在 docker-elk/[docker-compose.yml](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/docker-elk/docker-compose.yml) 中加上 `12201:12201/udp`\n\n```yml\n  logstash:\n    build:\n      context: logstash/\n    volumes:\n      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro\n      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro\n    ports:\n      - \"5000:5000\"\n      - \"12201:12201/udp\"\n    environment:\n      LS_JAVA_OPTS: \"-Xmx256m -Xms256m\"\n    networks:\n      - elk\n    depends_on:\n      - elasticsearch\n```\n\n接著在 docker-elk/logstash/pipeline/[logstash.conf](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/docker-elk/logstash/pipeline/logstash.conf) 底下加上 udp\n\n```conf\ninput {\n    tcp {\n        port =\u003e 5000\n    }\n    udp {\n        port =\u003e 12201\n    }\n}\n\n## Add your filters / logstash plugins configuration here\n\noutput {\n    elasticsearch {\n        hosts =\u003e \"elasticsearch:9200\"\n    }\n}\n\n```\n\n[logstash.conf](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/docker-elk/logstash/pipeline/logstash.conf) 可以設定的真的非常多，像是 filters ，大家可以自行去摸索，這邊先不介紹:smirk:\n\n確認啟動成功後，我們可以先來看看 Elasticsearch，瀏覽\n[http://localhost:9200/](http://localhost:9200/)\n\n![img](https://i.imgur.com/4FcbOCm.png)\n\n接著再來看看 Kibana（ 有時候你會發現無法瀏覽，這是因為還在初始化 ）\n\n等待初始化完畢後，可以瀏覽 [http://localhost:5601/](http://localhost:5601/)，你應該會看到\n\n![img](https://i.imgur.com/f9aYPd3.png)\n\n我們需要先設定 index pattern，MAC 或 Linux 用戶直接使用以下指令\n\n```cmd\ncurl -XPOST -D- \"http://localhost:5601/api/saved_objects/index-pattern\" \\\n    -H \"Content-Type: application/json\" \\\n    -H \"kbn-version: 6.1.0\" \\\n    -d \"{'attributes':{'title':'logstash-*','timeFieldName':'@timestamp'}}\"\n```\n\n如果你是 Windows 用戶，請用其他方法，雖然 Windows 也有 curl，但我裝上去執行指令，\n\n他都會報錯說 josn格式錯誤，所以我直接改用 [Postman](https://www.getpostman.com/)\n\n![img](https://i.imgur.com/lHh7thR.png)\n\n如果一切順利，你應該會看到 response\n\n![img](https://i.imgur.com/ideT84S.png)\n\n接著重新整理 [http://localhost:5601/](http://localhost:5601/)，你應該會看到 index pattern 建立成功\n\n![img](https://i.imgur.com/qB55XQp.png)\n\n接著我們可以嘗試送送看 log , 如果你是 MAC 或 Linux 用戶，你可以使用以下指令\n\n```cmd\nnc localhost 5000 \u003c README.md\n```\n\n上面這段指令其實只是將 README.md 往 logstash ( [http://localhost:5000/](http://localhost:5000/) ) 送資料，\n\n可以透過 Kibana 觀看結果，會發現有一堆 [README.md](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/README.md) 的文字\n\n![img](https://i.imgur.com/eSdsdcZ.png)\n\n如果你是 Windows 用戶，請跳過這段 :laughing:\n\n直接用 python 來測試吧:smirk:\n\n## 透過 python 送 log 到 ELK\n\n* [Youtube Tutorial PART 3 - 透過 python 送 log 到 ELK](https://youtu.be/EpEJGLzIK6A)\n\n剛剛簡單的介紹 ELK，現在讓我們透過 python 送 log 到 ELK 吧 :satisfied:\n\n建議大家可以先了解一下 python 中的 [logging](https://docs.python.org/3.6/howto/logging.html)，\n\n也可以參考這個簡單的範例 [logging_tutorial.py](https://github.com/twtrubiks/python-notes/blob/master/logging_tutorial.py)。\n\n要使用 python 發送 log 到 ELK，請先執行下列指令\n\n[python-logstash](https://github.com/vklochan/python-logstash)\n\n\u003e pip install python-logstash\n\n接著執行以下程式碼 python-logging/[demo_logging.py](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/python-logging/demo_logging.py)\n\n```python\nimport logging\nimport logstash\nimport sys\n\nhost = 'localhost'\n\ntest_logger = logging.getLogger('python-logstash-logger')\ntest_logger.setLevel(logging.INFO)\n\n# UDP\n# test_logger.addHandler(logstash.LogstashHandler(host, 12201, version=1))\n\n# TCP\ntest_logger.addHandler(logstash.TCPLogstashHandler(host, 5000, version=1))\n\ntest_logger.error('python-logstash: test logstash error message.')\ntest_logger.info('python-logstash: test logstash info message.')\ntest_logger.warning('python-logstash: test logstash warning message.')\n\n# add extra field to logstash message\nextra = {\n    'test_string': 'python version: ' + repr(sys.version_info),\n    'test_boolean': True,\n    'test_dict': {'a': 1, 'b': 'c'},\n    'test_float': 1.23,\n    'test_integer': 123,\n    'test_list': [1, 2, '3'],\n}\ntest_logger.info('python-logstash: test extra fields', extra=extra)\nprint('done,please see kibana')\n```\n\n接著可以到 Kibana 觀看\n\n![img](https://i.imgur.com/mmPfRs6.png)\n\nlog 訊息的確是我們剛剛送出去的\n\n![img](https://i.imgur.com/Rm4bVgQ.png)\n\n如果你要測試 UDP 的部份，就把 TCP 註解，UDP 打開（ 取消註解 ），\n\n這樣以後我們就可以將我們需要記錄的 log 資料通通都送到 ELK 中管理 :thumbsup:\n\n## logging for Django + ELK\n\n* [Youtube Tutorial PART 4 - logging for Django + ELK](https://youtu.be/_bkx0FfNRpQ)\n\n剛剛介紹了如何透過 python 送 log 到 ELK 中，現在要教大家如何在 Django 中設定 logging :smirk:\n\n如果不了解什麼是 Django，可參考我之前寫的 [Django 基本教學 - 從無到有 Django-Beginners-Guide 📝](https://github.com/twtrubiks/django-tutorial)\n\n一樣請記得安裝 [python-logstash](https://github.com/vklochan/python-logstash) :blush:\n\n\u003e pip install python-logstash\n\n我們就依照 [這篇](https://github.com/twtrubiks/django-tutorial) 的範例繼續介紹，\n\n先將 django-tutorial/django-tutorial/[settings.py](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/django-tutorial/django-tutorial/settings.py) 加入下方程式碼\n\n```python\nLOGGING = {\n    'version': 1,\n    'disable_existing_loggers': False,\n    'formatters': {\n        'verbose': {\n            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'\n        },\n        'simple': {\n            'format': '%(levelname)s %(message)s'\n        },\n    },\n    'handlers': {\n        'console': {\n            'level': 'INFO',\n            'class': 'logging.StreamHandler',\n            'formatter': 'simple'\n        },\n        'logstash': {\n            'level': 'WARNING',\n            'class': 'logstash.TCPLogstashHandler',\n            'host': 'localhost',\n            'port': 5000,  # Default value: 5000\n            'version': 1,\n            'message_type': 'django_logstash',  # 'type' field in logstash message. Default value: 'logstash'.\n            'fqdn': False,  # Fully qualified domain name. Default value: false.\n            'tags': ['django.request'],  # list of tags. Default: None.\n        },\n    },\n    'loggers': {\n        'django': {\n            'handlers': ['console'],\n            'level': 'INFO',\n            'propagate': True,\n        },\n        'django.request': {\n            'handlers': ['logstash'],\n            'level': 'WARNING',\n            'propagate': True,\n        },\n    }\n}\n```\n\n詳細的 django logging 可參考官網 [https://docs.djangoproject.com/en/2.0/topics/logging/](https://docs.djangoproject.com/en/2.0/topics/logging/)，\n\n這邊要稍微提一下 [django.request](https://docs.djangoproject.com/en/2.0/topics/logging/#django-request)\n\n```txt\ndjango.request\nLog messages related to the handling of requests.\n5XX responses are raised as ERROR messages;\n4XX responses are raised as WARNING messages.\n```\n\n接著到 django-tutorial/musics/[views.py](https://github.com/twtrubiks/django-tutorial/blob/master/musics/views.py)中修改程式碼\n\n```python\nfrom django.shortcuts import render\n\nfrom musics.models import Music\nfrom django.http import Http404\n\n# Create your views here.\ndef hello_view(request):\n    musics = Music.objects.all()\n    # raise Exception('error !!!!')\n    # raise Http404(\"sorry 404\")\n    return render(request, 'hello_django.html', {\n        'data': \"Hello Django \",\n        'musics': musics,\n})\n```\n\n以上註解的兩個地方，可以自行玩玩看，然後到 Kibana 中觀看，\n\n如果不太理解，可參考影片的說明  [Youtube Tutorial PART 4 - logging for Django + ELK](https://youtu.be/_bkx0FfNRpQ)\n\n`raise Exception('error !!!!')` 這行等於是 5XX responses，也就是 ERROR messages，\n\n`raise Http404(\"sorry 404\")` 這行等於是 ˋXX responses，也就是 WARNING messages。\n\n## docker logging + ELK\n\n* [Youtube Tutorial PART 5 - docker logging + ELK](https://youtu.be/gTqAjea4Ncg)\n\n既然都講到這裡了，一定要來說一下如何將 docker 的 log 送到 ELK 中，\n\n先來個 tcp 的簡單範例\n\n```cmd\ndocker run --log-driver=syslog --log-opt syslog-address=tcp://0.0.0.0:5000 --log-opt syslog-facility=daemon alpine echo hello world tcp\n```\n\n![img](https://i.imgur.com/CNlAb98.png)\n\n到 Kibana 觀看\n\n![img](https://i.imgur.com/tWSFH1B.png)\n\n再來個 udp 的簡單範例\n\n```cmd\ndocker run --log-driver=gelf --log-opt gelf-address=udp://0.0.0.0:12201 alpine echo hello world udp\n```\n\n![img](https://i.imgur.com/DZpe15V.png)\n\n這邊我覺得奇怪的是，如果用 gelf 送出去的 log 都會變成亂碼，\n\n如果有人知道原因再請解答:sweat_smile:\n\n![img](https://i.imgur.com/TobRFjw.png)\n\ndocker logging 詳細可參考 [https://docs.docker.com/engine/admin/logging/overview/](https://docs.docker.com/engine/admin/logging/overview/)\n\n那如果我希望寫在 docker-compose 中呢？\n\n請看 docker-logging/[docker-compose.yml](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/docker-logging/docker-compose.yml)\n\n```python\nversion: '3.3'\nservices:\n\n    db:\n      # container_name: 'postgres'\n      image: postgres\n      environment:\n        POSTGRES_PASSWORD: password123\n      ports:\n        - \"5432:5432\"\n        # (HOST:CONTAINER)\n      volumes:\n        - pgdata:/var/lib/postgresql/data/\n\n    web:\n      # build: ./api\n      # command: python manage.py runserver 0.0.0.0:8000\n      image: twtrubiks/my_django\n      restart: always\n      volumes:\n        - api_data:/docker_api\n        # (HOST:CONTAINER)\n      ports:\n        - \"8000:8000\"\n        # (HOST:CONTAINER)\n      depends_on:\n        - db\n\n      logging:\n        driver: syslog\n        options:\n            syslog-address: tcp://0.0.0.0:5000\n            tag: web-container-tcp\n\n      # logging:\n      #   driver: gelf\n      #   options:\n      #     gelf-address: udp://0.0.0.0:12201\n      #     tag: web-container-udp\n\nvolumes:\n    api_data:\n    pgdata:\n```\n\n以上這個範例是從 [Docker 基本教學 - 從無到有 Docker-Beginners-Guide](https://github.com/twtrubiks/docker-tutorial)修改過來的，\n\n一樣執行 `docker-compose up`，\n\n![img](https://i.imgur.com/FjiW7K9.png)\n\n接著到 Kibana 中可以看到 log 資訊\n\n![img](https://i.imgur.com/Y0F8BSD.png)\n\n## 後記：\n\n這篇文章主要是帶大家對 ELK 有一些基本的觀念，因為 ELK 可以玩的東西真的非常的多，\n\n坑很大，像是前面所說的 [logstash.conf](https://github.com/twtrubiks/docker-elk-tutorial/blob/master/docker-elk/logstash/pipeline/logstash.conf) 中可以設定的參數，像是 filters 之類的.......\n\n又或是 Kibana 如何呈現精美的圖表，甚至將 docker-elk 佈署到 Swarm 中，都可以玩玩\n\n看，所以大家有興趣可以再自行深入研究:smile:\n\n我本來是想要透過 Django 結合 Haystack 做個全文檢索的範例，但因為 Haystack 對於\n\nElasticSearch 的版本只支援到 2.X  ( ElasticSearch 都出到 6.X 了 )，最後就沒有將這範例\n\n寫出來了:sweat_smile:\n\n[elasticsearch-py](https://github.com/elastic/elasticsearch-py) 這個 library 也可以看看，我用 6.x 版本測試，還是有一點問題，問題\n\n如果解決再分享給各位:laughing:\n\n## 執行環境\n\n* Python 3.6.2\n\n## Reference\n\n* [docker-elk](https://github.com/deviantony/docker-elk)\n\n* [python-logstash](https://github.com/vklochan/python-logstash)\n\n* [Django](https://www.djangoproject.com/)\n\n## Donation\n\n文章都是我自己研究內化後原創，如果有幫助到您，也想鼓勵我的話，歡迎請我喝一杯咖啡:laughing:\n\n![alt tag](https://i.imgur.com/LRct9xa.png)\n\n[贊助者付款](https://payment.opay.tw/Broadcaster/Donate/9E47FDEF85ABE383A0F5FC6A218606F8)\n\n## License\n\nMIT license\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwtrubiks%2Fdocker-elk-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwtrubiks%2Fdocker-elk-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwtrubiks%2Fdocker-elk-tutorial/lists"}