{"id":19052953,"url":"https://github.com/kayvansol/elasticsearch","last_synced_at":"2025-09-07T06:33:11.687Z","repository":{"id":242122603,"uuid":"805573265","full_name":"kayvansol/elasticsearch","owner":"kayvansol","description":"Deploy elasticsearch, kibana \u0026 logstash (ELK Stack) with docker compose","archived":false,"fork":false,"pushed_at":"2024-06-15T10:32:14.000Z","size":849,"stargazers_count":5,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T20:12:38.047Z","etag":null,"topics":["docker-compose","dockerfile","elasticsearch","elk-stack","kibana","logstash","python","xpack"],"latest_commit_sha":null,"homepage":"https://medium.com/@kayvan.sol2/deploy-elasticsearch-kibana-logstash-with-docker-compose-df518d68731d","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kayvansol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-24T22:06:05.000Z","updated_at":"2025-02-20T07:50:14.000Z","dependencies_parsed_at":"2024-06-15T11:42:14.874Z","dependency_job_id":"58f3118c-0624-4488-aba7-0faec791d01c","html_url":"https://github.com/kayvansol/elasticsearch","commit_stats":null,"previous_names":["kayvansol/elasticsearch"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kayvansol/elasticsearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Felasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Felasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Felasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Felasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kayvansol","download_url":"https://codeload.github.com/kayvansol/elasticsearch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Felasticsearch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274005343,"owners_count":25205934,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-07T02:00:09.463Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker-compose","dockerfile","elasticsearch","elk-stack","kibana","logstash","python","xpack"],"created_at":"2024-11-08T23:28:33.597Z","updated_at":"2025-09-07T06:33:11.163Z","avatar_url":"https://github.com/kayvansol.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deploy Elasticsearch, Kibana \u0026amp; Logstash (ELK Stack) with Docker Compose\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/0.png?raw=true)\n\n1) Pull the all related docker **images** :\n```bash\ndocker pull docker.elastic.co/elasticsearch/elasticsearch:8.13.4\ndocker pull docker.elastic.co/kibana/kibana:8.13.4\ndocker pull docker.elastic.co/logstash/logstash:8.13.4\n```\n\n2) Write the **Docker Compose** file :\n```yaml\nversion: '3.7'\n\nservices:\n\n  # The 'setup' service runs a one-off script which initializes users inside\n  # Elasticsearch — such as 'logstash_internal' and 'kibana_system' — with the\n  # values of the passwords defined in the '.env' file. It also creates the\n  # roles required by some of these users.\n  #\n  # This task only needs to be performed once, during the *initial* startup of\n  # the stack. Any subsequent run will reset the passwords of existing users to\n  # the values defined inside the '.env' file, and the built-in roles to their\n  # default permissions.\n  #\n  # By default, it is excluded from the services started by 'docker compose up'\n  # due to the non-default profile it belongs to. To run it, either provide the\n  # '--profile=setup' CLI flag to Compose commands, or \"up\" the service by name\n  # such as 'docker compose up setup'.\n\n  setup:\n    profiles:\n      - setup\n    build:\n      context: setup/\n      args:\n        ELASTIC_VERSION: ${ELASTIC_VERSION}\n    init: true\n    volumes:\n      - ./setup/entrypoint.sh:/entrypoint.sh:ro,Z\n      - ./setup/lib.sh:/lib.sh:ro,Z\n      - ./setup/roles:/roles:ro,Z\n    environment:\n      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}\n      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}\n      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}\n      METRICBEAT_INTERNAL_PASSWORD: ${METRICBEAT_INTERNAL_PASSWORD:-}\n      FILEBEAT_INTERNAL_PASSWORD: ${FILEBEAT_INTERNAL_PASSWORD:-}\n      HEARTBEAT_INTERNAL_PASSWORD: ${HEARTBEAT_INTERNAL_PASSWORD:-}\n      MONITORING_INTERNAL_PASSWORD: ${MONITORING_INTERNAL_PASSWORD:-}\n      BEATS_SYSTEM_PASSWORD: ${BEATS_SYSTEM_PASSWORD:-}\n    networks:\n      - elk\n    depends_on:\n      - elasticsearch\n\n  elasticsearch:\n    build:\n      context: elasticsearch/\n      args:\n        ELASTIC_VERSION: ${ELASTIC_VERSION}\n    volumes:\n      #- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,Z\n      - elasticsearch:/usr/share/elasticsearch/data:Z\n    ports:\n      - 9200:9200\n      - 9300:9300\n    environment:\n      node.name: elasticsearch\n      ES_JAVA_OPTS: -Xms512m -Xmx512m\n      # Bootstrap password.\n      # Used to initialize the keystore during the initial startup of\n      # Elasticsearch. Ignored on subsequent runs.\n      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}\n      # Use single node discovery in order to disable production mode and avoid bootstrap checks.\n      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html\n      discovery.type: single-node\n    networks:\n      - elk\n    restart: unless-stopped\n\n  logstash:\n    build:\n      context: logstash/\n      args:\n        ELASTIC_VERSION: ${ELASTIC_VERSION}\n    volumes:\n      #- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z\n      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z\n    ports:\n      - 5044:5044\n      - 50000:50000/tcp\n      - 50000:50000/udp\n      - 9600:9600\n    environment:\n      LS_JAVA_OPTS: -Xms256m -Xmx256m\n      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}\n    networks:\n      - elk\n    depends_on:\n      - elasticsearch\n    restart: unless-stopped\n\n  kibana:\n    build:\n      context: kibana/\n      args:\n        ELASTIC_VERSION: ${ELASTIC_VERSION}\n    #volumes:\n    #  - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z\n    ports:\n      - 5601:5601\n    environment:\n      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}\n    networks:\n      - elk\n    depends_on:\n      - elasticsearch\n    restart: unless-stopped\n\nnetworks:\n  elk:\n    driver: bridge\n\nvolumes:\n  elasticsearch:\n\n```\n\n3) Start your deployment without 'Setup' service (**manual** deployment) :\n```bash\ndocker compose up\n```\n\n4) Test the **Elastic** node via browser :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/1.png?raw=true)\n\n5) Add elasticsearch DNS A record within etc/hosts :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/2.png?raw=true)\n\n6) On the elasticsearch node, write the below command :\n```bash\nbin/elasticsearch-reset-password -u kibana_system --auto\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/3.png?raw=true)\n\n7) And copy the New Value result to the **kibana** node and **kibana.yml** :\n```\nelasticsearch.username: \"kibana_system\"\nelasticsearch.password: \"q7pgSbw_JYm9ehk-=kc+\"\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/4.png?raw=true)\n\n8) And go to the kibana portal :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/5.png?raw=true)\n\n9) Get the elastic node healthy info from kibana :\n```\nGET /_cluster/health?pretty\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/6.png?raw=true)\n\n\n10) On the elasticsearch node, write the below command :\n```bash\nbin/elasticsearch-reset-password -u logstash_system --auto\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/7.png?raw=true)\n\n11) Copy the New Value result to the **logstash** node and **logstash.yml** :\n```\nxpack.monitoring.elasticsearch.username: \"logstash_system\"\nxpack.monitoring.elasticsearch.password: \"nMv9pFyreKMIkzQh-GEy\"\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/8.png?raw=true)\n\n12) Then change the elastic node auth info in the **logstash.conf** :\n```conf\noutput {\n\telasticsearch {\n\t\thosts =\u003e \"elasticsearch:9200\"\n\t\t#user =\u003e \"logstash_internal\"\n\t\t#password =\u003e \"${LOGSTASH_INTERNAL_PASSWORD}\"\n\t\tuser =\u003e \"elastic\"\n\t\tpassword =\u003e \"changeme\"\n\t}\n}\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/9.png?raw=true)\n\n13) Check the logstash log :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/10.png?raw=true)\n\n14) And logstash info :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/11.png?raw=true)\n\n15) And the docker desktop shows that stack is healthy :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/12.png?raw=true)\n\n16) Create your index named **testindex** in elasticsearch via kibana :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/13.png?raw=true)\n\n17) Add below lines at **logstash.conf** :\n```conf\noutput {\n\telasticsearch {\n\t\thosts =\u003e \"elasticsearch:9200\"\n\t\t#user =\u003e \"logstash_internal\"\n\t\t#password =\u003e \"${LOGSTASH_INTERNAL_PASSWORD}\"\n\t\tuser =\u003e \"elastic\"\n\t\tpassword =\u003e \"changeme\"\n        \tindex =\u003e \"testindex\"\n\t}\n}\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/14.png?raw=true)\n\n18) Write some **python** scripts or **C#** code for sending data to the related port (50000) :\n```python\nimport socket\nimport sys\n\nHOST = '192.168.1.4'\nPORT = 50000\n\ntry:\n  sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\nexcept socket.error as msg:\n  sys.stderr.write(\"[ERROR] %s\\n\" % msg[1])\n  sys.exit(1)\n\ntry:\n  sock.connect((HOST, PORT))\nexcept socket.error as msg:\n  sys.stderr.write(\"[ERROR] %s\\n\" % msg[1])\n  sys.exit(2)\n\nmsg = '{\"id\":\"1\",\"name\":\"kayvan soleimani\",\"email\":\"kayvan.sol2@gmail.com}'\n\nsock.send(msg.encode('utf-8') )\n\nsock.close()\nsys.exit(0)\n```\n\nC# Code :\n```c#\nusing System;\nusing System.Net.Sockets;\nusing System.Text;\n\nstring HOST = \"192.168.1.4\";\nint PORT = 50000;\n\ntry\n{\n    using (Socket sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))\n    {\n        sock.Connect(HOST, PORT);\n\n        string msg = \"{\\\"id\\\":\\\"2\\\",\\\"name\\\":\\\"Sorayya Asadi\\\",\\\"email\\\":\\\"sorayyaasadi6@gmail.com\\\"}\";\n\n        byte[] data = Encoding.UTF8.GetBytes(msg);\n\n        sock.Send(data);\n\n        Console.WriteLine(\"Data sent to logstash ...\");\n\n    }\n}\ncatch (SocketException ex)\n{\n    Console.Error.WriteLine($\"[ERROR] {ex.Message}\");\n    Environment.Exit(1);\n}\ncatch (Exception ex)\n{\n    Console.Error.WriteLine($\"[ERROR] {ex.Message}\");\n    Environment.Exit(2);\n}\n\nConsole.ReadLine();\n```\n\nThen run the python script :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/15.png?raw=true)\n\n16) After some seconds, data appear at your elastic index :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/16.png?raw=true)\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/17.png?raw=true)\n\n17) You can search index records via Dev Tools at kibana :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/elasticsearch/main/img/18.png?raw=true)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayvansol%2Felasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkayvansol%2Felasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayvansol%2Felasticsearch/lists"}