{"id":20329761,"url":"https://github.com/f5devcentral/nap-dos-elk-dashboards","last_synced_at":"2025-10-03T13:24:58.989Z","repository":{"id":40424545,"uuid":"373009709","full_name":"f5devcentral/nap-dos-elk-dashboards","owner":"f5devcentral","description":"NGINX App Protect Denial of Service dashboard for the ELK stack","archived":false,"fork":false,"pushed_at":"2024-05-29T15:12:19.000Z","size":983,"stargazers_count":7,"open_issues_count":1,"forks_count":7,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-01-14T15:18:25.470Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/f5devcentral.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-02T01:54:22.000Z","updated_at":"2024-12-18T03:41:07.000Z","dependencies_parsed_at":"2024-05-29T17:44:11.716Z","dependency_job_id":"a3a355e3-0731-49a3-8b9f-ed38cbeebe40","html_url":"https://github.com/f5devcentral/nap-dos-elk-dashboards","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5devcentral%2Fnap-dos-elk-dashboards","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5devcentral%2Fnap-dos-elk-dashboards/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5devcentral%2Fnap-dos-elk-dashboards/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5devcentral%2Fnap-dos-elk-dashboards/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/f5devcentral","download_url":"https://codeload.github.com/f5devcentral/nap-dos-elk-dashboards/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241844565,"owners_count":20029679,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T20:13:22.270Z","updated_at":"2025-10-03T13:24:58.911Z","avatar_url":"https://github.com/f5devcentral.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# NGINX App Protect DoS ELK Dashboards\nA community supported repo for NGINX App Protect Denial of Service dashboards on the ELK stack.\n\n\u003cimg src=\"images/dashboard1.png\" width=\"800px\"/\u003e\n\n## How does it work?\nELK stands for Elasticsearch, Logstash, and Kibana. Logstash receives logs from NGINX App Protect DoS, normalizes them and stores them in the Elasticsearch index. Kibana allows you to visualize and navigate through logs using purpose built dashboards.\n\n## Requirements\n- The installation instructions assume you are using a bash or zsh shell. The [Docker](https://docker.com), [docker-compose](https://docs.docker.com/compose/) and [jq](https://stedolan.github.io/jq/) packages are also assumed to be installed.\n\n- The provided Kibana dashboards require a minimum version of 8.11.1. If you are using the provided [docker-compose.yaml](docker-compose.yaml) file, this version requirement is met.\n\n- In `docker-compose.yaml`, the subnet configuration is added in order to override the ip assignment from the Docker default subnet, i.e 172.18.x.x/16. \n```\n...\nnetworks:\n  default:\n    ipam:\n      driver: default\n      config:\n        - subnet: 172.33.0.0/16\n```\n- In case there is an error creating the docker network, restart docker: `systemctl restart docker`.\n\n- The ELK stack docker container will likely exceed the default host's virtual memory system limits. Use [these directions](https://www.elastic.co/guide/en/elasticsearch/reference/5.0/vm-max-map-count.html#vm-max-map-count) to increase this limit on the docker host machine. If you do not, the ELK container will continually restart itself and never fully initialize.\n\n## Installation Overview\nIt is assumed you will be running ELK using the Quick Start directions below. The template in `logstash/conf.d` will create a new Logstash pipeline to ingest logs and store them in Elasticsearch. If you use the supplied `docker-compose.yaml`, this template will be copied into the docker container instance for you. Once the DoS logs are being ingested into the Elasticsearch index, you will need to import files from the [kibana](kibana/) folder to create all necessary objects including the index pattern, visualization and dashboards.\n\n\n### Deploying ELK Stack\n1. Use docker-compose to deploy your own ELK stack.\n\n```shell\ndocker-compose -f docker-compose.yaml up -d\n```\n\n2. Verify the installation by logging into Kibana via a browser at `http://\u003c your container host\u003e:5601/`\n\n**NOTE:**\n- It assumed that your current working directory is `nap-dos-elk-dashboards`.\n- The `logstash` folder will be created in the working directory.\n- The `logstash/conf.d` folder is mapped to `/etc/logstash/conf.d` in the ELK container.\n\n3. Open a *new* terminal window and ssh into the Elasticsearch container:\n\n```shell\ndocker exec -it nap-dos-elk-dashboards_elasticsearch_1 /bin/bash\n```\n\n4. From inside the container, stop the `logstash` process:\n\n```shell\nservice logstash stop\n```\n\n5. Install Logstash plugins :\n\n```shell\n/opt/logstash/bin/logstash-plugin install logstash-output-syslog\n/opt/logstash/bin/logstash-plugin install logstash-input-syslog\n/opt/logstash/bin/logstash-plugin install logstash-input-tcp\n/opt/logstash/bin/logstash-plugin install logstash-input-udp\n```\n\n6. Print the contents of the Logstash configuration file to ensure it exists:\n\n```shell\ncat /etc/logstash/conf.d/apdos-logstash.conf\n```\n\n7. In your *original* terminal window (outside the container), create the Elasticsearch index with the following cURL command:\n\n```shell\ncurl -XPUT \"http://localhost:9200/app-protect-dos-logs\"  -H \"Content-Type: application/json\" -d  @apdos_mapping.json\n```\n\n**NOTE:**\nIn case there is error in this step, it may indicate that the `app-protect-dos-logs` index already exists from a previous Kibana installation, or has been created automatically by Logstash processing incoming App Protect DoS messages. If so, you will need to delete the index with the following cURL command:\n\n```shell\ncurl -XDELETE http://localhost:9200/app-protect-dos-logs\n```\n\nTo verify that `app-protect-dos-logs` index has been deleted:\n\n```shell\ncurl -XGET \"http://localhost:9200/_cat/indices\"\n```\n\nThe result should not include an `app-protect-dos-logs` index. Then, re-attempt to create the index using the instructions in the step above.\n\n8. Update mapping with geo fields:\n\n```shell\ncurl -XPOST \"http://localhost:9200/app-protect-dos-logs/_mapping\"  -H \"Content-Type: application/json\" -d  @apdos_geo_mapping.json\n```\n\n9. Import dashboards to Kibana through the UI (Kibana -\u003e Management -\u003e Saved Objects) or alternatively, use API call below:\n\n```shell\n\nKIBANA_CONTAINER_URL=http://localhost:5601\n\njq -s . kibana/apdos-dashboard.ndjson | jq '{\"objects\": . }' | \\\ncurl -k --location --request POST \"$KIBANA_CONTAINER_URL/api/kibana/dashboards/import\" \\\n    --header 'kbn-xsrf: true' \\\n    --header 'Content-Type: text/plain' -d @- \\\n    | jq\n\n```\n\n10. From your terminal window *inside* the container, start Logstash:\n\n```shell\nservice logstash start\n```\n\n### NGINX App Protect DoS Configuration\nNGINX App Protect DoS configuration directives as should appear in your `nginx.conf`. You will need to replace `ip_kibana` in the snippet below with the hostname of the server hosting your ELK Docker container:\n\n```\nhttp {\n    log_format log_dos ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ip_tls=$remote_addr:$app_protect_dos_tls_fp, ';\n    ...\n    server {\n       ...\n\n       app_protect_dos_security_log_enable on;\n       app_protect_dos_security_log \"/etc/app_protect_dos/log-default.json\" syslog:server=ip_kibana:5261;\n\n\n       location / {\n           app_protect_dos_enable       on;\n           set $loggable '0';\n           access_log syslog:server=ip_kibana:5561 log_dos if=$loggable;\n\n        ...   \n       }\n       \n    }\n    ...\n}\n```\n\n**NOTE:**\nThe Logstash listener in this solution is configured to listen for TCP syslog messages on port `5261` and UDP port `5561`. Kibana and Elasticsearch are using TCP ports `9200`, `5601` respectively.\nThese ports must be opened on your firewall for TCP/UDP accordingly.\n\n\n## Distinguishing services in the case of multiple protected objects (vss)\n\nUsing the dashboard filter:\nAs an example, for protected object name \"example.com\", the filter should be as follows:\n`vs_name_al : \"example.com/\"  or vs_name : \"example.com/\"`\n\n**NOTE:**\n`vs_name_al` and `vs_name` must be both at dashboard filter level, and be connected with: \"or\".\n\n## Installing Both App Protect WAF and App Protect DoS Dashboards\n\nBoth the [App Protect WAF](https://github.com/f5devcentral/f5-waf-elk-dashboards) and DoS Dashboards (this repo) can work in parallel on the same ELK instance. However some preparation is necessary to ensure they will run successfully in parallel with each other.\n\n1. Clone the [f5-waf-elk-dashboards](https://github.com/f5devcentral/f5-waf-elk-dashboards) and App Protect DoS Dashboards (this repo) using the following commands:\n\n```shell\ngit clone https://github.com/f5devcentral/f5-waf-elk-dashboards.git\ngit clone https://github.com/f5devcentral/nap-dos-elk-dashboards.git\n```\n\n2. Copy the `apdos-logstash.conf` file to the `f5-waf-elk-dashboards` working directory:\n```shell\ncp nap-dos-elk-dashboards/logstash/conf.d/apdos-logstash.conf f5-waf-elk-dashboards/logstash/conf.d/\n```\n\n3. Create a `f5-waf-elk-dashboards/logstash/pipelines.yml` file with the following contents:\n```yaml\n- pipeline.id: napwaf\n  path.config: \"/opt/logstash/config/30-waf-logs-full-logstash.conf\"\n\n- pipeline.id: napdos\n  path.config: \"/opt/logstash/config/apdos-logstash.conf\"\n```\n\n4. The `f5-waf-elk-dashboards/docker-compose.yaml` needs to be modified to accommodate 3 necessary changes: the version of the ELK container image used, additional syslog ports are required, and Logstash's [Multiple Pipelines](https://www.elastic.co/blog/logstash-multiple-pipelines) feature needs to be used in order to isolate the message flows. A properly modified `docker-compose.yaml` will look like this:\n\n```yaml\nversion: \"2.4\"\nservices:\n  elasticsearch:\n    image: sebp/elk:793\n    restart: always\n    volumes:\n      - ./logstash/pipelines.yml:/opt/logstash/config/pipelines.yml:ro\n      - ./logstash/conf.d/30-waf-logs-full-logstash.conf:/opt/logstash/config/30-waf-logs-full-logstash.conf:ro\n      - ./logstash/conf.d/apdos-logstash.conf:/opt/logstash/config/apdos-logstash.conf:ro\n      - elk:/var/lib/elasticsearch\n    ports:\n      - 9200:9200/tcp\n      - 5601:5601/tcp\n      - 5144:5144/tcp\n      - 5261:5261/tcp\n      - 5561:5561/udp\nvolumes:\n  elk:\n\n```\n\n5. Change to the `f5-waf-elk-dashboards` directory:\n``` shell\ncd f5-waf-elk-dashboards\n```\n\n6. Follow the [Deploying ELK Stack](https://github.com/f5devcentral/f5-waf-elk-dashboards#deploying-elk-stack) instructions in the f5-waf-elk-dashboards README document, then advance to step 7 in this guide when complete.\n\n7. Change to the `nap-dos-elk-dashboards` directory:\n``` shell\ncd ../nap-dos-elk-dashboards\n```\n\n8. Open a *new* terminal window and ssh into the Elasticsearch container:\n\n```shell\ndocker exec -it f5-waf-elk-dashboards_elasticsearch_1 /bin/bash\n```\n\n9. Follow the [Deploying ELK Stack](#deploying-elk-stack) instructions starting at step 4.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ff5devcentral%2Fnap-dos-elk-dashboards","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ff5devcentral%2Fnap-dos-elk-dashboards","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ff5devcentral%2Fnap-dos-elk-dashboards/lists"}