https://github.com/ei-grad/nginx2es

Put parsed access.log to Elasticsearch (the right way)
https://github.com/ei-grad/nginx2es

elasticsearch logging nginx

Last synced: 15 days ago
JSON representation

Put parsed access.log to Elasticsearch (the right way)

• Host: GitHub
• URL: https://github.com/ei-grad/nginx2es
• Owner: ei-grad
• Created: 2017-09-18T12:11:58.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2021-01-19T06:46:40.000Z (almost 4 years ago)
• Last Synced: 2024-12-06T13:20:53.546Z (28 days ago)
• Topics: elasticsearch, logging, nginx
• Language: Python
• Homepage: https://github.com/ei-grad/nginx2es
• Size: 63.5 KB
• Stars: 3
• Watchers: 2
• Forks: 3
• Open Issues: 2
• Metadata Files:
  • Readme: README.rst

Awesome Lists containing this project

README

        
Put parsed Nginx access.log to Elasticsearch

============================================

Nginx access.log have to be formatted with this format:

.. code-block:: nginx

    log_format main_ext

        '$remote_addr $http_host $remote_user [$time_local] "$request" '

        '$status $body_bytes_sent "$http_referer" '

        '"$http_user_agent" "$http_x_forwarded_for" '

        'rt=$request_time ua="$upstream_addr" '

        'us="$upstream_status" ut="$upstream_response_time" '

        'ul="$upstream_response_length" '

        'cs=$upstream_cache_status';

Install

-------

Install with pip:

.. code-block:: bash

    pip install nginx2es

Features

--------

- Stable log record ID (hostname + file inode number + timestamp + file

  position). It makes possible to import log file more than once (adding some

  additional processing to ``nginx2es``, or dropping a daily index containing

  only a half of records, etc) without creating a duplicate records.

- Parse query params and split request uri path components to separate fields

  for complex log filtering / aggregations.

- Optional use of the GeoIP database (requires the ``geoip`` module and the

  ``GeoIPCity.dat`` database file) - adds ``city`` and ``region_name`` fields.

- Correctly parse log records containing information about multiple upstream

  responses.

- The ``tail -F``-like mode implemented with inotify.