{"id":13891529,"url":"https://github.com/facundoolano/ngtop","last_synced_at":"2025-04-09T05:11:22.551Z","repository":{"id":249885246,"uuid":"831019608","full_name":"facundoolano/ngtop","owner":"facundoolano","description":"nginx access logs analytics","archived":false,"fork":false,"pushed_at":"2024-09-14T19:22:31.000Z","size":102,"stargazers_count":573,"open_issues_count":6,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-02T04:04:17.291Z","etag":null,"topics":["analytics","cli","metrics","nginx","self-hosted"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facundoolano.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-19T13:26:42.000Z","updated_at":"2025-03-31T12:37:55.000Z","dependencies_parsed_at":"2024-09-28T01:50:30.490Z","dependency_job_id":null,"html_url":"https://github.com/facundoolano/ngtop","commit_stats":{"total_commits":103,"total_committers":1,"mean_commits":103.0,"dds":0.0,"last_synced_commit":"2f529bee69aa3fc257a0e5fea34339bc57c65427"},"previous_names":["facundoolano/ngtop"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facundoolano%2Fngtop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facundoolano%2Fngtop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facundoolano%2Fngtop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facundoolano%2Fngtop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facundoolano","download_url":"https://codeload.github.com/facundoolano/ngtop/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247980837,"owners_count":21027808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","cli","metrics","nginx","self-hosted"],"created_at":"2024-08-06T17:00:24.673Z","updated_at":"2025-04-09T05:11:22.534Z","avatar_url":"https://github.com/facundoolano.png","language":"Go","readme":"# ngtop\n\nngtop is a command-line program to query request counts from nginx's access.log files.\n\n```\n$ ngtop url user_agent --since 1d --where url=/blog/% --where status=200 --limit 5\nPATH                                             USER_AGENT     #REQS\n/blog/deconstructing-the-role-playing-videogame/ Safari         120\n/blog/on-ai-assistance/                          Go-http-client 101\n/blog/a-note-on-essential-complexity/            Safari         91\n/blog/deconstructing-the-role-playing-videogame/ Chrome         84\n/blog/from-rss-to-my-kindle/                     Safari         79\n```\n\n## Installation\n\nDownload the [latest release binary](https://github.com/facundoolano/ngtop/releases/latest) for your platform, for example:\n\n    $ wget https://github.com/facundoolano/ngtop/releases/latest/download/ngtop-linux-arm64  \\\n        -O ngtop \u0026\u0026 chmod +x ngtop \u0026\u0026 mv ngtop /usr/local/bin\n\nAlternatively, install with go:\n\n    $ go install github.com/facundoolano/ngtop@latest\n\n## Usage examples\n\nCount requests from the last hour:\n\n    $ ngtop --since 1h\n    $ ngtop -s 1h\n    $ ngtop\n\nCount requests from the last second, minute, day, week, or month:\n\n    $ ngtop -s 1s\n    $ ngtop -s 1m\n    $ ngtop -s 1d\n    $ ngtop -s 1w\n    $ ngtop -s 1M\n\nCount requests from the day before:\n\n    $ ngtop --since 2d --until 1d\n    $ ngtop -s 2d -u 1d\n\nShow the top 5 urls in the last hour:\n\n    $ ngtop url\n    $ ngtop path\n\nShow the top 5 urls in the last minute:\n\n    $ ngtop url -s 1m\n\nShow the top 10 urls in the last hour:\n\n    $ ngtop url --limit 10\n    $ ngtop url -l 10\n\nCount total requests to a specific url in the last hour:\n\n\t$ ngtop --where url=/blog/code-is-run-more-than-read\n\t$ ngtop --where path=/blog/code-is-run-more-than-read\n\t$ ngtop -w url=/blog/code-is-run-more-than-read\n\nCount total requests to urls matching a pattern:\n\n\t$ ngtop -w url=/blog/%\n\nCount total requests to urls excluding a value or pattern:\n\n\t$ ngtop -w url!=/feed.xml\n\t$ ngtop -w url!=/feed%\n\nCount total requests to one of mutliple urls (one OR another):\n\n\t$ ngtop -w url=/blog/code-is-run-more-than-read -w url=/blog/a-note-on-essential-complexity\n\nCount total requests to a specific urls AND referer:\n\n\t$ ngtop -w url=/blog/code-is-run-more-than-read -w referer=news.ycombinator.com\n\nShow the top visited urls matching a pattern:\n\n\t$ ngtop url -w url=/blog/%\n\nShow the top requesting ips:\n\n    $ ngtop ip\n\nShow the top url visits by ip:\n\n    $ ngtop url -w ip=77.16.76.86\n\nShow the top user agents by url:\n\n    $ ngtop user_agent -w url=/blog/code-is-run-more-than-read\n    $ ngtop ua -w url=/blog/code-is-run-more-than-read\n\nShow the top urls by user agent details (parsed with [mileusna/useagent](https://pkg.go.dev/github.com/mileusna/useragent)):\n\n    $ ngtop url -w ua=Firefox\n    $ ngtop url -w ua_type=bot\n    $ ngtop url -w device=iPhone\n    $ ngtop url -w os=Linux\n\nShow the top referers for a url pattern:\n\n    $ ngtop referer -w url=/blog/%\n\nShow the top user agent and referer combination\n\n    $ ngtop ua referer\n\nShow the top user agent and referer combination for a specific url\n\n    $ ngtop ua referer -w url=/blog/code-is-run-more-than-read\n\nCount total 404 status responses:\n\n    $ ngtop -w status=404\n\nCount non successful responses:\n\n    $ ngtop status -w status=4% -w status=5%\n\n## How it works\n\n- Whenever the program is run, it looks for the nginx access.logs, parses them and stores the data into an SQLite DB.\n  - By default, the logs are looked up at `/var/log/nginx/access.log*`, which can be overridden with the `NGTOP_LOGS_PATH` environment variable.\n  - By default, the logs are assumed to have the [nginx combined log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format). The format can be customized with `NGTOP_LOG_FORMAT`.\n    - This could likely be made to work with non nginx logs, although that hasn't been tested.\n  - Subsequent runs of the program only parse and store the logs up until the time of the previous run.\n  - The SQLite DB is stored at `./ngtop.db`, which can be overridden with the `NGTOP_DB` environment variable.\n- The command line arguments express a filtering criteria, used to build the SQL query that counts the requests.\n  - For instance, the command `ngtop url -w url=/blog/%` produces:\n    ```sql\n    SELECT path,count(1) '#reqs' FROM access_logs\n    WHERE time \u003e ? AND time \u003c ? AND (path LIKE ?)\n    GROUP BY 1\n    ORDER BY count(1)\n    DESC LIMIT 5\n    ```\n\n## Configuration\n\nThe command-line arguments and flags are intended exclusively to express a requests count query. The configuration, which isn't expected to change across command invocations, is left to environment variables:\n\n- `NGTOP_LOGS_PATH`: path pattern to find the nginx access logs. Defaults to `/var/log/nginx/access.log*`. The pattern is expanded using Go's [`path/filepath.Glob`](https://pkg.go.dev/path/filepath#Glob).\n- `NGTOP_LOG_FORMAT`: The [nginx log_format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format \"\n\") specification to parse the log entries. By default combined logs are assumed, which is equivalent to:\n  ```\n  NGTOP_LOG_FORMAT='$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\"'\n  ```\n- `NGTOP_DEBUG`: when set, internal logs will be printed to standard output.\n- `NGTOP_DB`: location of the SQLite db where the parsed logs are stored. Defaults to `./ngtop.db`.\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacundoolano%2Fngtop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacundoolano%2Fngtop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacundoolano%2Fngtop/lists"}