{"id":13856834,"url":"https://github.com/dustalov/ballcone","last_synced_at":"2025-07-13T19:32:51.352Z","repository":{"id":38219777,"uuid":"239359573","full_name":"dustalov/ballcone","owner":"dustalov","description":"Ballcone is a fast and lightweight server-side Web analytics solution.","archived":false,"fork":false,"pushed_at":"2024-05-21T21:42:54.000Z","size":1572,"stargazers_count":57,"open_issues_count":5,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-21T22:37:28.038Z","etag":null,"topics":["analytics","backend","columnar-storage","dashboard","hacktoberfest","metrics","monetdb","nginx","no-javascript","python","server","server-side","syslog","web-analytics"],"latest_commit_sha":null,"homepage":"https://news.ycombinator.com/item?id=23054322","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dustalov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-09T19:15:39.000Z","updated_at":"2024-05-21T21:42:52.000Z","dependencies_parsed_at":"2023-12-06T21:31:10.501Z","dependency_job_id":"0fa84f89-8ece-48f7-961d-1265c0fd47d8","html_url":"https://github.com/dustalov/ballcone","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustalov%2Fballcone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustalov%2Fballcone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustalov%2Fballcone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dustalov%2Fballcone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dustalov","download_url":"https://codeload.github.com/dustalov/ballcone/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":213988277,"owners_count":15666955,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","backend","columnar-storage","dashboard","hacktoberfest","metrics","monetdb","nginx","no-javascript","python","server","server-side","syslog","web-analytics"],"created_at":"2024-08-05T03:01:15.296Z","updated_at":"2024-08-05T03:02:10.976Z","avatar_url":"https://github.com/dustalov.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Ballcone\n\nBallcone is a fast and lightweight server-side Web analytics solution. It requires no JavaScript on your website.\n\n[![GitHub Tests][github_tests_badge]][github_tests_link] [![Docker Hub][docker_hub_badge]][docker_hub_link]\n\n[github_tests_badge]: https://github.com/dustalov/ballcone/workflows/Unit%20Tests/badge.svg?branch=master\n[github_tests_link]: https://github.com/dustalov/ballcone/actions?query=workflow%3A%22Unit+Tests%22\n[docker_hub_badge]: https://img.shields.io/docker/pulls/dustalov/ballcone\n[docker_hub_link]: https://hub.docker.com/r/dustalov/ballcone\n\n## Screenshots\n\n![Ballcone](https://user-images.githubusercontent.com/40397/80874920-4c9b9f00-8cc3-11ea-9848-18384d826e9c.png)\n\n![Ballcone: petrovich](https://user-images.githubusercontent.com/40397/80874963-4f968f80-8cc3-11ea-8342-666fe3be139c.png)\n\n## Design Goals\n\n* **Simplicity.** Ballcone requires *almost* zero set-up as it prefers convention over configuration\n* **Efficiency.** Ballcone performs *lightning-fast analytic queries* over data thanks to the underlying columnar database\n* **Specificity.** Ballcone aims at providing visual insights on the HTTP access logs with *no bloat*\n\n## Features\n\n* No JavaScript snippets required\n* GeoIP mapping with the [GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/) database\n* Extraction of platform and browser information from User-Agent\n\n## Architecture\n\nBallcone captures the `access_log` entries exported in JSON by nginx via the bundled [syslog logger](https://nginx.org/en/docs/syslog.html) (`65140/udp`). These entries are stored in the embedded DuckDB database. Ballcone uses it to perform data manipulation and analytic queries. Also, Ballcone provides a convenient Web interface (`8080/tcp`) for accessing and observing the gathered data.\n\n```\n          +-----------+            +------------+\n   HTTP   |           |   syslog   |            |   HTTP\n\u003c--------\u003e+   nginx   +-----------\u003e+  Ballcone  +\u003c--------\u003e\n          |           |    JSON    |            |\n          +-----------+            +------------+\n                                   |   DuckDB   |\n                                   +------------+\n```\n\nFor better performance, Ballcone inserts data in batches, committing them to DuckDB every few seconds (five seconds by default).\n\n## Requirements\n\n* [Python](https://www.python.org/) 3.9\n* [DuckDB](https://duckdb.org/) \u0026geq; 0.4.0\n* [nginx](https://nginx.org/) \u0026geq; 1.7.1\n\n## Demo\n\nThis repository contains an example configuration of nginx and Ballcone. Just run the container from Docker Hub or build it locally. nginx will be available at \u003chttp://127.0.0.1:8888/\u003e and Ballcone will be available at \u003chttp://127.0.0.1:8080/\u003e.\n\n```shell\ndocker-compose up\n# or\ndocker run --rm -p '127.0.0.1:8888:80' -p '127.0.0.1:8080:8080' dustalov/ballcone:demo\n```\n\n## Naming and Meaning\n\n**Ballcone** has two meanings.\n\nFirst, it is the romanization of the Russian word *балкон* that means a [balcony](https://en.wikipedia.org/wiki/Balcony). You go to the balcony to breath some fresh air and look down at the things outside.\n\nSecond, if a *ball* is inscribed in a *cone*, it resembles the all-seeing eye (help wanted: [dustalov/ballcone#8](https://github.com/dustalov/ballcone/issues/8)).\n\nRegardless of the meaning you prefer, Ballcone helps you to watch your websites.\n\n## Installation\n\nThe simplest way to get started is to run `make pipenv` after cloning the repository. Just make sure [Pipenv](https://pipenv.pypa.io/en/latest/) is installed.\n\n### Getting Ballcone\n\nRunning the Docker image is the simplest way to get started. Docker Hub contains automated builds of the Ballcone source code from GitHub: \u003chttps://hub.docker.com/r/dustalov/ballcone\u003e. The following command runs Ballcone on `127.0.0.1`: the syslog protocol will be available via `65140/udp`, the Web interface will be available via `8080/tcp`, and the data will be stored in the `/var/lib/ballcone` directory on the host machine.\n\n```shell\ndocker run -p '127.0.0.1:8080:8080' -p '127.0.0.1:65140:65140/udp' -v '/var/lib/ballcone:/usr/src/app/duckdb' --restart=unless-stopped dustalov/ballcone ballcone -sh '0.0.0.0' -wh '0.0.0.0' -d 'duckdb/ballcone.duckdb'\n```\n\nHowever, Docker is not the only option. Alternatively, Ballcone can be packaged into a standalone executable using [PyInstaller](http://www.pyinstaller.org/) and runned as a [systemd](https://systemd.io/) service (see [ballcone.service](ballcone.service) as an example):\n\n```shell\nmake pyinstaller\nsudo make install-systemd\nsudo systemctl start ballcone\n```\n\nFinally, Ballcone can be installed directly on the host machine for manual runs:\n\n```shell\npip3 install -e git+https://github.com/dustalov/ballcone@master#egg=ballcone\n```\n\nNote that `ballcone` without arguments creates the `ballcone.duckdb` database file inside the current directory.\n\n### Configuring nginx\n\nYou need to define the JSON-compatible log format for your service in the nginx configuration file. Let us call it `ballcone_json_example`. This format is similar to the one used in Matomo (see [matomo-log-analytics](https://github.com/matomo-org/matomo-log-analytics)). It should be put *before* the `server` context.\n\n```Nginx\nlog_format ballcone_json_example escape=json\n    '{'\n    '\"service\": \"example\", '\n    '\"ip\": \"$remote_addr\", '\n    '\"host\": \"$host\", '\n    '\"path\": \"$request_uri\", '\n    '\"status\": \"$status\", '\n    '\"referrer\": \"$http_referer\", '\n    '\"user_agent\": \"$http_user_agent\", '\n    '\"length\": $bytes_sent, '\n    '\"generation_time_milli\": $request_time, '\n    '\"date\": \"$time_iso8601\"'\n    '}';\n```\n\nThen, you should put this `access_log` directive *inside* the `server` context to transfer logs via the [syslog protocol](https://nginx.org/en/docs/syslog.html).\n\n```Nginx\naccess_log syslog:server=127.0.0.1:65140 ballcone_json_example;\n```\n\nPlease look at the complete example of nginx configuration in [demo/nginx.conf](demo/nginx.conf).\n\n## Roadmap\n\nRoadmap is available at \u003chttps://github.com/dustalov/ballcone/issues\u003e.\n\n## Alternatives\n\n* Web analytics solutions: [Matomo](https://matomo.org/), [Google Analytics](http://google.com/analytics/), [Yandex.Metrica](https://metrica.yandex.com/), etc.\n* Columnar data storages: [ClickHouse](https://clickhouse.tech/), [Citus](https://github.com/citusdata/citus), [MariaDB ColumnStore](https://mariadb.com/kb/en/mariadb-columnstore/), etc.\n* Log management: [Graylog](https://www.graylog.org/), [Fluentd](https://www.fluentd.org/), [Elasticsearch](https://github.com/elastic/elasticsearch), etc.\n\n## Copyright\n\nCopyright \u0026copy; 2020\u0026ndash;2023 Dmitry Ustalov. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdustalov%2Fballcone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdustalov%2Fballcone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdustalov%2Fballcone/lists"}