{"id":19501164,"url":"https://github.com/gitbookio/micro-analytics","last_synced_at":"2025-04-25T23:30:59.864Z","repository":{"id":73453527,"uuid":"46444153","full_name":"GitbookIO/micro-analytics","owner":"GitbookIO","description":"A micro multi-website analytics database service designed to be fast and robust, built with Go and SQLite.","archived":false,"fork":false,"pushed_at":"2016-10-28T12:06:30.000Z","size":7317,"stargazers_count":76,"open_issues_count":9,"forks_count":18,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-04T05:04:57.289Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GitbookIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-11-18T20:04:58.000Z","updated_at":"2024-04-10T07:46:13.000Z","dependencies_parsed_at":"2023-02-28T13:01:45.556Z","dependency_job_id":null,"html_url":"https://github.com/GitbookIO/micro-analytics","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Fmicro-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Fmicro-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Fmicro-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Fmicro-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GitbookIO","download_url":"https://codeload.github.com/GitbookIO/micro-analytics/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250912660,"owners_count":21506865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T22:11:39.412Z","updated_at":"2025-04-25T23:30:59.858Z","avatar_url":"https://github.com/GitbookIO.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# µAnalytics\n\nA micro multi-website analytics database service designed to be fast and robust, built with Go and SQLite.\n\n![Schema](./schema.png)\n\n## Principle\n\nAnalytics databases tend to grow fast and exponentially.\nRequesting data for one specific website from a single database thus become very slow over time.\nBut analytics data are highly decoupled between two websites.\n\nThe idea behind **µAnalytics** is to shard your analytics data on a key, which is usually a website name.\nEach shard thus only contains a specific website data, allowing faster response times and easy horizontal scaling.\n\nTo handle requests even faster, **µAnalytics** automatically manages a pool of connections to multiple shards at a time.\n\nBy default, the service keeps 10 connections alive.\nBut you can easily increase/decrease the max number of alive shards with the `--connections` flag when launching the app.\n\n\n## Install\n\n```Shell\n$ go install github.com/GitbookIO/micro-analytics\n```\n\n\n## Service launching\n\nTo launch the application, simply run:\n```\n$ ./micro-analytics\n```\n\nThe command takes the following optional parameters:\n\nParameter | Environment Variable | Usage | Type | Default Value\n---- | ---- | ---- | ---- | ----\n`--user, -u` | `MA_USER` | Username for basic auth | String | `\"\"`\n`--password, -w` | `MA_PASSWORD` | Password for basic auth | String | `\"\"`\n`--port, -p` | `MA_PORT` | Port to listen on | String | `\"7070\"`\n`--root, -r` | `MA_ROOT` | Database directory | String | `\"./dbs\"`\n`--connections, -c` | `MA_POOL_SIZE` | Max number of alive shards connections | Number | `1000`\n`--idle-timeout, -i` | `MA_POOL_TIMEOUT` | Idle timeout for DB connections in seconds | Number | `60`\n`--cache-directory, -d` | `MA_CACHE_DIR` | Cache directory | String | `\".diskache\"`\n\nIf `--user` is provided, the service will automatically use [basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) on all requests.\n\nThe actual cache directory will be a subdirectory named after the app major version. The default will then be `./.diskache/0`.\n\n## Analytics schema\n\nAll shards of the **µAnalytics** database share the same TABLE schema:\n```SQL\nCREATE TABLE visits (\n    time            INTEGER,\n    event           TEXT,\n    path            TEXT,\n    ip              TEXT,\n    platform        TEXT,\n    refererDomain   TEXT,\n    countryCode     TEXT\n)\n```\n\n\n## Service requests\n\n### GET requests\n\n##### Common Parameters\n\nEvery query for a specific website can be executed using a time range.\nEvery following GET request thus takes the two following optional query string parameters:\n\nName | Type | Description | Default | Example\n---- | ---- | ---- | ---- | ----\n`start` | Date | Start date to query a range | none | `\"2015-11-20T12:00:00.000Z\"`\n`end` | Date | End date to query a range | none | `\"2015-11-21T12:00:00.000Z\"`\n\nThe dates can be passed either as:\n - ISO (RFC3339) `\"2015-11-20T12:00:00.000Z\"`\n - UTC (RFC1123) `\"Fri, 20 Nov 2015 12:00:00 GMT\"`\n - A Unix timestamp as a String `\"1448020800\"`\n\n##### Common Aggregation Parameters\n\nName | Type | Description | Default | Example\n---- | ---- | ---- | ---- | ----\n`unique` | Boolean | Include the total number of unique visitors in response | none | `true`\n\n##### Common Aggregation Response Values\n\nExcept for `GET /:website`, every response to a GET request will contain the two following values:\n\nName | Type | Description\n---- | ---- | ----\n`total` | Integer | Total number of visits\n`unique` | Integer | Total number of unique visitors based on `ip`, set to `0` unless `unique=true` is passed as a query string parameter\n\n#### GET `/:website`\n\nReturns the full analytics for a website.\n\n##### Response\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"time\": \"2015-11-25T16:00:00+01:00\",\n            \"event\": \"download\",\n            \"path\": \"/somewhere\",\n            \"ip\": \"127.0.0.1\",\n            \"platform\": \"Windows\",\n            \"refererDomain\": \"gitbook.com\",\n            \"countryCode\": \"fr\"\n        },\n    ...\n    ]\n}\n```\n\n#### GET `/:website/count`\n\nReturns the count of analytics for a website. The `unique` query string parameter is not necessary for this request.\n\n##### Response\n\n```JavaScript\n{\n    \"total\": 1000,\n    \"unique\": 900\n}\n```\n\n#### GET `/:website/countries`\n\nReturns the number of visits per `countryCode`.\n\n##### Response\n\n`label` contains the country full name.\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"id\": \"fr\",\n            \"label\": \"France\",\n            \"total\": 1000,\n            \"unique\": 900\n        },\n        ...\n    ]\n}\n```\n\n#### GET `/:website/platforms`\n\nReturns the number of visits per `platform`.\n\n##### Response\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"id\": \"Linux\",\n            \"label\": \"Linux\",\n            \"total\": 1000,\n            \"unique\": 900\n        },\n        ...\n    ]\n}\n```\n\n#### GET `/:website/domains`\n\nReturns the number of visits per `refererDomain`.\n\n##### Response\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"id\": \"gitbook.com\",\n            \"label\": \"gitbook.com\",\n            \"total\": 1000,\n            \"unique\": 900\n        },\n        ...\n    ]\n}\n```\n\n#### GET `/:website/events`\n\nReturns the number of visits per `event`.\n\n##### Response\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"id\": \"download\",\n            \"label\": \"download\",\n            \"total\": 1000,\n            \"unique\": 900\n        },\n        ...\n    ]\n}\n```\n\n#### GET `/:website/time`\n\nReturns the number of visits as a time serie. The interval in seconds can be specified as an optional query string parameter. Its default value is `86400`, equivalent to one day.\n\n##### Parameters\n\nName | Type | Description | Default | Example\n---- | ---- | ---- | ---- | ----\n`interval` | Integer | Interval of the time serie | `86400` (1 day) | `3600`\n\n##### Response\n\nExample with interval set to `3600`:\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"start\": \"2015-11-24T12:00:00.000Z\",\n            \"end\": \"2015-11-24T13:00:00.000Z\",\n            \"total\": 450,\n            \"unique\": 390\n        },\n        {\n            \"start\": \"2015-11-24T13:00:00.000Z\",\n            \"end\": \"2015-11-24T14:00:00.000Z\",\n            \"total\": 550,\n            \"unique\": 510\n        },\n        ...\n    ]\n}\n```\n\n### POST requests\n\n#### POST `/:website`\n\nInsert new data for the specified website.\n\n##### POST Body\n\n```JavaScript\n{\n    \"time\": \"2015-11-24T13:00:00.000Z\", // optional\n    \"event\": \"download\",\n    \"ip\": \"127.0.0.1\",\n    \"path\": \"/README.md\",\n    \"headers\": {\n        // ...\n        // HTTP headers received from your visitor\n    }\n}\n```\n\nThe `time` parameter is optional and is set to the date of your POST request by default.\n\nPassing the HTTP headers in the POST body allows the service to extract the `refererDomain` and `platform` values.\nThe `countryCode` will be deduced from the passed `ip` parameter using [Maxmind's GeoLite2 database](http://dev.maxmind.com/geoip/geoip2/geolite2/).\n\n#### POST `/:website/bulk`\n\nInsert a list of analytics for a specific website. The analytics can be sent directly in DB format, with `time` being a String value.\n\n`time` can be passed as either:\n - ISO (RFC3339) `\"2015-11-20T12:00:00.000Z\"`\n - UTC (RFC1123) `\"Fri, 20 Nov 2015 12:00:00 GMT\"`\n - A Unix timestamp as a String `\"1448020800\"`\n\nIf the `time` parameter is not provided, it will be defaulted to the exact time of the server processing the `POST` request.\n\nAs for the `POST /:website` method, the analytics can also have an optional `headers` parameter.\nIf the `refererDomain` and/or `platform` values are not passed in the JSON body, the `headers` parameter will be used to set these values automatically.\n\n##### POST Body\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"time\": \"1450098642\",\n            \"ip\": \"127.0.0.1\",\n            \"event\": \"download\",\n            \"path\": \"/somewhere\",\n            \"platform\": \"Apple Mac\",\n            \"refererDomain\": \"www.gitbook.com\",\n            \"countryCode\": \"fr\"\n        },\n        {\n            \"time\": \"2015-11-20T12:00:00.000Z\",\n            \"ip\": \"127.0.0.1\",\n            \"event\": \"login\",\n            \"path\": \"/someplace\",\n            \"headers\": {\n                // ...\n                // HTTP headers received from your visitor\n            }\n        }\n    ]\n}\n```\n\nThe `countryCode` will be reprocessed by the service using GeoLite2 based on the `ip`.\n\n#### POST `/bulk`\n\nInsert a list of analytics for different websites. The analytics have the same format as `POST /:website/bulk`, with a mandatory `website` parameter.\n\n##### POST Body\n\n```JavaScript\n{\n    \"list\": [\n        {\n            \"website\": \"website-1\",\n            \"time\": \"1450098642\",\n            \"ip\": \"127.0.0.1\",\n            \"event\": \"download\",\n            \"path\": \"/somewhere\",\n            \"platform\": \"Apple Mac\",\n            \"refererDomain\": \"www.gitbook.com\",\n            \"countryCode\": \"fr\"\n        },\n        {\n            \"website\": \"website-2\",\n            \"time\": \"2015-11-20T12:00:00.000Z\",\n            \"ip\": \"127.0.0.1\",\n            \"event\": \"login\",\n            \"path\": \"/someplace\",\n            \"headers\": {\n                // ...\n            }\n        }\n    ]\n}\n```\n\n### DELETE requests\n\n#### DELETE `/:website`\n\nFully delete a shard from the file system.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitbookio%2Fmicro-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgitbookio%2Fmicro-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitbookio%2Fmicro-analytics/lists"}