{"id":18888016,"url":"https://github.com/codex-team/hawk.collector","last_synced_at":"2025-07-17T08:38:38.812Z","repository":{"id":38329749,"uuid":"162291356","full_name":"codex-team/hawk.collector","owner":"codex-team","description":"High-performance messages collector for the Hawk project","archived":false,"fork":false,"pushed_at":"2024-12-26T14:45:32.000Z","size":10537,"stargazers_count":1,"open_issues_count":20,"forks_count":0,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-12-26T15:25:54.386Z","etag":null,"topics":["collector","fasthttp","hawk","high-performance","websockets"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codex-team.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-18T13:20:03.000Z","updated_at":"2024-12-26T14:22:48.000Z","dependencies_parsed_at":"2024-01-23T21:16:04.333Z","dependency_job_id":"ced138b7-6048-4c63-a1d6-6272d44b6a45","html_url":"https://github.com/codex-team/hawk.collector","commit_stats":{"total_commits":181,"total_committers":11,"mean_commits":"16.454545454545453","dds":"0.33149171270718236","last_synced_commit":"67943874805c15f54cb6bb475d4f11ddff2f6835"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codex-team","download_url":"https://codeload.github.com/codex-team/hawk.collector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231947007,"owners_count":18450140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collector","fasthttp","hawk","high-performance","websockets"],"created_at":"2024-11-08T07:41:21.639Z","updated_at":"2025-07-17T08:38:38.800Z","avatar_url":"https://github.com/codex-team.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hawk.collector\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/codex-team/hawk.collector)](https://goreportcard.com/report/github.com/codex-team/hawk.collector)\n\nPowerful module that can handle most errors around the web.\n\n# Build and run\n\nInstall RabbitMQ manually and build Hawk.collector\n\n```bash\nmake build\n./bin/hawk.collector\n```\n\n## Build for specific os\n\n```bash\nmake build-linux\nmake build-windows\nmake build-darwin\n```\n\nor build all\n\n```bash\nmake build-all\n```\n\n## Import RabbitMQ definitions\n\n```bash\nrabbitmqadmin import config/rabbit.definitions.json\n```\n\n## Run in Docker\n\n```bash\nmake docker\n```\n\n# Data flows\n\n## Request from errors catcher\n\nThe following structure represents data that go through the HTTP request (`POST` request to `'/'` with `Content-Type: application/json`)\n\n| name         | type            | description                                         |\n| ------------ | --------------- | --------------------------------------------------- |\n| token        | string (base64) | JWT in base64 format                                |\n| payload      | raw JSON        | Nested valid JSON                                   |\n| catcherType  | string          | Type of the catcher (`errors/golang`, `errors/php`) |\n\nJSON example\n```\n{\n  \"token\": \"...\",\n  \"catcherType\": \"errors/golang\",\n  \"payload\": {\n    \"title\": \"Test exception\",\n    \"timestamp\": 1545203808\n  }\n}\n```\n\n## Request to upload sourcemap\n\nThe following structure represents data got through the HTTP request (`POST` request to `'/release'` with `Content-Type: multipart/form-data`)\n\n### Form values\n| name         | type                       | description                                         |\n| ------------ | -------------------------- | --------------------------------------------------- |\n| release      | string                     | Release name                                        |\n| file         | multipart (optional)       | Content of the binary file                          |\n| commits      | commitMessage (optional)   | Suspected commits                                   |\n\n**commitMessage** has the following format:\n\n| name         | type       | description                                         |\n| ------------ | ---------- | --------------------------------------------------- |\n| hash         | string     | Commit hash                                         |\n| title        | string     | Commit description                                  |\n| author       | string     | Commit author                                       |\n| date         | string     | Commit date                                         |   \n\nAuthentication is made via `bearer` token.\n\ncURL sending example\n```\ncurl --request POST \\\n -F 'release=1.0.1'\\\n -F 'commits=[{\"hash\":\"557940a440352d9d86ad5610f2e366aafb2729e4\",\"title\":\"Add some stuff\",\"author\":\"somebody@codex.so\",\"date\":\"Wed May 6 13:37:00 2021 +0300\"}]'\\\n -F \"repository=https://github.com/codex-team/hawk.api.nodejs\"\\\n -F file=@\"main.min.js.map\"\\\n -H \"Authorization: Bearer TOKEN\" https://test.stage-k1.hawk.so/release\n```\n\n## Response message\nHTTP response from the collector. It is provided as JSON with HTTP status code.\n\n| name    | type   | description               |\n| ------- | ------ | ------------------------- |\n| code    | int    | internal error code       |\n| error   | bool   | if the error was occurred |\n| message | string | result details            |\n\nFor now there are two possible HTTP status codes: `200 (OK)` and `400 (Bad request)`.\n\nExamples\n\n```\n{\"error\": true, \"message\": \"Token is empty\", \"code\": 200}\n```\n\n```\n{\"error\": true, \"message\": \"Invalid JSON format\", \"code\": 400}\n```\n\nNo body will be returned for the valid response (`200`).\n\n## Websocket transport\n\nErrors can be sent via websockets (for example with the help of [wscat](https://github.com/websockets/wscat) util).\n```\n~# wscat -c wss://test.stage-k1.hawk.so/ws                                                                                                                                                                                                              ✔  11720  20:53:23\nconnected (press CTRL+C to quit)\n\u003e f\n\u003c {\"code\":400,\"error\":true,\"message\":\"Invalid JSON format\"}\n\u003e {\"token\": \"...\",\"catcherType\": \"errors/golang\",\"payload\": {\"title\": \"Test exception\",\"timestamp\": 1545203808}}\n\u003c {\"code\":200,\"error\":false,\"message\":\"OK\"}\n```\n\n# Message broker\n\nFor now we support RabbitMQ as a general AMQP broker.\nWe declare a durable **exchange** with `errors` name.\nThe valid payload JSON from `Request` structure goes directly to the exchange with the route specified by `catcherType` value.\n\n# Environment variables\n\nBasic configuration is taken from `.env` file.\n\n| variable    | example value   | description               |\n| ------- | ------ | ------------------------- |\n| BROKER_URL   | amqp://guest:guest@localhost:5672/   | Connection URI to RabbitMQ  |\n| EXCHANGE | errors | Basic exchange for errors             |\n| RELEASE_EXCHANGE | release | Basic exchange for releases            |\n| RETRY_NUMBER | 10 | Try to establish connection with broker for N times            |\n| RETRY_INTERVAL | 4 | Wait N seconds before retry to establish connection with broker            |\n| JWT_SECRET | qwerty | JWT token secret key            |\n| MAX_REQUEST_BODY_SIZE | 20000000 | Maximum available HTTP body size for any request (in bytes)            |\n| MAX_ERROR_CATCHER_MESSAGE_SIZE | 25000 | Maximum available HTTP body size for error request (in bytes)            |\n| MAX_SOURCEMAP_CATCHER_MESSAGE_SIZE | 250000 | Maximum available HTTP body size for sourcemap request (in bytes)            |\n| LISTEN | localhost:3000 | Listen host and port            |\n| REDIS_URL | localhost:6379 | Redis address |\n| REDIS_PASSWORD | password | Redis password |\n| REDIS_DISABLED_PROJECT_SET | DisabledProjectsSet | Name of set that contains disabled projects IDs |\n| REDIS_BLACKLIST_IP_SET | BlacklistIPsSet | Name of set that contains IPs blacklist |\n| REDIS_ALL_IPS_MAP | AllIPsMap | Name of map with all IPs and their request counters |\n| REDIS_CURRENT_PERIOD_MAP | CurrentPeriodMap | Name of map that contains IPs and their request counters for current period |\n| BLOCKED_PROJECTS_UPDATE_PERIOD | 5s | Time interval to update blocked projects list |\n| BLACKLIST_UPDATE_PERIOD | 15s | Time interval to update blacklist |\n| BLACKLIST_THRESHOLD | 10000 | Amount of requests, which, when achieved, forces IP to get blocked |\n| NOTIFY_URL | https://notify.bot.ifmo.su/u/ABCD1234 | Address to send alerts in case of too many requests |\n| TOKEN_UPDATE_PERIOD | 10s | Time interval to update token cache |\n| PROJECTS_LIMITS_UPDATE_PERIOD | 3600 | Time interval to update projects limits cache (in seconds) |\n# Rate Limiting\n\nRate limiting is implemented using Redis to track and enforce request limits per project. The system supports configurable limits at the project, workspace and plan level.\n\n## Configuration\n\nRate limits can be configured at multiple levels and applied in the following order (highest to lowest):\n\n1. Project level - Individual project-specific limits\n2. Workspace level - Limits that apply to all projects in a workspace\n3. Plan level - Default limits from the workspace's tariff plan\n\n## Implementation\n\nRate limits are tracked in `rate_limit` Redis set with the following pattern:\n\n```go\n// Key: \"project_id\" -\u003e value: \"timestamp:count\"\n// example: \"6762b5db032b200023854b2c\" -\u003e \"1737483572:5\"\n```\n\nEach project's rate limit data contains:\n- Timestamp of the current window\n- Request count in the current window\n\n### Rate Limit Parameters\n\nTwo main parameters control the rate limiting:\n\n- `EventsLimit` - Maximum number of events allowed in the period\n- `EventsPeriod` - Time window in seconds for the limit (in seconds)\n\n## Configuration\n\nRate limits are fetched from MongoDB. You can find them in the `rateLimitSettings` field of the `plans,workspaces,projects` collections.\n\n`rateLimitSettings` is object with two fields:\n- `N` - Maximum number of events allowed in the period (`int64`)\n- `T` - Time window in seconds for the limit (in seconds) (`int64`)\n\n```json\n{\n  \"rateLimitSettings\": {\n    \"N\": {\n      \"$numberLong\": \"15\"\n    },\n    \"T\": {\n      \"$numberLong\": \"100\"\n    }\n  }\n}\n```\n\nRate limits are automatically enforced for all incoming error and release events. No additional configuration is needed at the client level.\n\nWhen a rate limit is exceeded, clients will receive a response like:\n\n```json\n{\n  \"code\": 402,\n  \"error\": true,\n  \"message\": \"Rate limit exceeded\"\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodex-team%2Fhawk.collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodex-team%2Fhawk.collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodex-team%2Fhawk.collector/lists"}