{"id":18888073,"url":"https://github.com/codex-team/hawk.workers","last_synced_at":"2025-04-14T23:11:09.615Z","repository":{"id":40932922,"uuid":"157996256","full_name":"codex-team/hawk.workers","owner":"codex-team","description":"Hawk workers system","archived":false,"fork":false,"pushed_at":"2025-04-10T14:10:21.000Z","size":2387,"stargazers_count":7,"open_issues_count":38,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-10T14:47:25.388Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codex-team.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-17T15:12:58.000Z","updated_at":"2025-04-10T14:10:23.000Z","dependencies_parsed_at":"2024-01-23T21:16:05.548Z","dependency_job_id":"3cfd2856-01c1-443f-8573-3f25aac7ce2b","html_url":"https://github.com/codex-team/hawk.workers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.workers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.workers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.workers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codex-team%2Fhawk.workers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codex-team","download_url":"https://codeload.github.com/codex-team/hawk.workers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248975316,"owners_count":21192210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:42:02.406Z","updated_at":"2025-04-14T23:11:09.597Z","avatar_url":"https://github.com/codex-team.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/codex-team/hawk.workers.svg?branch=master)](https://travis-ci.com/codex-team/hawk.workers) [![Coverage Status](https://codecov.io/gh/codex-team/hawk.workers/branch/master/graphs/badge.svg?branch=master)](https://codecov.io/gh/codex-team/hawk.workers) [![Total alerts](https://img.shields.io/lgtm/alerts/g/codex-team/hawk.workers.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/codex-team/hawk.workers/alerts/) [![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/codex-team/hawk.workers.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/codex-team/hawk.workers/context:javascript)\n\n# Hawk Workers\n\nWorkers are services for processing hawk's background tasks\n\n\n\n## Requirements\n\n- [Registry](https://github.com/codex-team/hawk.registry)\n\nRegistry - RabbitMQ\n\n\u003e More info on setting up Registry [here](https://github.com/codex-team/hawk.registry)\n\nFor simplicity, Hawk workers can be used as part of the [Mono repository](https://github.com/codex-team/hawk.mono)\n\n## How to write a Worker\n\n- Inherit from `Worker` class and implement `handle` method which process tasks from registry (see more in [`lib/worker.js`](lib/worker.js)) [Example](workers/javascript/src/index.ts)\n\n- Define `type` - worker type (e.g `errors/nodejs`), which is also a Registry queue from where worker pulls tasks\n\n- Edit `.env` file (see more below)\n\n- Use `worker.start()` to start your worker. You **should** write a simple runner like [this](workers/nodejs/runner.js)\n\n- Set `LOG_LEVEL` to `verbose` if you want message logs\n\n  \u003e Also you can use `worker.logger` which is [`winston.Logger`](https://github.com/winstonjs/winston) to log something\n                                                           \u003e\n                                                           \u003e\n## How to run workers\n\n1. Make sure you are in Workers root directory\n2. Add worker package as [Yarn Workspace](https://yarnpkg.com/lang/en/docs/workspaces/) — add worker's path to the root's package.json at \"workspaces\" section\n3. `yarn install`\n4. `yarn worker worker-package-name` (package name from worker's package.json). You can pass several workers separated by space.\n5. (Optionally) You can add own script to run specified worker, for example `\"run-js\": \"ts-node ./runner.ts hawk-worker-javascript\"`\n\n\u003e Note. You can override some env variables on worker running:\n\n```\nSIMULTANEOUS_TASKS=1 yarn worker hawk-worker-release\n```\n\n## Running workers with PM2\n\n- Install PM2\n\n  ```bash\n  yarn global add pm2\n  OR\n  npm i -g pm2\n  ```\n\n- If you've written your worker add it to `ecosystem.config.js` like the existing ones\n\n- Edit `.env` files\n\n- Run it\n\n  ```bash\n  # Run all workers\n  pm2 start\n\n  # Run specific worker, e.g. nodejs\n  pm2 start nodejs\n  ```\n\n\u003e Feel free to tune your setting in `ecosystem.config.js` file, [more info](https://pm2.io/doc/en/runtime/reference/ecosystem-file/)\n\n## Running workers with Docker\n\nBasic configuration is in `docker-compose.dev.yml`. \nPull image from https://hub.docker.com/r/codexteamuser/hawk-workers\n```\ndocker-compose -f docker-compose.dev.yml pull\n```\n\nIf you run mongodb and rabbitmq with `hawk.mono` repository, by default your docker network will be named `hawkmono_default`. \nThis network name is written as external for workers.\n\nRun chosen worker (say hawk-worker-javascript)\n```\ndocker-compose -f docker-compose.dev.yml up hawk-worker-javascript\n```\n\n### Adding new workers\nMake sure that your `.env` configurations exists.\n\nAdd new section to the `docker-compose.{dev,prod}.yml` files.\n\n```\n hawk-worker-telegram:\n    image: \"codexteamuser/hawk-workers:prod\"\n    env_file:\n      - .env\n      - workers/telegram/.env\n    restart: unless-stopped\n    entrypoint: /usr/local/bin/node runner.js hawk-worker-telegram\n```\n\n## Error handling\n\nIf an error is thrown inside `handle` method it will be ignored, except if it is `CriticalError` or `NonCriticalError`\n\nOn `CriticalError` the currently processing message will be requeued to the same queue in Registry using `Worker.requeue` method\n\nOn `NonCriticalError` the currently processing message will be queued to stash queue in Registry using `Worker.sendToStash` method\n\n## Env vars\n\n| Variable           | Description                                                                                              | Default value      |\n| ------------------ | -------------------------------------------------------------------------------------------------------- | ------------------ |\n| REGISTRY_URL       | RabbitMQ connection URL                                                                                  | `amqp://localhost` |\n| SIMULTANEOUS_TASKS | RabbitMQ Consumer prefetch value (How many tasks can do simultaneously)                                  | 1                  |\n| LOG_LEVEL          | Log level (error,warn,info,versobe,debug,silly) [See more](https://github.com/winstonjs/winston#logging) | `info`             |\n\n**IMPORTANT**\n\n\u003e `.env` file in root act like _global_ preferences for all workers in `workers` folder.\n\u003e\n\u003e If some variable is present in root `.env` file, it is **NOT** overwritten by local `.env` in worker's folder\n\u003e\n\u003e This allows to set global MongoDB or/and RabbitMQ connection settings while leaving possibility to set local options for each worker\n\n## Testing\n\n- Make `.env` file with test settings\n- Run `yarn test:\u003ccomponent name\u003e`\n\n| Component                       | Command            | Requirements |\n| ------------------------------- | ------------------ | ------------ |\n| Base worker(`lib`)              | `yarn test:base`   | - RabbitMQ   |\n| NodeJS worker(`workers/nodejs`) | `yarn test:nodejs` | None         |\n\n## Database controller\n\nMongoDB controller is bundled(see [`lib/db`](lib/db))\n\nYou can tweak it (add schemas, etc) and use it in your workers to handle database communication\n\n### Example\n\n```javascript\nconst db = require(\"lib/db/mongoose-controller\");\n\nawait db.connect(); // Requires `MONGO_URL`\n\nawait db.saveEvent(event);\n```\n\n### Env vars\n\n| Variable  | Description            | Default value                      |\n| --------- | ---------------------- | ---------------------------------- |\n| MONGO_URL | MongoDB connection URL | mongodb://localhost:27017/hawk-dev |\n\n### Testing\n\n`yarn test:db`\n\n### Worker message format\n\n```jsonc\n{\n  // Access token with `projectId` in payload\n  \"token\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...\",\n  // Worker specific payload\n  \"payload\": {\n    \"title\": \"Error: ...\"\n    // other fields\n  }\n}\n```\n\n## Cache controller\n\nTo reduce an amount of requests or any performance improvements you can use an [lib/cache/controller](./lib/cache/controller).\n\nTo use it in worker, you need to:\n\n1. Call `this.prepareCache();` somewhere in worker to activate the cache module. For example, in `start()` method\n2. Use `this.cache.get(key, resover?, ttl?)` or `this.cache.set(key, value, ttl?)`\n\nAvailable methods:\n\n- `set(key: string, value: any, ttl?: number)` — cache data\n- `get(key: string, resolver?: Function, ttl?: number)` — get cached data (or resolve and cache). If you're passing a resolver, you may pass ttl too (for internal **set** command with resolver)\n- `del(key: string|string[])` — delete cached data\n- `flushAll()` — flush the whole data\n \n\u003e `ttl` (time to live) in seconds\n \n## Migrations\n\nTo create new migration use command\n\n```jsonc\nyarn migration create {migrationName}\n```\n\nEach migration file contains two methods: up and down.\n\n`Up` method executes revision and increases database version.\n\n`Down` method rollbacks database changes\n\nTo execute migration run\n\n```jsonc\nyarn migrate\n```\n\n### Todo\n\nRefactor mongo-migrate commands to have an opportunity to create or rollback\n\n[More details](https://www.npmjs.com/package/migrate-mongo)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodex-team%2Fhawk.workers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodex-team%2Fhawk.workers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodex-team%2Fhawk.workers/lists"}