{"id":18545763,"url":"https://github.com/wakatime/wakaq-ts","last_synced_at":"2025-04-09T19:32:09.861Z","repository":{"id":214473049,"uuid":"697493649","full_name":"wakatime/wakaq-ts","owner":"wakatime","description":"Background task queue for TypeScript backed by Redis, a super minimal Celery","archived":false,"fork":false,"pushed_at":"2025-01-05T13:51:38.000Z","size":159,"stargazers_count":26,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-24T11:11:25.361Z","etag":null,"topics":["async","asyncronous-tasks","background-jobs","celery","distributed","job-queue","job-scheduler","queue","redis","rq","task-queue","task-queues","typescript","worker"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wakatime.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null}},"created_at":"2023-09-27T21:00:57.000Z","updated_at":"2025-03-07T19:38:50.000Z","dependencies_parsed_at":"2024-03-14T00:28:00.958Z","dependency_job_id":"0f1e14ad-dece-4f00-b771-6a8a2fab2bb9","html_url":"https://github.com/wakatime/wakaq-ts","commit_stats":null,"previous_names":["wakatime/wakaq-ts"],"tags_count":40,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakatime%2Fwakaq-ts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakatime%2Fwakaq-ts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakatime%2Fwakaq-ts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakatime%2Fwakaq-ts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wakatime","download_url":"https://codeload.github.com/wakatime/wakaq-ts/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247980833,"owners_count":21027803,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","asyncronous-tasks","background-jobs","celery","distributed","job-queue","job-scheduler","queue","redis","rq","task-queue","task-queues","typescript","worker"],"created_at":"2024-11-06T20:22:07.267Z","updated_at":"2025-04-09T19:32:09.423Z","avatar_url":"https://github.com/wakatime.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ![logo](https://raw.githubusercontent.com/wakatime/wakaq-ts/main/wakatime-logo.png 'WakaQ') WakaQ\n\n[![wakatime](https://wakatime.com/badge/github/wakatime/wakaq-ts.svg)](https://wakatime.com/badge/github/wakatime/wakaq-ts)\n\nBackground task queue for TypeScript backed by Redis, a super minimal Celery.\n\nFor the original Python version, see [WakaQ for Python][wakaq python].\n\n## Features\n\n- Queue priority\n- Delayed tasks (run tasks after a duration eta)\n- Scheduled periodic tasks\n- [Broadcast][broadcast] a task to all workers\n- Task [soft][soft timeout] and [hard][hard timeout] timeout limits\n- Optionally retry tasks on soft timeout\n- Combat memory leaks with `maxMemPercent` or `maxTasksPerWorker`\n- Super minimal\n\nWant more features like rate limiting, task deduplication, etc? Too bad, feature PRs are not accepted. Maximal features belong in your app’s worker tasks.\n\n## Installing\n\n    npm i --save wakaq\n\n## Using\n\n`app.ts`\n\n```TypeScript\nimport { Duration } from 'ts-duration';\nimport { CronTask, WakaQ, WakaQueue, WakaQWorker } from 'wakaq';\nimport { z } from 'zod';\nimport { prisma } from './db';\n\nexport const wakaq = new WakaQ({\n\n  /* Raise SoftTimeout in a task if it runs longer than 14 minutes. Can also be set per\n     task or queue. If no soft timeout set, tasks can run forever.\n  */\n  softTimeout: Duration.minute(14),\n\n  /* SIGKILL a task if it runs longer than 15 minutes. Can also be set per queue or\n     when enqueuing a task.\n  */\n  hardTimeout: Duration.minute(15),\n\n  /* Number of worker processes. Must be an int or str which evaluates to an\n     int. The variable \"cores\" is replaced with the number of processors on\n     the current machine.\n  */\n  concurrency: 'cores*4',\n\n  /* List your queues and their priorities.\n  */\n  queues: [\n    new WakaQueue('high priority'),\n    new WakaQueue('default'),\n  ],\n\n  /* Redis normally doesn't use TLS, but some cloud providers need it.\n  */\n  tls: process.env.NODE_ENV == 'production' ? { cert: '', key: '' } : undefined,\n\n  /* If the task soft timeouts, retry up to 3 times. Max retries comes first\n     from the task decorator if set, next from the Queue's maxRetries,\n     lastly from the option below. If No maxRetries is found, the task\n     is not retried on a soft timeout.\n  */\n  maxRetries: 3,\n\n  /* Schedule two tasks, the first runs every minute, the second once every ten minutes.\n     To run scheduled tasks you must keep `npm run scheduler` running as a daemon.\n  */\n  schedules: [\n\n    // Runs myTask once every 5 minutes.\n    new CronTask('*/5 * * * *', 'myTask'),\n  ],\n});\n\nexport const createUserInBackground = wakaq.task(\n  async (firstName: string) =\u003e {\n    const name = z.string().safeParse(firstName);\n    if (!result.success) {\n      throw new Error(result.error.message);\n    }\n    await prisma.User.create({\n      data: { firstName: result.data },\n    });\n  },\n  { name: 'createUserInBackground' },\n);\n\n```\n\nAdd these scripts to your `package.json`:\n\n```JSON\n{\n  \"scripts\": {\n    \"worker\": \"tsx scripts/wakaqWorker.ts\",\n    \"scheduler\": \"tsx scripts/wakaqScheduler.ts\",\n    \"info\": \"tsx scripts/wakaqInfo.ts\",\n    \"purge\": \"tsx scripts/wakaqPurge.ts\"\n  }\n}\n```\n\nCreate these files in your `scripts` folder:\n\n`scripts/wakaqWorker.ts`\n\n```TypeScript\nimport { WakaQWorker } from 'wakaq';\nimport { wakaq } from '../app.js';\n\n// Can't use tsx directly because it breaks IPC (https://github.com/esbuild-kit/tsx/issues/201)\nawait new WakaQWorker(wakaq, ['node', '--no-warnings=ExperimentalWarning', '--import', 'tsx', 'scripts/wakaqChild.ts']).start();\nprocess.exit(0);\n```\n\n`scripts/wakaqScheduler.ts`\n\n```TypeScript\nimport { WakaQScheduler } from 'wakaq';\nimport { wakaq } from '../app.js';\n\nawait new WakaQScheduler(wakaq).start();\nprocess.exit(0);\n```\n\n`scripts/wakaqChild.ts`\n\n```TypeScript\nimport { WakaQChildWorker } from 'wakaq';\nimport { wakaq } from '../app.js';\n\n// import your tasks so they're registered\n// also make sure to enable tsc option verbatimModuleSyntax\n\nawait new WakaQChildWorker(wakaq).start();\nprocess.exit(0);\n```\n\n`scripts/wakaqInfo.ts`\n\n```TypeScript\nimport { inspect } from 'wakaq';\nimport { wakaq } from '../app.js';\nconsole.log(JSON.stringify(await inspect(await wakaq.connect()), null, 2));\nwakaq.disconnect();\n```\n\n`scripts/wakaqPurge.ts`\n\n```TypeScript\nimport { numPendingTasksInQueue, numPendingEtaTasksInQueue, purgeQueue, purgeEtaQueue } from 'wakaq';\nimport { wakaq } from '../app.js';\n\nconst queueName = process.argv.slice(2)[0];\nconst queue = wakaq.queuesByName.get(queueName ?? '');\nif (!queue) {\n  throw new Error(`Queue not found: ${queueName}`);\n}\nawait wakaq.connect();\nlet count = await numPendingTasksInQueue(wakaq, queue);\nawait purgeQueue(wakaq, queue);\ncount += await numPendingEtaTasksInQueue(wakaq, queue);\nawait purgeEtaQueue(wakaq, queue);\nconsole.log(`Purged ${count} tasks from ${queue.name}`);\nwakaq.disconnect();\n```\n\nAfter running `npm run worker` when you run `createUserInBackground.enqueue('alan')` your task executes in the background on the worker server.\n\n## Deploying\n\n#### Optimizing\n\nSee the [WakaQ init params][wakaq init] for a full list of options, like Redis host and Redis socket timeout values.\n\nWhen using in production, make sure to [increase the max open ports][max open ports] allowed for your Redis server process.\n\nWhen using eta tasks a Redis sorted set is used, so eta tasks are automatically deduped based on task name, args, and kwargs.\nIf you want multiple pending eta tasks with the same arguments, just add a throwaway random string or uuid to the task’s args.\n\n#### Running as a Daemon\n\nHere’s an example systemd config to run `wakaq worker` as a daemon:\n\n```systemd\n[Unit]\nDescription=WakaQ Worker Service\n\n[Service]\nWorkingDirectory=/opt/yourapp\nExecStart=npm run worker\nRemainAfterExit=no\nRestart=always\nRestartSec=30s\nKillSignal=SIGINT\nLimitNOFILE=99999\n\n[Install]\nWantedBy=multi-user.target\n```\n\nCreate a file at `/etc/systemd/system/wakaqworker.service` with the above contents, then run:\n\n    systemctl daemon-reload \u0026\u0026 systemctl enable wakaqworker\n\n[wakaq python]: https://github.com/wakatime/wakaq\n[broadcast]: https://github.com/wakatime/wakaq-ts/blob/v1.0.0/src/task.ts#L61\n[soft timeout]: https://github.com/wakatime/wakaq-ts/blob/v1.0.0/src/childWorker.ts#L98\n[hard timeout]: https://github.com/wakatime/wakaq-ts/blob/v1.0.0/src/worker.ts#L194\n[wakaq init]: https://github.com/wakatime/wakaq-ts/blob/v1.0.0/src/wakaq.ts#L27\n[max open ports]: https://wakatime.com/blog/47-maximize-your-concurrent-web-server-connections\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwakatime%2Fwakaq-ts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwakatime%2Fwakaq-ts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwakatime%2Fwakaq-ts/lists"}