{"id":31829942,"url":"https://github.com/legout/naq","last_synced_at":"2025-10-11T20:58:57.893Z","repository":{"id":286420083,"uuid":"961344441","full_name":"legout/naq","owner":"legout","description":"Simple, asynchronous job queueing library for Python on top of NATS.io","archived":false,"fork":false,"pushed_at":"2025-09-04T16:24:51.000Z","size":4501,"stargazers_count":17,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-10T03:27:11.933Z","etag":null,"topics":["job","nats","nats-io","python","queue","task"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/legout.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.qmd","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-06T10:13:55.000Z","updated_at":"2025-08-18T10:17:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"843109ac-7d7b-4864-8be3-9419d2496efb","html_url":"https://github.com/legout/naq","commit_stats":null,"previous_names":["legout/naq"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/legout/naq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fnaq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fnaq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fnaq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fnaq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/legout","download_url":"https://codeload.github.com/legout/naq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/legout%2Fnaq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279008636,"owners_count":26084480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["job","nats","nats-io","python","queue","task"],"created_at":"2025-10-11T20:58:51.906Z","updated_at":"2025-10-11T20:58:57.887Z","avatar_url":"https://github.com/legout.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NAQ - NATS Asynchronous Queue\n\n[![PyPI version](https://badge.fury.io/py/naq.svg)](https://badge.fury.io/py/naq) \u003c!-- Placeholder - Add actual badge once published --\u003e\n[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/legout/naq)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/legout/naq/blob/main/LICENSE)\n[![Python version](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/release/python-3120/)\n\n\n**NAQ** is a simple, asynchronous job queueing library for Python, inspired by [RQ (Redis Queue)](https://python-rq.org/), but built entirely on top of [NATS](https://nats.io/) and its JetStream persistence layer.\n\nThink of it as **N**ATS **A**synchronous **Q**ueue - your simple way of *Naqin' on NATS's Door* for background job processing.\n\nIt allows you to easily enqueue Python functions to be executed asynchronously by worker processes, leveraging the power and resilience of NATS JetStream for message persistence and delivery.\n\n## Security\n\n**:warning: Important Security Notice :warning:**\n\nBy default, `naq` uses `cloudpickle` to serialize job data for maximum flexibility with Python objects. However, `cloudpickle` can execute arbitrary code and is **not secure** if the job producer cannot be trusted.\n\nIf you are accepting jobs from untrusted sources, **you must switch to the `JsonSerializer`**.\n\nYou can do this by setting the `NAQ_JOB_SERIALIZER` environment variable:\n```bash\nexport NAQ_JOB_SERIALIZER=json\n```\n\nThe `JsonSerializer` is safer as it only serializes data to and from basic JSON types and handles functions by referencing their import path, preventing arbitrary code execution. See the [`SECURITY.md`](SECURITY.md) file for more details.\n\n## Timezone Handling\n\nAll internal scheduling and time handling within `naq` are based on UTC (Coordinated Universal Time). This is to ensure consistent and unambiguous behavior across different systems and timezones.\n\nWhen scheduling jobs, it is highly recommended to use timezone-aware `datetime` objects, specifically those set to UTC.\n\n### Scheduling with UTC\n\nThe best practice is to always use `datetime.datetime.now(datetime.timezone.utc)` for the current time or to create `datetime` objects with `tzinfo=datetime.timezone.utc`.\n\n```python\n# schedule_example_utc.py\nimport datetime\nfrom naq import enqueue_at_sync, enqueue_in_sync\nfrom tasks import count_words\n\n# Get the current time in UTC\nnow_utc = datetime.datetime.now(datetime.timezone.utc)\n\n# Schedule to run at a specific UTC time\nrun_at_utc = now_utc + datetime.timedelta(seconds=30)\njob_at = enqueue_at_sync(run_at_utc, count_words, \"Job scheduled with explicit UTC time.\")\nprint(f\"Job {job_at.job_id} scheduled for {run_at_utc} (UTC)\")\n\n# Scheduling with a timedelta is also implicitly UTC-based\n# as the scheduler operates in UTC.\nrun_in_delta = datetime.timedelta(minutes=5)\njob_in = enqueue_in_sync(run_in_delta, count_words, \"Job scheduled with a delay from now (UTC).\")\nprint(f\"Job {job_in.job_id} scheduled to run in {run_in_delta}\")\n```\n\n### Handling Timezone-Naive Datetimes\n\nIf you provide a timezone-naive `datetime` object (one without `tzinfo` set) to scheduling functions like `enqueue_at` or `enqueue_in`, `naq` will treat it as **UTC**.\n\n**Warning:** Relying on timezone-naive datetimes can lead to unexpected behavior if your system or the environment where the scheduler/worker runs has a different local timezone, or if daylight saving time changes occur. It's always safer to be explicit.\n\n```python\n# schedule_example_naive.py\nimport datetime\nfrom naq import enqueue_at_sync\nfrom tasks import count_words\n\n# This datetime is naive (no timezone info)\n# naq will interpret this as UTC.\nnaive_run_time = datetime.datetime(2024, 12, 25, 10, 30, 0)\n\njob_naive = enqueue_at_sync(naive_run_time, count_words, \"Job scheduled with a naive datetime (treated as UTC).\")\nprint(f\"Job {job_naive.job_id} scheduled for {naive_run_time} (interpreted as UTC).\")\n```\n\n### Best Practices\n\n1.  **Always Use UTC for Scheduling:** When creating `datetime` objects for scheduling, always make them timezone-aware with UTC.\n2.  **Convert Local Time to UTC:** If you have a local time that you want to schedule a job for, first convert it to UTC before passing it to `naq`.\n\n    ```python\n    import datetime\n    from naq import enqueue_at_sync\n\n    # Example: Scheduling for 9 AM Berlin time\n    local_time_str = \"2024-12-25 09:00:00\"\n    berlin_tz = datetime.timezone(datetime.timedelta(hours=1), name=\"CET\") # CET is UTC+1 in winter\n\n    # Parse the local time as timezone-aware\n    local_dt = datetime.datetime.fromisoformat(local_time_str).replace(tzinfo=berlin_tz)\n\n    # Convert to UTC before scheduling\n    utc_dt = local_dt.astimezone(datetime.timezone.utc)\n\n    # job = enqueue_at_sync(utc_dt, my_function, ...)\n    print(f\"Scheduled for {local_dt} (Berlin) which is {utc_dt} (UTC)\")\n    ```\n3.  **Store and Display in Local Time (Optional):** If your application needs to display scheduled times to users in their local timezone, perform the conversion from UTC to the user's local timezone at the display layer, not during scheduling.\n\nBy following these guidelines, you can avoid common pitfalls related to timezones and ensure your jobs run exactly when you expect them to.\n\n## Features\n\n*   Simple API similar to RQ.\n*   Asynchronous core using `asyncio` and `nats-py`.\n*   Job persistence via NATS JetStream streams.\n*   Support for scheduled jobs (run at a specific time or after a delay).\n*   Support for recurring jobs (cron-style or interval-based).\n*   Job dependencies (run a job only after others complete).\n*   Job retries with configurable backoff.\n*   Result backend using NATS KV store (with TTL).\n*   Worker monitoring and heartbeating using NATS KV store.\n*   High Availability for the scheduler process via leader election.\n*   Optional web dashboard (requires `naq[dashboard]`).\n*   Command-line interface (`naq`) for workers, scheduler, queue management, and dashboard.\n## Installation\n\nInstall `naq` using pip:\n\n```bash\npip install naq\n```\n\nTo include the optional web dashboard dependencies (Sanic, Jinja2, Datastar):\n```bash\npip install naq[dashboard]\n```\n\nYou also need a running NATS server with JetStream enabled. You can easily start one using the provided Docker Compose file:\n```bash\ncd docker\ndocker-compose up -d\n```\n\n## Basic Usage\n### 1. Define your function\n\n```python\n# tasks.py\nimport time\n\ndef count_words(text):\n    print(f\"Counting words in: '{text[:20]}...'\")\n    time.sleep(1) # Simulate work\n    count = len(text.split())\n    print(f\"Word count: {count}\")\n    return count\n```\n\n### 2. Enqueue the job\n```python\n# main.py\nfrom naq import enqueue_sync\nfrom tasks import count_words\n\nprint(\"Enqueuing job...\")\n# Enqueue synchronously (blocks until published)\njob = enqueue_sync(count_words, \"This is a sample text with several words.\")\n\nprint(f\"Job {job.job_id} enqueued.\")\nprint(\"Run 'naq worker default' to process it.\")\n```\n\n### 3. Run the worker:\n\nOpen a terminal and run the `naq` worker, telling it which queue(s) to listen to (default is `naq_default_queue`, often shortened to `default` in examples):\n```bash\nnaq worker default\n```\n\nThe worker will pick up the job and execute the `count_words` function.\n\n### 4. Scheduling Jobs:\n\nJobs can be scheduled to run later.\n\n```python\n# schedule_example.py\nimport datetime\nfrom naq import enqueue_at_sync, enqueue_in_sync\nfrom tasks import count_words\n\nnow = datetime.datetime.now(datetime.timezone.utc)\nrun_at = now + datetime.timedelta(seconds=10)\nrun_in = datetime.timedelta(minutes=1)\n\n# Schedule to run at a specific time (UTC recommended)\njob_at = enqueue_at_sync(run_at, count_words, \"Job scheduled for a specific time.\")\nprint(f\"Job {job_at.job_id} scheduled for {run_at}\")\n\n# Schedule to run after a delay\njob_in = enqueue_in_sync(run_in, count_words, \"Job scheduled after a delay.\")\nprint(f\"Job {job_in.job_id} scheduled to run in {run_in}\")\n\nprint(\"Run 'naq scheduler' and 'naq worker default' to process scheduled jobs.\")\n````\n\n5. Run the Scheduler\nFor scheduled jobs (`enqueue_at`, `enqueue_in`, `schedule`), you also need to run the `naq` scheduler process:\n```bash\nnaq scheduler\n```\nThe scheduler monitors scheduled jobs and enqueues them onto the appropriate queue when they are due.\n\n## Efficient connection handling and batching\n\nSynchronous producers (CLI tools, scripts, web handlers) often enqueue many jobs in quick succession. Reconnecting to NATS for every call can severely degrade performance. naq provides optimized connection reuse for both async and sync paths.\n\n- Async producers:\n  - Reuse a Queue instance for batching:\n    ```python\n    import asyncio\n    from naq.queue import Queue\n\n    async def produce(url: str):\n        q = Queue(nats_url=url)\n        for i in range(1000):\n            await q.enqueue(my_func, i)\n        await q.close()\n    ```\n  - The async path uses a process-wide pooled connection per URL.\n\n- Sync producers:\n  - Use enqueue_sync for simple scripts. It reuses a thread-local connection in the calling thread automatically:\n    ```python\n    from naq.queue import enqueue_sync, close_sync_connections\n\n    for i in range(1000):\n        enqueue_sync(my_func, i)\n\n    # Optionally close at the end of the batch\n    close_sync_connections()\n    ```\n  - All other sync helpers (enqueue_at_sync, enqueue_in_sync, schedule_sync, purge_queue_sync, etc.) reuse the same thread-local connection for efficiency.\n\nTrade-offs:\n- Thread-local reuse provides excellent performance for repeated calls from the same thread.\n- If you need maximal control or use multiple threads, manage Queue instances asynchronously and keep them alive across operations.\n\nCleanup:\n- Thread-local connections are cleaned up on process exit.\n- To end a batch sooner, call close_sync_connections() from the producing thread.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flegout%2Fnaq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flegout%2Fnaq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flegout%2Fnaq/lists"}