{"id":18295506,"url":"https://github.com/riverqueue/riverqueue-python","last_synced_at":"2025-04-05T12:31:33.808Z","repository":{"id":246997687,"uuid":"822850910","full_name":"riverqueue/riverqueue-python","owner":"riverqueue","description":"Python insert-only client for River.","archived":false,"fork":false,"pushed_at":"2025-02-27T15:58:00.000Z","size":142,"stargazers_count":9,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-21T04:01:48.575Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://riverqueue.com/docs","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/riverqueue.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-02T00:46:35.000Z","updated_at":"2025-03-08T07:08:16.000Z","dependencies_parsed_at":"2024-07-06T04:13:09.447Z","dependency_job_id":"21329bf5-128c-4463-af25-a87a6e9f5157","html_url":"https://github.com/riverqueue/riverqueue-python","commit_stats":null,"previous_names":["riverqueue/riverqueue-python"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riverqueue%2Friverqueue-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riverqueue%2Friverqueue-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riverqueue%2Friverqueue-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riverqueue%2Friverqueue-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/riverqueue","download_url":"https://codeload.github.com/riverqueue/riverqueue-python/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247338860,"owners_count":20922998,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T14:36:03.023Z","updated_at":"2025-04-05T12:31:28.794Z","avatar_url":"https://github.com/riverqueue.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# River client for Python\n\nAn insert-only Python client for [River](https://github.com/riverqueue/river) packaged in the [`riverqueue` package on PyPI](https://pypi.org/project/riverqueue/). Allows jobs to be inserted in Python and run by a Go worker, but doesn't support working jobs in Python.\n\n## Basic usage\n\nYour project should bundle the [`riverqueue` package](https://pypi.org/project/riverqueue/) in its dependencies. How to go about this will depend on your toolchain, but for example in [Rye](https://github.com/astral-sh/rye), it'd look like:\n\n```shell\nrye add riverqueue\n```\n\nInitialize a client with:\n\n```python\nimport riverqueue\nfrom riverqueue.driver import riversqlalchemy\n\nengine = sqlalchemy.create_engine(\"postgresql://...\")\nclient = riverqueue.Client(riversqlalchemy.Driver(engine))\n```\n\nDefine a job and insert it:\n\n```python\n@dataclass\nclass SortArgs:\n    strings: list[str]\n\n    kind: str = \"sort\"\n\n    def to_json(self) -\u003e str:\n        return json.dumps({\"strings\": self.strings})\n\ninsert_res = client.insert(\n    SortArgs(strings=[\"whale\", \"tiger\", \"bear\"]),\n)\ninsert_res.job # inserted job row\n```\n\nJob args should comply with the `riverqueue.JobArgs` [protocol](https://peps.python.org/pep-0544/):\n\n```python\nclass JobArgs(Protocol):\n    kind: str\n\n    def to_json(self) -\u003e str:\n        pass\n```\n\n* `kind` is a unique string that identifies them the job in the database, and which a Go worker will recognize.\n* `to_json()` defines how the job will serialize to JSON, which of course will have to be parseable as an object in Go.\n\nThey may also respond to `insert_opts()` with an instance of `InsertOpts` to define insertion options that'll be used for all jobs of the kind.\n\nWe recommend using [`dataclasses`](https://docs.python.org/3/library/dataclasses.html) for job args since they should ideally be minimal sets of primitive properties with little other embellishment, and `dataclasses` provide a succinct way of accomplishing this.\n\n## Insertion options\n\nInserts take an `insert_opts` parameter to customize features of the inserted job:\n\n```python\ninsert_res = client.insert(\n    SortArgs(strings=[\"whale\", \"tiger\", \"bear\"]),\n    insert_opts=riverqueue.InsertOpts(\n        max_attempts=17,\n        priority=3,\n        queue=\"my_queue\",\n        tags=[\"custom\"]\n    ),\n)\n```\n\n## Inserting unique jobs\n\n[Unique jobs](https://riverqueue.com/docs/unique-jobs) are supported through `InsertOpts.unique_opts()`, and can be made unique by args, period, queue, and state. If a job matching unique properties is found on insert, the insert is skipped and the existing job returned.\n\n```python\ninsert_res = client.insert(\n    SortArgs(strings=[\"whale\", \"tiger\", \"bear\"]),\n    insert_opts=riverqueue.InsertOpts(\n        unique_opts=riverqueue.UniqueOpts(\n            by_args=True,\n            by_period=15*60,\n            by_queue=True,\n            by_state=[riverqueue.JobState.AVAILABLE]\n        )\n    ),\n)\n\n# contains either a newly inserted job, or an existing one if insertion was skipped\ninsert_res.job\n\n# true if insertion was skipped\ninsert_res.unique_skipped_as_duplicated\n```\n\n### Custom advisory lock prefix\n\nUnique job insertion takes a Postgres advisory lock to make sure that its uniqueness check still works even if two conflicting insert operations are occurring in parallel. Postgres advisory locks share a global 64-bit namespace, which is a large enough space that it's unlikely for two advisory locks to ever conflict, but to _guarantee_ that River's advisory locks never interfere with an application's, River can be configured with a 32-bit advisory lock prefix which it will use for all its locks:\n\n```python\nclient = riverqueue.Client(riversqlalchemy.Driver(engine), advisory_lock_prefix=123456)\n```\n\nDoing so has the downside of leaving only 32 bits for River's locks (64 bits total - 32-bit prefix), making them somewhat more likely to conflict with each other.\n\n## Inserting jobs in bulk\n\nUse `#insert_many` to bulk insert jobs as a single operation for improved efficiency:\n\n```python\nnum_inserted = client.insert_many([\n    SimpleArgs(job_num=1),\n    SimpleArgs(job_num=2)\n])\n```\n\nOr with `InsertManyParams`, which may include insertion options:\n\n```python\nnum_inserted = client.insert_many([\n    InsertManyParams(args=SimpleArgs(job_num=1), insert_opts=riverqueue.InsertOpts(max_attempts=5)),\n    InsertManyParams(args=SimpleArgs(job_num=2), insert_opts=riverqueue.InsertOpts(queue=\"high_priority\"))\n])\n```\n\n## Inserting in a transaction\n\nTo insert jobs in a transaction, open one in your driver, and pass it as the first argument to `insert_tx()` or `insert_many_tx()`:\n\n```python\nwith engine.begin() as session:\n    insert_res = client.insert_tx(\n        session,\n        SortArgs(strings=[\"whale\", \"tiger\", \"bear\"]),\n    )\n```\n\n## Asynchronous I/O (asyncio)\n\nThe package supports River's [`asyncio` (asynchronous I/O)](https://docs.python.org/3/library/asyncio.html) through an alternate `AsyncClient` and `riversqlalchemy.AsyncDriver`. You'll need to make sure to use SQLAlchemy's alternative async engine and an asynchronous Postgres driver like [`asyncpg`](https://github.com/MagicStack/asyncpg), but otherwise usage looks very similar to use without async:\n\n```python\nengine = sqlalchemy.ext.asyncio.create_async_engine(\"postgresql+asyncpg://...\")\nclient = riverqueue.AsyncClient(riversqlalchemy.AsyncDriver(engine))\n\ninsert_res = await client.insert(\n    SortArgs(strings=[\"whale\", \"tiger\", \"bear\"]),\n)\n```\n\nWith a transaction:\n\n```python\nasync with engine.begin() as session:\n    insert_res = await client.insert_tx(\n        session,\n        SortArgs(strings=[\"whale\", \"tiger\", \"bear\"]),\n    )\n```\n\n## MyPy and type checking\n\nThe package exports a `py.typed` file to indicate that it's typed, so you should be able to use [MyPy](https://mypy-lang.org/) to include it in static analysis.\n\n## Drivers\n\n### SQLAlchemy\n\nOur read is that [SQLAlchemy](https://www.sqlalchemy.org/) is the dominant ORM in the Python ecosystem, so it's the only driver available for River. Under the hood of SQLAlchemy, projects will also need a Postgres driver like [`psycopg2`](https://pypi.org/project/psycopg2/) or [`asyncpg`](https://github.com/MagicStack/asyncpg) (for async).\n\nRiver's driver system should enable integration with other ORMs, so let us know if there's a good reason you need one, and we'll consider it.\n\n## Development\n\nSee [development](./docs/development.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friverqueue%2Friverqueue-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Friverqueue%2Friverqueue-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friverqueue%2Friverqueue-python/lists"}