{"id":40216559,"url":"https://github.com/noverde/serpens","last_synced_at":"2026-05-25T18:01:15.638Z","repository":{"id":37082660,"uuid":"313109486","full_name":"noverde/serpens","owner":"noverde","description":"A set of Python utilities, recipes and snippets","archived":false,"fork":false,"pushed_at":"2025-10-09T19:06:44.000Z","size":267,"stargazers_count":2,"open_issues_count":14,"forks_count":6,"subscribers_count":14,"default_branch":"main","last_synced_at":"2026-03-06T08:58:48.089Z","etag":null,"topics":["aws","aws-sam","python","team-architecture"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/noverde-serpens/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/noverde.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-11-15T19:43:13.000Z","updated_at":"2025-10-09T19:05:57.000Z","dependencies_parsed_at":"2023-11-13T19:25:04.337Z","dependency_job_id":"4c773927-8a51-47b0-bf84-4b935e71206d","html_url":"https://github.com/noverde/serpens","commit_stats":null,"previous_names":[],"tags_count":71,"template":false,"template_full_name":null,"purl":"pkg:github/noverde/serpens","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noverde%2Fserpens","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noverde%2Fserpens/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noverde%2Fserpens/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noverde%2Fserpens/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/noverde","download_url":"https://codeload.github.com/noverde/serpens/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noverde%2Fserpens/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33486787,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-25T14:31:05.219Z","status":"ssl_error","status_checked_at":"2026-05-25T14:31:02.878Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-sam","python","team-architecture"],"created_at":"2026-01-19T21:37:23.456Z","updated_at":"2026-05-25T18:01:15.624Z","avatar_url":"https://github.com/noverde.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# serpens\n\nShared Python building blocks for DotzInc services: thin layers over SQLAlchemy\n2.0, Redis, httpx, Pub/Sub, SQS and friends. The lib stays out of the way —\napps import the underlying SDKs directly for the heavy lifting and use serpens\nfor the configuration that should not be duplicated across 24 repos.\n\n- [SQS](#sqs) · [Lambda API](#lambda-api) · [Schema](#schema) · [CSV](#csv)\n- [Database](#database) · [Migrations](#migrations) · [Pony → SQLAlchemy](#migrating-from-pony-orm)\n- [DynamoDB](#dynamodb) · [Async HTTP](#async-http-client) · [Rate limiter](#rate-limiter)\n- [Cache](#cache) · [Async Pub/Sub](#async-pub-sub-publisher) · [Test infra](#test-infrastructure-testgres)\n\n## SQS\n\n```python\nfrom serpens import sqs\n\n@sqs.handler\ndef message_processor(record: sqs.Record):\n    print(record.body)\n```\n\n`sqs.Record` exposes `data`, `body`, `message_attributes`, `queue_name`, `sent_datetime`.\n\n## Lambda API\n\n```python\nfrom serpens import api\n\n@api.handler\ndef lambda_handler(request: api.Request):\n    print(request.body)\n```\n\n`api.Request` exposes `authorizer`, `body`, `path`, `query`, `headers`, `identity`.\nAll but `body` are `AttrDict` — `request.path.user_id` shortcut for `request.path[\"user_id\"]`.\n\n### Async handlers (`api.async_handler`)\n\nSame contract as `@api.handler`, but for `async def` handlers — needed when\nthe body awaits coroutines (FastAPI/Mangum integration, async DB sessions,\nasync HTTP clients).\n\n```python\nfrom serpens import api\nfrom serpens.database import async_db_session\n\n@api.async_handler\nasync def lambda_handler(request: api.Request):\n    async with async_db_session() as sess:\n        ...\n```\n\n**Why use it**\n\n- **Single response/error contract.** `_build_response` and `_error_response`\n  are shared with the sync version: same response shape, same elastic-apm\n  capture, same JSON encoding via `SchemaEncoder`. No drift between sync\n  and async handler responses across services.\n- **Required to use async SQLAlchemy / `httpx` / `cache_async` inside a\n  Lambda.** A regular `@api.handler` cannot `await`.\n\n**Migration scenarios**\n\n| Today's code | Move to | What you gain |\n|---|---|---|\n| `@api.handler def lambda_handler(...)` that calls `asyncio.run(...)` internally | `@api.async_handler async def lambda_handler(...)` | One event loop per invocation, no `asyncio.run` boilerplate, can use async libs throughout the handler |\n| FastAPI / Mangum bridge with hand-rolled response shaping | `@api.async_handler` | Same response shape across sync/async Lambdas, central elastic-apm capture |\n\nThe sync `@api.handler` remains the right choice when the handler has no\nasync work.\n\n## Schema\n\nDataclass with static type checks and dict/JSON helpers.\n\n```python\nfrom dataclasses import dataclass\nfrom serpens.schema import Schema\n\n@dataclass\nclass Person(Schema):\n    name: str\n    age: int\n\nPerson.load({\"name\": \"Mike\", \"age\": 18})\nPerson.loads('{\"name\": \"Mike\", \"age\": 18}')\nPerson(\"Mike\", 18).dump()      # dict\nPerson(\"Mike\", 18).dumps()     # JSON string\n```\n\n## CSV\n\n```python\nfrom serpens import csvutils as csv\n\nfor row in csv.open_csv_reader(\"fruits.csv\"):\n    print(row)\n\nw = csv.open_csv_writer(\"out.csv\")\nw.writerow([\"id\", \"name\"]); w.writerow([\"1\", \"Açaí\"])\n```\n\n## Database\n\nThin layer over **SQLAlchemy 2.0**. Owns:\n\n- Engine setup with production defaults (Postgres `statement_timeout` /\n  `lock_timeout` / `idle_in_transaction_session_timeout`, Cloud SQL\n  keepalives, scheme normalization, `pool_pre_ping` / `pool_use_lifo`,\n  Lambda-aware tuning).\n- Session factories (`SessionLocal`, `AsyncSessionLocal`).\n- Declarative `Base` and `TimestampMixin`.\n- Alembic helper (see [Migrations](#migrations)).\n- Generic `Repository[T]` / `AsyncRepository[T]` covering the CRUD every\n  service used to re-implement.\n\nQuery construction stays in `sqlalchemy` proper — the lib does not re-export\n`select`, `Integer`, etc.\n\n### Why use it\n\n- **Production hardening baked in.** `statement_timeout`, `lock_timeout`,\n  `idle_in_transaction_session_timeout`, Cloud SQL keepalives, pool tuning\n  and `pool_pre_ping` apply automatically. Apps that hand-roll\n  `create_engine(...)` skip this; the lib makes it the default.\n- **One source of truth for engine config.** Pool size, recycle, LIFO\n  checkout, Postgres timeouts — all env-driven, consistent across services.\n  No more per-app drift on `max_overflow` or `pool_recycle`.\n- **Symmetric sync/async.** `bind` / `async_bind`, `SessionLocal` /\n  `AsyncSessionLocal`, `db_session` / `async_db_session`. Same mental\n  model on either side; mix freely.\n- **`Repository[T]` removes CRUD boilerplate.** PK lookup, filtered query,\n  paginate, add, bulk_add, `upsert` (Postgres `ON CONFLICT RETURNING`),\n  with deliberate gaps where services should diverge (no hard-delete, no\n  partial update — see recipes).\n- **Alembic glue.** `serpens.database.alembic.run_migrations(metadata)`\n  drives migrations from a Lambda or CLI with one line in `env.py`.\n\n### Migration scenarios\n\n| Today's code | Move to | What you gain |\n|---|---|---|\n| Hand-rolled `create_engine(...) + sessionmaker(...)` | `serpens.database.bind` + `SessionLocal` / `db_session` | Postgres timeouts, Cloud SQL keepalives, env-driven pool tuning, scheme normalization, `pool_pre_ping`, Lambda-aware defaults |\n| Pony ORM | `serpens.database` + SQLAlchemy 2.0 | Async support, typed `Mapped[...]` columns, Alembic instead of yoyo, the SA 2.0 ecosystem; see [Migrating from Pony ORM](#migrating-from-pony-orm) |\n| Per-service CRUD repositories (each app reimplements `get_by_id`, `list`, `paginate`) | `serpens.database.Repository[T]` / `AsyncRepository[T]` | Shared base, `upsert` primitive for idempotency, `NotFound` exception, free pagination |\n| Direct `redis.asyncio.Redis` + manual `aclose()` in DB-adjacent code | `serpens.database.async_bind` + `async_db_session` | Lifecycle helpers, `autoflush=False`, `expire_on_commit=False` sensible defaults |\n\n### Hello world (sync)\n\n```python\nfrom sqlalchemy import Integer, String, select\nfrom sqlalchemy.orm import Mapped, mapped_column\n\nfrom serpens.database import Base, SessionLocal, TimestampMixin, bind\n\n\nclass User(TimestampMixin, Base):\n    __tablename__ = \"users\"\n    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)\n    name: Mapped[str] = mapped_column(String, nullable=False)\n\n\nbind()  # reads DATABASE_URL\n\nwith SessionLocal() as sess:\n    sess.add(User(name=\"Ana\"))\n    sess.commit()\n```\n\n### Hello world (async)\n\n```python\nfrom sqlalchemy import select\nfrom serpens.database import async_bind, async_db_session\n\nasync_bind()  # reads DATABASE_URL, normalizes scheme to asyncpg\n\nasync def fetch(user_id: int):\n    async with async_db_session() as sess:\n        return (await sess.scalars(select(User).filter_by(id=user_id))).first()\n```\n\nAsync requires `asyncpg` (Postgres) or `aiosqlite` (SQLite).\n\n### Two ways to open a session\n\n**Explicit (preferred).** `SessionLocal()` / `AsyncSessionLocal()` — caller owns\ncommit/rollback. FastAPI handlers should use the per-request dependency:\n\n```python\nfrom fastapi import Depends\nfrom sqlalchemy.orm import Session\nfrom serpens.database import fastapi_session  # or fastapi_async_session\n\n@app.get(\"/users/{id}\")\ndef get_user(id: int, db: Session = Depends(fastapi_session)):\n    return db.scalars(select(User).filter_by(id=id)).first()\n```\n\n**Auto-managed.** `db_session()` / `async_db_session()` — context managers that\ncommit on success, roll back on exception and close always. Convenient for\nLambda handlers and short scripts. Pass the session explicitly to helpers:\n\n```python\nfrom serpens.database import async_db_session\n\nasync def fetch(sess, user_id: int):\n    return (await sess.scalars(select(User).filter_by(id=user_id))).first()\n\nasync with async_db_session() as sess:\n    user = await fetch(sess, 1)\n```\n\nThere is no `current_session()` global — sessions are passed.\n\n### Schema scoping\n\n```python\nfrom serpens.database import declarative_base\n\nBase = declarative_base(schema=\"public\")\n```\n\n### Configuration\n\n| Variable | Default | Purpose |\n|---|---|---|\n| `DATABASE_URL` | — | Connection string. `postgres://` and `postgresql://` are normalized to `postgresql+psycopg2://` (sync) / `postgresql+asyncpg://` (async). |\n| `APP_NAME` | `serpens` | Postgres `application_name`. Overridden by `K_SERVICE` (Cloud Run) or `AWS_LAMBDA_FUNCTION_NAME` (Lambda). |\n| `DB_POOL_SIZE` | `10` | Pool size. **Set to `1` on Lambda.** |\n| `DB_MAX_OVERFLOW` | `20` | Extra connections. **Set to `0` on Lambda.** |\n| `DB_POOL_TIMEOUT` | `10` | Seconds to wait for a free connection. |\n| `DB_POOL_RECYCLE` | `1800` | Recycle older connections. |\n| `DB_STATEMENT_TIMEOUT_MS` | `5000` | Postgres `statement_timeout`. |\n| `DB_LOCK_TIMEOUT_MS` | `2000` | Postgres `lock_timeout`. |\n| `DB_IDLE_IN_TX_TIMEOUT_MS` | `10000` | Postgres `idle_in_transaction_session_timeout`. |\n| `DB_POOL_USE_LIFO` | `true` | LIFO checkout (warm connections preferred). Set `false` for FIFO. |\n| `DB_ECHO` | `false` | Log every SQL statement. |\n\n`bind()` and `async_bind()` also accept `pool_use_lifo=True/False` as a direct\noverride of the env var.\n\nCloud SQL keepalives (`keepalives=1`, `keepalives_idle=30`,\n`keepalives_interval=10`, `keepalives_count=3`) are applied automatically on\nPostgres connections.\n\n### Bind from FastAPI lifespan, never at import time\n\n```python\nfrom contextlib import asynccontextmanager\nfrom fastapi import FastAPI\nfrom serpens.database import async_bind, async_dispose\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n    async_bind()\n    yield\n    await async_dispose()\n\napp = FastAPI(lifespan=lifespan)\n```\n\nCalling `bind()` at the top of `models.py` makes import order matter and\nbreaks under cold-start.\n\n### Repository (optional)\n\n`Repository[T]` (sync) and `AsyncRepository[T]` (async) cover the CRUD that\nevery service rewrites: PK lookups, filtered queries, paginate, add, upsert.\nSubclass with `model = X` and add your own methods for anything custom — the\n`query` property exposes a `Select(model)` you compose on. The base\nintentionally **does not** ship hard-delete or partial update: services do\nsoft-delete differently and updates often need optimistic locking.\n\n```python\nfrom serpens.database import AsyncRepository\n\nclass ProductRepo(AsyncRepository[Product]):\n    model = Product\n\n    async def by_slug(self, slug):                 # custom lookup\n        return await self.get_by(slug=slug)\n\nasync with async_db_session() as sess:\n    p = await ProductRepo(sess).by_slug(\"some-product\")\n```\n\nBuilt-in methods: `get`, `get_or_raise` (raises `serpens.database.NotFound`),\n`get_by`, `exists`, `count`, `list(order_by=, limit=, offset=, **filters)`,\n`paginate(stmt=, page=, size=)`, `add(obj, flush=True)`, `bulk_add(objs)`,\n`upsert(values, conflict_on=, update_fields=)`.\n\n#### Recipe: soft-delete\n\nDon't expose hard delete. Add a method on your repo:\n\n```python\nclass PaymentRepo(AsyncRepository[Payment]):\n    model = Payment\n\n    async def cancel(self, payment, *, reason: str):\n        payment.status = \"cancelled\"\n        payment.cancel_reason = reason\n        await self.sess.flush()\n```\n\n#### Recipe: optimistic locking\n\nDeclare `version_id_col` on the model — `Repository` doesn't fight it:\n\n```python\nclass Payment(TimestampMixin, Base):\n    __tablename__ = \"payments\"\n    id: Mapped[int] = mapped_column(Integer, primary_key=True)\n    amount: Mapped[Decimal] = mapped_column(Numeric)\n    version: Mapped[int] = mapped_column(Integer, nullable=False)\n    __mapper_args__ = {\"version_id_col\": version}\n```\n\nSA will raise `StaleDataError` on concurrent update. Catch it in the handler.\n\n#### Recipe: idempotent insert (`upsert`, not `get_or_create`)\n\n`get_or_create` has a race between SELECT and INSERT. Use `upsert` instead —\nPostgres `INSERT ... ON CONFLICT ... RETURNING`:\n\n```python\nclass IdempotentPaymentRepo(AsyncRepository[Payment]):\n    model = Payment\n\nawait IdempotentPaymentRepo(sess).upsert(\n    {\"external_id\": req.idempotency_key, \"amount\": req.amount, \"status\": \"received\"},\n    conflict_on=[\"external_id\"],\n    update_fields=[\"amount\"],   # omit to do nothing on conflict\n)\n```\n\n## Migrations\n\n**New repos use Alembic.** `serpens.migrations` (yoyo) is kept only for legacy\nservices that haven't migrated yet.\n\n`serpens.database.alembic.run_migrations` wires Alembic against your app's\n`Base.metadata` with sensible defaults (offline/online, scheme normalization to\npsycopg2, `NullPool` to keep migration jobs from leaking pool slots).\n\n`alembic/env.py`:\n\n```python\nfrom myapp.models import Base\nfrom serpens.database.alembic import run_migrations\n\nrun_migrations(target_metadata=Base.metadata)\n```\n\n`alembic.ini` (minimal):\n\n```ini\n[alembic]\nscript_location = alembic\nprepend_sys_path = .\nfile_template = %%(year)d%%(month).2d%%(day).2d_%%(rev)s_%%(slug)s\nsqlalchemy.url =\n```\n\nRun with `DATABASE_URL` set:\n\n```bash\nalembic revision -m \"add user table\" --autogenerate\nalembic upgrade head\n```\n\nFor a Lambda migration job (no `alembic.ini` in the package):\n\n```python\nimport os\nfrom alembic import command\nfrom alembic.config import Config\n\ndef migrate_handler(event, context):\n    cfg = Config()\n    cfg.set_main_option(\"script_location\", os.path.dirname(os.path.abspath(__file__)))\n    command.upgrade(cfg, \"head\")\n```\n\n## Migrating from Pony ORM\n\nPony lacks async, lacks typed `Mapped[T]`, and is in limited maintenance. The\nplatform standardised on SQLAlchemy 2.0 via `serpens.database`. **Use the async\nAPI by default.**\n\n### Mapping table\n\n| Pony | SQLAlchemy 2.0 + serpens |\n|---|---|\n| `class X(db.Entity):` | `class X(TimestampMixin, Base):` |\n| `name = Required(str)` | `name: Mapped[str] = mapped_column(String, nullable=False)` |\n| `email = Optional(str)` | `email: Mapped[str \\| None] = mapped_column(String, nullable=True)` |\n| `created_at = Required(datetime)` | inherit `TimestampMixin` |\n| `loans = Set(lambda: Loan)` | `loans: Mapped[list[\"Loan\"]] = relationship(back_populates=\"user\")` |\n| `composite_key(a, b)` | `__table_args__ = (UniqueConstraint(\"a\", \"b\"),)` |\n| `_table_ = (\"public\", \"users\")` | `__tablename__ = \"users\"; __table_args__ = {\"schema\": \"public\"}` |\n| `@db_session` decorator | `with db_session()` block or pass `Session` |\n| `X(field=value)` (auto-flush) | `obj = X(...); sess.add(obj); sess.flush()` |\n| `X.get(field=value)` | `sess.scalars(select(X).filter_by(field=value)).first()` |\n| `X.select(...).order_by(X.id)[:10]` | `sess.scalars(select(X).order_by(X.id).limit(10)).all()` |\n| `X.select_by_sql(\"SELECT ...\", params)` | `sess.scalars(select(X).from_statement(text(\"SELECT ...\"))).all()` |\n| `db.generate_mapping(create_tables=True)` | `Base.metadata.create_all(engine)` |\n\n### Per-file recipe\n\nBranch from `staging` → `feat/migrate-pony-to-sqlalchemy`.\n\n1. **`requirements.txt`** — bump `noverde-serpens`, ensure `SQLAlchemy\u003e=2.0`,\n   add `asyncpg` (for async) and/or `psycopg2-binary` (for sync + Alembic),\n   drop `pony` and `yoyo-migrations`.\n2. **Models** — replace `db = Database()` and `class X(db.Entity)`. Methods\n   stuck to the entity (`X.get_by_slug`, `X.create`) become module-level\n   functions taking `Session` / `AsyncSession` as first argument:\n   ```python\n   async def get_product_by_slug(sess: AsyncSession, slug: str) -\u003e Product | None:\n       return (await sess.scalars(select(Product).filter_by(slug=slug))).first()\n   ```\n3. **Handlers** — open `async with async_db_session() as sess:` and pass `sess`\n   down. In FastAPI routes use `Depends(fastapi_async_session)`.\n4. **`main.py`** — `async_bind()` runs in a FastAPI `lifespan`, never at\n   import time.\n5. **Tests** — `setUp` uses `async with async_db_session() as sess: sess.add(...)`.\n   `tearDown` uses `await sess.execute(delete(...))`. `testgres.setup(Base)`\n   still works.\n6. **Migrations** — yoyo → Alembic. Add `alembic.ini`, `alembic/env.py`\n   delegating to `serpens.database.alembic.run_migrations`, and a baseline\n   revision wrapping the existing schema with `IF NOT EXISTS`. Lambda\n   `Migrate` switches to `Handler: migrate_handler.migrate_handler`. Run\n   `alembic stamp 0001_baseline` once per environment.\n\n### Common gotchas\n\n- **Optimistic lock changes**. Pony locks read rows by default; SA 2.0 does\n  not. If a job relied on it, opt back in with `version_id_col` on the model.\n- **`X(...)` does not INSERT in SA 2.0**. Use `sess.add(obj)` and, if you need\n  `obj.id` populated, `sess.flush()`.\n- **`autoflush=False`**. Serpens disables autoflush so a stray `select`\n  doesn't flush pending changes. Call `sess.flush()` explicitly when needed.\n- **Lambda pool**: `DB_POOL_SIZE=1`, `DB_MAX_OVERFLOW=0`. A larger pool\n  causes Cloud SQL churn under burst.\n- **`postgres://` vs `postgresql://`**: SA 2.0 dropped the short prefix.\n  serpens normalises both.\n- **Schema declaration**: pick one place — `declarative_base(schema=...)`\n  centralized, or `__table_args__={\"schema\":...}` per model. Don't mix.\n\n### When NOT to use serpens.database\n\n- The repo already has an idiomatic SA 2.0 `SessionLocal`. Don't migrate\n  just for standardization.\n- You need an SA feature serpens does not expose — import from `sqlalchemy`\n  directly. Serpens is a thin layer by design.\n\n## DynamoDB\n\n```python\nfrom dataclasses import dataclass\nfrom serpens.document import BaseDocument\n\n@dataclass\nclass PersonDocument(BaseDocument):\n    _table_name_ = \"person\"\n    id: str\n    name: str\n\nPersonDocument(id=\"1\", name=\"Ana\").save()\nPersonDocument.get_by_key({\"id\": \"1\"})\nPersonDocument.get_table()\n```\n\n## Async HTTP client\n\nSingleton `httpx.AsyncClient` — connection pools survive across requests.\n\n```python\nfrom contextlib import asynccontextmanager\nfrom fastapi import Depends, FastAPI\nfrom httpx import AsyncClient\nfrom serpens.http_client import close_client, get_client, init_client\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n    await init_client()\n    yield\n    await close_client()\n\napp = FastAPI(lifespan=lifespan)\n\n@app.get(\"/proxy\")\nasync def proxy(client: AsyncClient = Depends(get_client)):\n    return (await client.get(\"https://example.com\")).json()\n```\n\nTimeout defaults to `HTTP_CLIENT_TIMEOUT` (env, seconds) or 30s. Extra kwargs\npass through to `httpx.AsyncClient`.\n\n### Why use it\n\n- **Pool reuse across requests.** A new `AsyncClient(...)` per request\n  creates and tears down TCP+TLS connections every call. The singleton\n  amortises the handshake across the lifetime of the process — material\n  latency win on chatty integrations.\n- **One lifecycle to wire.** `init_client` / `close_client` plug into\n  FastAPI `lifespan` (or `startup`/`shutdown` for older versions). No\n  per-handler instantiation boilerplate.\n- **Env-driven timeout.** `HTTP_CLIENT_TIMEOUT` standardises the cap;\n  per-service overrides through the same channel.\n\n### Migration scenarios\n\n| Today's code | Move to | What you gain |\n|---|---|---|\n| `async with httpx.AsyncClient() as client: await client.get(...)` per call | `init_client()` in lifespan + `get_client()` in handlers | Pool reuse, lower TCP/TLS overhead, central timeout |\n| `requests.get(...)` (sync) inside a FastAPI handler | `init_client()` + `await client.get(...)` | Stops blocking the event loop for the duration of the call |\n| Per-service `aiohttp` / `httpx` singleton with hand-rolled lifecycle | `serpens.http_client` | Shared implementation, one less file per repo |\n\nGreenfield FastAPI services adopt this from day one; existing async services\nswap a couple of imports.\n\n## Rate limiter\n\nToken-bucket limiter for outbound calls plus an `auth_lock` that serializes\ntoken refresh (avoids thundering-herd re-auths on expiry).\n\n```python\nfrom serpens.rate_limit import RateLimiter\n\nlimiter = RateLimiter(rate=20, per_seconds=1.0)\n\n@asynccontextmanager\nasync def lifespan(_app):\n    limiter.start()\n    yield\n    await limiter.stop()\n\nasync def call_external():\n    await limiter.acquire()\n    return await client.get(...)\n\nasync def fetch_token():\n    async with limiter.auth_lock:\n        return await cached_get_or_set(\"token\", 1800, _refresh_token)\n```\n\n### Why use it\n\n- **Respects upstream quotas without manual back-off.** Most third-party\n  APIs (banks, KYC providers, credit bureaus) cap requests-per-second;\n  exceeding it triggers 429s or temporary blocks. Token bucket gives a\n  smooth, predictable throughput at the configured ceiling.\n- **`auth_lock` solves the thundering herd.** When a JWT/OAuth token\n  expires, every concurrent coroutine tries to refresh at the same time.\n  The lock guarantees one refresh per expiry while the rest wait on it.\n- **Asyncio-native.** No third-party `aiolimiter` dependency; replenisher\n  runs as an `asyncio.Task` you control via `start`/`stop`.\n\n### Migration scenarios\n\n| Today's code | Move to | What you gain |\n|---|---|---|\n| `asyncio.Semaphore(N)` hand-rolled to cap concurrency | `RateLimiter(rate=N, per_seconds=...)` | Time-based replenishment instead of pure concurrency cap; predictable RPS |\n| `aiolimiter` / external rate-limit lib | `serpens.rate_limit` | One less dep; same primitive |\n| No rate limiting at all (calls 3rd-party until 429) | `serpens.rate_limit` | Stops invalidating provider relationships and triggering exponential back-off cascades |\n| Hand-rolled `asyncio.Lock` around token refresh | `limiter.auth_lock` | Bundled with the rate limit; less wiring |\n\nTypical fit: any FastAPI / async service calling a quoted upstream\n(banking, KYC, payment processor). Worth adopting alongside\n`serpens.http_client` since they cover the same call path.\n\n## Cache\n\n`serpens.cache` ships three flavors in a single module. Pick by sync/async\nand by scope (process-local vs distributed).\n\n### Why use it\n\n- **Fails open.** A Redis outage degrades to \"no cache\" instead of crashing\n  the caller. Reads return `None` (treated as miss), writes/deletes become\n  no-ops, decorators fall through to the wrapped function. Each failure\n  logs a warning. Most plain `redis.asyncio.Redis` wrappers don't trap\n  connection errors and propagate them to handlers.\n- **Single source of truth.** Stops the per-service drift (in-process TTL\n  caches reimplemented in each repo, Redis lifecycle wired by hand, etc.).\n- **Symmetric APIs.** Sync, async in-process and async Redis share the\n  same mental model: decorator-based caching plus low-level get/set/delete.\n- **Lifecycle helpers built in** for the Redis flavor — `redis_init` /\n  `redis_close` for module-level singleton usage, `redis_pool` for FastAPI\n  `Depends` injection.\n- **JSON serialization for free** on `redis_get` / `redis_set` /\n  `redis_cached`; raw bytes are still available through `redis_pool`\n  when needed.\n\n### Migration scenarios\n\n| Today's code | Move to | What you gain |\n|---|---|---|\n| Third-party Redis `Depends` factory (callable yielding `redis.asyncio.Redis`) | `serpens.cache.redis_pool` | Fail-open client on Redis outage; same `Depends` contract, one-line swap |\n| App-local `acached` / in-process async TTL cache | `serpens.cache.acached` | One implementation maintained centrally; monotonic-clock TTL; same `self`-aware key heuristic |\n| Direct `Redis.from_url(...)` + manual `aclose()` in Lambda | `serpens.cache.redis_init` / `redis_close` / `redis_get` / `redis_set` / `redis_cached` | Lifecycle helpers, auto JSON serialization, fail-open, env-driven prefix/TTL |\n| `serpens.cache.cached` (sync legacy) | unchanged | Already lives here; consumed by `parameters` and `secrets_manager` |\n\nThe migration is intentionally minimal — typically a single import line per app. The behavior gain (Redis outages no longer break callers) is automatic on the new APIs.\n\n### Sync, in-process (legacy)\n\nUsed by `serpens.parameters`, `serpens.secrets_manager` and downstream\nservices. TTL bucketed by name.\n\n```python\nfrom serpens.cache import cached, clear_cache\n\n@cached(\"secrets_manager\", 900)\ndef get(secret_id):\n    ...\n\nclear_cache(\"secrets_manager\")\n```\n\n### Async, in-process\n\n`acached` / `clear_acache` — same idea, for `async def` callers. The\ndecorator drops the first positional argument from the key, on the\nassumption it's `self` (a repository or service object pointing at the\nsame store). Different instances therefore share entries — fine for\nread-mostly data. Uses `time.monotonic` so TTL is immune to clock\nadjustments.\n\n```python\nfrom serpens.cache import acached, clear_acache\n\nclass ProductRepo:\n    @acached(\"products\", ttl_seconds=600)\n    async def get_by_slug(self, slug: str):\n        return await self.session.scalar(select(Product).where(Product.slug == slug))\n\nclear_acache(\"products\")  # one bucket\nclear_acache()            # everything\n```\n\n### Async, Redis-backed\n\nFor FastAPI / long-running services that need a cache shared across\nworkers and instances. Lifecycle: `redis_init` once at startup,\n`redis_close` at shutdown.\n\n**Fails open.** On `RedisError` (host unreachable, timeout, refused\nconnection) reads return `None` (treated as miss), writes/deletes become\nno-ops, and `redis_cached_get_or_set` falls through to the wrapped\nfunction. Each failure logs a warning. Programming errors (using the\nclient before `redis_init`) still raise `RuntimeError`.\n\n```python\nfrom serpens.cache import (\n    redis_init, redis_close, redis_get, redis_set, redis_delete,\n    redis_cached, redis_cached_get_or_set,\n)\n\n@asynccontextmanager\nasync def lifespan(_app):\n    await redis_init()\n    yield\n    await redis_close()\n\nawait redis_set(\"user:42\", {\"name\": \"Ana\"}, ttl=60)\nuser = await redis_get(\"user:42\")\n\n@redis_cached(\"products\", ttl=600)\nasync def get_product(slug: str):\n    return await fetch_product(slug)\n```\n\n| Variable | Default | Purpose |\n|---|---|---|\n| `REDIS_URL` | — | Redis connection string. |\n| `CACHE_PREFIX` | `serpens` | Prefix prepended to every key. Set per-service. |\n| `CACHE_TTL` | `300` | Default TTL for `redis_set` / `redis_cached`. |\n\n#### FastAPI `Depends` style\n\n`redis_pool(url)` returns a callable suitable for FastAPI `Depends`,\nyielding a fail-open Redis client per request. The same fail-open\nsemantics apply: `get` returns `None`, `set`/`delete` no-op on\n`RedisError`.\n\n```python\nfrom fastapi import Depends, FastAPI\nfrom redis.asyncio import Redis\nfrom serpens.cache import redis_pool\n\napp = FastAPI()\ncache = redis_pool(settings.REDIS_URL)\n\n@app.get(\"/users/{user_id}\")\nasync def get_user(user_id: str, client: Redis = Depends(cache)):\n    return await client.get(f\"user:{user_id}\")\n```\n\nUse this when the rest of the app expects a `redis.asyncio.Redis`\nclient (e.g. when wiring third-party libraries that take `cache_gen`).\nFor Lambda / single-process apps, the `redis_*` module-level functions\nabove are simpler.\n\n#### In tests\n\n`testgres.setup(Base, redis_mode=True)` spins a Redis container alongside\nPostgres and exports `REDIS_URL` — `redis_init()` picks it up without\nfurther config. If `REDIS_URL` is already set, the existing instance is\nreused.\n\n## Async Pub/Sub publisher\n\n`serpens.pubsub.AsyncPublisher` wraps the sync Google SDK with\n`asyncio.wrap_future` so `await publish(...)` does not block the event loop.\nInstantiate once per process in `lifespan`, close on shutdown.\n\n`topic` is the full topic id (`projects/PROJECT/topics/NAME`) — the same value\nTerraform exposes as an env var, no need to rebuild it via\n`client.topic_path(...)`. When `elasticapm` is installed, every publish emits a\nmessaging span labeled with the topic.\n\n```python\nfrom serpens.pubsub import AsyncPublisher\n\npublisher = AsyncPublisher()\n\n@asynccontextmanager\nasync def lifespan(_app):\n    yield\n    publisher.close()\n\nasync def emit(payload: dict):\n    await publisher.publish(settings.MY_TOPIC, payload)\n```\n\n### Why use it\n\n- **Doesn't block the event loop.** The Google SDK is synchronous (returns\n  a `concurrent.futures.Future`). Without `asyncio.wrap_future`, awaiting\n  a publish in a FastAPI handler stalls every other request on the same\n  worker. `AsyncPublisher` bridges the gap.\n- **Terraform-friendly topic id.** Accepts `projects/.../topics/...`\n  directly — same value already exposed as `MY_TOPIC` env in your\n  Terraform module. No need to keep `project_id` separately and call\n  `client.topic_path(project, topic)` everywhere.\n- **APM observability for free.** When `elasticapm` is installed, every\n  publish emits a `messaging` span with `queue_name=\u003ctopic\u003e`. No-op if\n  APM isn't present.\n- **Single connection per process.** The client is instantiated once on\n  app boot; gRPC channel reuse cuts per-publish overhead.\n\n### Migration scenarios\n\n| Today's code | Move to | What you gain |\n|---|---|---|\n| `pubsub_v1.PublisherClient()` + `client.topic_path(project, topic)` + `future.result()` per publish | `AsyncPublisher()` + `await publisher.publish(topic, payload)` | Non-blocking publish, full topic id, central APM span emission, one client per process |\n| `serpens.pubsub.publish_message(...)` (sync, creates a new `PublisherClient` per call) inside an async handler | `AsyncPublisher` | Avoids the per-call client construction; no event loop blocking |\n| Per-service `TracedMessagePublisher` / APM-aware wrapper | `AsyncPublisher` | Removes the duplicated wrapper; spans emitted from the lib |\n\nThe sync `serpens.pubsub.publish_message` / `publish_message_batch` remain\nthe right choice for Lambda one-shots or non-async code paths.\n\n## Test infrastructure (testgres)\n\n`serpens.testgres.setup` wires a Postgres (and optionally Redis) container\ninto a `unittest`/`pytest` suite, running `create_all` against your\n`Base.metadata` before the first test.\n\n```python\n# conftest.py\nfrom serpens import testgres\nfrom myapp.models import Base\n\ntestgres.setup(Base, async_mode=True, redis_mode=True)\n```\n\n### Modes\n\n| Flag | Effect | When to enable |\n|---|---|---|\n| (default) | Spins a Postgres container, binds `database.SessionLocal` (sync) | Lambda services / sync codebases |\n| `async_mode=True` | Also binds `database.AsyncSessionLocal` with `NullPool` | FastAPI services / async tests; remove the per-conftest async engine wiring |\n| `redis_mode=True` | Spins a Redis container alongside Postgres, exports `REDIS_URL` | Services using `serpens.cache.redis_*` or any Redis client; replaces the manual `docker run redis` boilerplate teams currently keep in their own `conftest.py` |\n| `default_schema=\"x,y\"` | Pre-creates the schemas and sets `search_path` | Services using schema scoping via `declarative_base(schema=...)` |\n| `uri=...` / `DATABASE_URL` env | Skips the container, uses the provided URI | CI runners that already have Postgres available |\n| `REDIS_URL` env (when `redis_mode=True`) | Skips the Redis container, uses the provided URL | CI runners with existing Redis |\n\n### Why use it\n\n- **One conftest line replaces a dozen.** Tests that previously wired\n  `create_engine`, `metadata.create_all`, `sessionmaker`, container\n  lifecycle, schema setup and Redis container setup collapse to a single\n  `setup(...)` call.\n- **Real Postgres in tests.** Caught the kind of bug SQLite can't model\n  (`ON CONFLICT`, `JSONB` operators, schema-qualified names). Same engine\n  semantics as production.\n- **Async + sync engines wired together.** Both `SessionLocal` and\n  `AsyncSessionLocal` point at the same database — tests can mix.\n- **Defers `create_all` to `startTestRun`.** Models registered after\n  `setup()` returns (common when tests do dynamic imports) still get\n  their tables.\n- **Graceful failure.** Waits for the published TCP port and a real\n  `psycopg2.connect`, raising a clear `RuntimeError` if the container\n  doesn't come up — no more silent test hangs.\n\n### Migration scenarios\n\n| Today's code | Move to | What you gain |\n|---|---|---|\n| Hand-rolled `docker run postgres:13` in test setup + manual `create_engine` + `metadata.create_all` | `serpens.testgres.setup(Base)` | Container lifecycle, schema bootstrap, sane defaults, error propagation |\n| Test suite that wires async engine separately from sync (parallel `AsyncSessionLocal` setup in `conftest.py`) | `setup(Base, async_mode=True)` | One factory wired automatically with `NullPool` (correct for tests) |\n| `docker run postgres` + `docker run redis` boilerplate in `conftest.py` | `setup(Base, redis_mode=True)` | One call, both containers, env vars exported |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoverde%2Fserpens","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnoverde%2Fserpens","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoverde%2Fserpens/lists"}