{"id":50483315,"url":"https://github.com/prisma/streams","last_synced_at":"2026-06-01T19:31:04.094Z","repository":{"id":345034015,"uuid":"1183973373","full_name":"prisma/streams","owner":"prisma","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-28T11:20:06.000Z","size":19147,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T11:21:01.637Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prisma.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"docs/code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-17T05:57:10.000Z","updated_at":"2026-05-21T12:31:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/prisma/streams","commit_stats":null,"previous_names":["prisma/streams"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/prisma/streams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prisma%2Fstreams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prisma%2Fstreams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prisma%2Fstreams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prisma%2Fstreams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prisma","download_url":"https://codeload.github.com/prisma/streams/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prisma%2Fstreams/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33790705,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-01T19:31:02.190Z","updated_at":"2026-06-01T19:31:04.082Z","avatar_url":"https://github.com/prisma.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prisma Streams\n\nPrisma Streams is a Bun + TypeScript implementation of the Durable Streams HTTP\nprotocol.\n\nIt provides:\n\n- a full server mode with SQLite WAL storage, segmenting, upload, and index\n  maintenance\n- a trusted local mode for Prisma development workflows\n- a stream profile model that cleanly separates durable storage semantics from\n  payload structure\n\nThe canonical documentation index is [docs/index.md](./docs/index.md). The\ndedicated stream profile reference is\n[docs/stream-profiles.md](./docs/stream-profiles.md).\n\n## Deploy To Prisma Compute\n\nDeploy Compute services from the published `@prisma/streams-server` npm package\nrather than from a checkout of this repository. Create a small Compute app with\nthe package dependency and an entrypoint that selects the auth mode before\nloading the package Compute entrypoint:\n\n```json\n{\n  \"private\": true,\n  \"type\": \"module\",\n  \"dependencies\": {\n    \"@prisma/streams-server\": \"\u003cpin-the-version\u003e\"\n  }\n}\n```\n\n```ts\n// compute-entry.ts\nprocess.argv.push(\"--auth-strategy\", \"api-key\");\nawait import(\"@prisma/streams-server/compute\");\n```\n\nDeploy that app with the Compute CLI:\n\n```bash\nPRISMA_API_TOKEN=... \\\n  bunx @prisma/compute-cli compute deploy \\\n    --service your-service-id \\\n    --path . \\\n    --entrypoint compute-entry.ts\n```\n\nSet these service environment variables:\n\n- `DS_HOST=0.0.0.0`\n- `DS_ROOT=/mnt/app/prisma-streams`\n- `DS_MEMORY_LIMIT_MB=1024`\n- `API_KEY=replace-with-at-least-10-characters`\n- `DURABLE_STREAMS_R2_BUCKET`\n- `DURABLE_STREAMS_R2_ACCOUNT_ID`\n- `DURABLE_STREAMS_R2_ACCESS_KEY_ID`\n- `DURABLE_STREAMS_R2_SECRET_ACCESS_KEY`\n\nThe package Compute entrypoint injects `--object-store r2`, and injects\n`--auto-tune` when `DS_MEMORY_LIMIT_MB` is set. It does not inject auth; keep\nthe explicit `--auth-strategy api-key` in your app entrypoint and send requests\nwith `Authorization: Bearer $API_KEY`.\n\n## Core Model\n\nThe system now has three separate concepts:\n\n- a **stream** is the durable append-only storage object with Durable Streams\n  semantics\n- a **profile** defines the semantic contract of that stream\n- a **schema** defines payload structure\n\nShort rule:\n\n- profile = semantics\n- schema = structure\n\nMore concretely:\n\n- the **stream** owns ordered append/read behavior, offsets, and durable\n  storage\n- the **profile** owns semantic behavior, profile-specific endpoints, and\n  profile-specific runtime configuration\n- the **schema** owns JSON validation, version boundaries, lenses, and\n  routing-key extraction\n\nProfiles sit on top of the durable stream engine. They do not replace streams.\n\nBuilt-in profiles are implemented under `src/profiles/`. The core engine\nresolves a profile definition and dispatches through its hooks instead of\nbranching on profile kinds in request or background processing paths.\n\nProfile-specific behavior must live in a dedicated profile module such as\n`src/profiles/stateProtocol.ts`. The supported extension model is:\n\n- implement the profile in its own file or subdirectory under `src/profiles/`\n- register it once in `src/profiles/index.ts`\n- let core paths call profile hooks instead of adding `if (profile.kind ===\n  \"...\")` checks\n\n## Profile Defaults\n\nEvery stream has a profile.\n\n- if you declare a profile, that is the stream's profile\n- if you create a stream without declaring a profile, the server treats it as a\n  `generic` stream\n\nThis means old or unconfigured streams still have a clear meaning: they are\nplain `generic` durable streams.\n\n## Built-In Profiles\n\nCurrent built-ins:\n\n- `evlog`\n- `generic`\n- `metrics`\n- `state-protocol`\n\nPlanned next built-ins:\n\n- `queue`\n\n### `evlog`\n\n`evlog` is the built-in profile for request-centric wide-event logging.\n\nIt means:\n\n- the stream content type must be `application/json`\n- JSON appends are normalized into a canonical evlog envelope\n- sensitive context keys are redacted before durable append\n- installing the profile also installs the canonical evlog schema registry and\n  default search fields\n- the default routing key is `requestId`, with `traceId` fallback\n\nV1 evlog uses the normal stream append and read APIs. It does not add a local\nSQLite observability index, and it does not require a separate manual\n`/_schema` call for the default search-ready setup.\n\n### `generic`\n\n`generic` means:\n\n- plain ordered append-only storage\n- optional user-managed schema validation\n- optional schema-managed routing-key extraction\n- no profile-owned payload envelope\n- no profile-specific endpoints\n\n`generic` is intentionally narrow. It is the baseline durable stream and the\nautomatic default when no other profile is declared.\n\n### `state-protocol`\n\n`state-protocol` is the built-in profile for JSON streams that carry State\nProtocol change records and expose the live `/touch/*` API surface.\n\nIt means:\n\n- the stream content type must be `application/json`\n- the payload semantics are State Protocol records\n- touch configuration belongs to the profile\n- `/touch/*` exists only when `touch.enabled=true`\n\nSchemas remain optional on `state-protocol` streams. If present, they validate\nthe JSON payload shape, but they do not own live/touch behavior.\n\nState Protocol is a profile, not a schema feature, because it defines stream\nsemantics and profile-owned endpoints, not just JSON shape.\n\n### `metrics`\n\n`metrics` is the built-in profile for canonical metric interval streams.\n\nIt means:\n\n- the stream content type must be `application/json`\n- JSON appends are normalized into the canonical metrics interval envelope\n- installing the profile also installs the canonical metrics schema/search and\n  default rollups\n- the canonical routing key is `seriesKey`\n- metrics streams use the `.mblk` metrics-block family in addition to `.agg`\n\nThe internal `__stream_metrics__` stream is created with this profile\nautomatically.\n\n## Profile Versus Schema\n\nWhat belongs in a profile:\n\n- semantic meaning of records\n- profile-specific runtime config\n- profile-specific endpoints\n- profile-owned indexes or projections\n- future canonical envelopes for specialized stream types\n\nWhat belongs in a schema:\n\n- JSON validation\n- version boundaries\n- lens-based read promotion\n- routing-key extraction rules\n- schema-owned `search` field declarations used by `GET ...?filter=...` and\n  `POST .../_search`\n\nWhat does **not** belong in `/_schema`:\n\n- profile selection\n- touch configuration\n- State Protocol runtime behavior\n- evlog envelope normalization or redaction\n\nThe supported model is strict: `/_profile` manages profile semantics,\n`/_schema` manages schema evolution.\n\nIndexed JSON streams can also use the main read path with `filter=...`.\nFilters are limited to schema `search.fields`, use the internal exact family\nand bundled `.cix` companion sections to prune sealed history where possible,\nand still scan the local unsealed tail for correctness. If an exact field\ndefinition changes on an existing stream, the old exact state is treated as\nstale and the read path falls back cleanly until async rebuild catches up. One\nfiltered response stops after 100 MB of examined payload bytes and reports that\ncap in response headers.\n\nJSON streams with `search` configured also support `_search`:\n\n- `POST /v1/stream/{name}/_search`\n- `GET /v1/stream/{name}/_search?q=...`\n\nThe current `_search` surface supports fielded exact keyword queries, keyword\nprefix, typed equality/range, `has:field`, bare terms over\n`search.defaultFields`, and quoted phrase queries on text fields with\n`positions=true`.\n\nSchema-owned rollups are also available through:\n\n- `POST /v1/stream/{name}/_aggregate`\n\nRollups are configured under `search.rollups`, stored as object-store-native\nbundled companion sections, and used for aligned time windows with raw source\nscans for partial edges and uncovered ranges.\n\n## Management And Introspection API\n\nFor stream management UIs, the current per-stream inspection surface is:\n\n- `GET /v1/streams`\n  Summary list view with name, offsets, expiry, and profile kind.\n- `GET /v1/stream/{name}/_profile`\n  Full typed profile resource.\n- `GET /v1/stream/{name}/_schema`\n  Full schema registry, including profile-owned canonical registries such as\n  the auto-installed evlog schema.\n- `GET /v1/stream/{name}/_index_status`\n  Current manifest, segment, and async index/search-family status for that\n  stream.\n- `GET /v1/stream/{name}/_details`\n  Combined stream summary, including `stream.total_size_bytes`, full profile\n  resource, full schema registry, nested index status, storage accounting, and\n  node-local object-store request counters in one response. This endpoint also\n  supports conditional long-polling with\n  `If-None-Match`, `live=long-poll`, and `timeout=...`, and only wakes when\n  the stream head or descriptor-visible metadata changes.\n\n`/_details.stream` is the full per-stream summary shape. In practice that means\nit includes the stream head and lifecycle fields a UI usually needs for an\nactive stream page, including:\n\n- `created_at`\n- `expires_at`\n- `epoch`\n- `next_offset`\n- `sealed_through`\n- `uploaded_through`\n- `total_size_bytes`\n\nFor a storage/cost popover, `/_details` also includes:\n\n- `storage.object_storage`\n  Uploaded bytes and object counts split into segments, indexes, and\n  manifest/schema metadata.\n- `storage.local_storage`\n  Current local retained bytes split into WAL, pending sealed segments, caches,\n  and shared SQLite footprint.\n- `storage.companion_families`\n  Bundled companion bytes split into `col`, `fts`, `agg`, and `mblk`.\n- `object_store_requests`\n  Node-local per-stream object-store request counters, split into puts and\n  reads, plus a per-artifact breakdown.\n\nThat means a GUI can create streams, inspect the active profile and schema,\nshow current indexing progress, and edit profile/schema configuration through\nthe normal API surface. A charting UI can additionally use `/_aggregate` for\ntime-window summaries driven by schema `search.rollups`.\n\nFor an active stream page, the recommended pattern is:\n\n- fetch `GET /v1/stream/{name}/_details`\n- keep the returned `ETag`\n- reissue `GET /v1/stream/{name}/_details?live=long-poll\u0026timeout=30s` with\n  `If-None-Match: \u003cetag\u003e`\n\nThe server returns:\n\n- `200` with a new body and new `ETag` when events or descriptor metadata have\n  changed\n- `304` when the timeout expires without a visible change\n\n## Profile API\n\nProfiles are managed through a dedicated subresource:\n\n- `GET /v1/stream/{name}/_profile`\n- `POST /v1/stream/{name}/_profile`\n\nExample default response:\n\n```json\n{\n  \"apiVersion\": \"durable.streams/profile/v1\",\n  \"profile\": { \"kind\": \"generic\" }\n}\n```\n\nExplicit `generic` declaration:\n\n```json\n{\n  \"apiVersion\": \"durable.streams/profile/v1\",\n  \"profile\": { \"kind\": \"generic\" }\n}\n```\n\nState Protocol profile with touch enabled:\n\n```json\n{\n  \"apiVersion\": \"durable.streams/profile/v1\",\n  \"profile\": {\n    \"kind\": \"state-protocol\",\n    \"touch\": {\n      \"enabled\": true,\n      \"onMissingBefore\": \"coarse\"\n    }\n  }\n}\n```\n\nEvlog profile with redaction:\n\n```json\n{\n  \"apiVersion\": \"durable.streams/profile/v1\",\n  \"profile\": {\n    \"kind\": \"evlog\",\n    \"redactKeys\": [\"sessiontoken\"]\n  }\n}\n```\n\nTo switch a stream back to the baseline behavior, set `profile` to\n`{ \"kind\": \"generic\" }`.\n\n## Storage Model\n\nThe stored profile is kept in stream metadata.\n\n- `streams.profile` stores the profile kind\n- `stream_profiles.profile_json` stores non-generic profile configuration\n- `NULL` profile metadata means there is no explicit stored entry and the\n  stream is treated as `generic`\n\nThis keeps storage simple while still letting runtime code assume a profile\nalways exists.\n\n## Current Durability Model\n\nIn full mode today:\n\n- append ACK means the write is durable in local SQLite\n- object-store durability happens only after segment upload and manifest\n  publication\n\n`--bootstrap-from-r2` rebuilds published stream history and metadata from\nmanifest, segment, and schema objects in object storage. It does\nnot restore transient local SQLite state such as the unuploaded WAL tail,\nproducer dedupe state, or runtime live/template state.\n\nA stream becomes recoverable from object storage after its first manifest is\npublished.\n\n## Possible Future Durability Modes\n\nNot implemented today:\n\n- an object-store-acked mode that would batch writes and ACK only after\n  persistence to R2\n- a cluster quorum mode that would ACK only after a durability quorum accepts\n  the write\n\n## Current Supported Paths\n\nThe supported behavior is:\n\n- use `/_profile` to choose `generic`, `state-protocol`, or `evlog`\n- use `/_schema` only for schema validation, routing-key config, and schema\n  evolution\n- use `/touch/*` only on `state-protocol` streams with touch enabled\n- use normal JSON appends on `evlog` streams to store canonical evlog events\n\nLegacy compatibility branches are intentionally not part of the supported\nsurface.\n\n## Start Here\n\n- [docs/index.md](./docs/index.md) for the documentation map\n- [docs/overview.md](./docs/overview.md) for product and package overview\n- [docs/stream-profiles.md](./docs/stream-profiles.md) for the full stream /\n  profile / schema reference\n- [docs/durable-streams-spec.md](./docs/durable-streams-spec.md) for the HTTP\n  protocol contract\n- [docs/live.md](./docs/live.md) for the State Protocol live/touch model\n- [docs/schemas.md](./docs/schemas.md) for schema registry and lens behavior\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprisma%2Fstreams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprisma%2Fstreams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprisma%2Fstreams/lists"}