{"id":35832886,"url":"https://github.com/laphilosophia/strime","last_synced_at":"2026-02-15T20:05:15.755Z","repository":{"id":332413723,"uuid":"1128238805","full_name":"laphilosophia/strime","owner":"laphilosophia","description":"Streaming projection engine — extract fields at multi-gigabit speeds with O(1) memory","archived":false,"fork":false,"pushed_at":"2026-02-11T14:44:34.000Z","size":271,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-11T20:49:35.608Z","etag":null,"topics":["data-extraction","data-processing","fsm","high-performance","json","json-parser","json-streaming","ndjson","parser","projection","streaming","typescript","zero-copy"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/laphilosophia.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-05T10:44:49.000Z","updated_at":"2026-02-11T20:01:08.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/laphilosophia/strime","commit_stats":null,"previous_names":["laphilosophia/strime"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/laphilosophia/strime","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laphilosophia%2Fstrime","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laphilosophia%2Fstrime/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laphilosophia%2Fstrime/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laphilosophia%2Fstrime/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/laphilosophia","download_url":"https://codeload.github.com/laphilosophia/strime/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/laphilosophia%2Fstrime/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29487804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T19:29:10.908Z","status":"ssl_error","status_checked_at":"2026-02-15T19:29:10.419Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-extraction","data-processing","fsm","high-performance","json","json-parser","json-streaming","ndjson","parser","projection","streaming","typescript","zero-copy"],"created_at":"2026-01-07T23:14:31.650Z","updated_at":"2026-02-15T20:05:15.746Z","avatar_url":"https://github.com/laphilosophia.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Strime\n\nA streaming JSON projection engine. Selects and extracts fields from JSON without parsing the entire structure.\n\n[![NPM Download](https://img.shields.io/badge/npm-strime-red.svg)](https://www.npmjs.com/package/@laphilosophia/strime)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache2.0-blue.svg)](LICENSE)\n[![Tests](https://img.shields.io/badge/tests-52%2F52-brightgreen.svg)](src/__tests__)\n\n---\n\n## Why Strime\n\nMost JSON parsers convert the entire input to objects before you can query it. Strime inverts this: it filters during traversal, touching only the bytes you need. The result is constant memory usage regardless of file size—a 10GB file uses the same baseline memory as a 10KB file.\n\nThis isn't a marginal improvement. On high-volume workloads, Strime processes JSON at throughputs typically reserved for native implementations.\n\n---\n\n## Performance\n\n| Metric                   | Result         | Guarantee      |\n| ------------------------ | -------------- | -------------- |\n| Throughput (1GB)         | 809-939 Mbps   | O(N) time      |\n| Skip-heavy (chunked)     | **4,408 Mbps** | 6.5x faster    |\n| Memory                   | Constant       | O(D) space     |\n| 1M NDJSON rows           | ~4.2s          | 220k+ rows/s   |\n| Deep nesting (1k levels) | \u003c 1ms          | Stack-safe     |\n| Large strings (50MB)     | 1.6 Gbps       | No degradation |\n\n_Validated on 1GB+ files. See [Performance Contract](docs/performance.md) for methodology._\n\n---\n\n## Install\n\n```bash\nnpm install @laphilosophia/strime\n```\n\n---\n\n## Use\n\n### Query\n\n```typescript\nimport { query } from '@laphilosophia/strime'\n\nconst result = await query(data, '{ id, user { name, email } }')\n```\n\n### Stream NDJSON\n\n```typescript\nimport { ndjsonStream } from '@laphilosophia/strime'\n\nfor await (const row of ndjsonStream(stream, '{ id, name }')) {\n  console.log(row)\n}\n```\n\n### Fault-Tolerant\n\n```typescript\nfor await (const row of ndjsonStream(stream, '{ id, name }', {\n  skipErrors: true,\n  onError: (info) =\u003e console.error(`Line ${info.lineNumber}: ${info.error.message}`),\n})) {\n  console.log(row)\n}\n```\n\n### Real-Time Subscribe\n\n```typescript\nimport { subscribe } from '@laphilosophia/strime'\n\nsubscribe(telemetryStream, '{ lat, lon }', {\n  onMatch: (data) =\u003e console.log(data),\n  onComplete: () =\u003e console.log('Done'),\n})\n```\n\n### CLI\n\n```bash\nstrime data.json \"{ name, meta { type } }\"\ncat massive.json | strime \"{ actor { login } }\"\ntail -f telemetry.log | strime --ndjson \"{ lat, lon }\"\n```\n\n---\n\n## Features\n\n**Core**\n\n- Zero-allocation tokenizer with object recycling\n- String interning for repeated keys\n- Integer fast-path parsing\n- Binary line splitting for NDJSON\n\n**API**\n\n- Streaming and pull modes\n- Parallel NDJSON processing (`ndjsonParallel`)\n- Raw byte emission for zero-copy pipelines\n- Compression sink (gzip/brotli)\n- Budget enforcement for DoS protection\n\n**Error Handling**\n\n- Typed errors with position tracking\n- Fault-tolerant NDJSON streaming\n- Controlled termination via AbortSignal\n\n---\n\n## Advanced\n\n### Chunked Execution (Large Files)\n\nFor maximum throughput on large single-buffer inputs:\n\n```typescript\nimport { Engine } from '@laphilosophia/strime'\nimport { StrimeParser } from '@laphilosophia/strime'\nimport { readFileSync } from 'fs'\n\nconst buffer = readFileSync('large-file.json')\nconst schema = new StrimeParser('{ id, name }').parse()\nconst engine = new Engine(schema)\n\n// 6.5x faster on skip-heavy workloads\nconst result = engine.executeChunked(buffer) // 64KB chunks (default)\nconst result = engine.executeChunked(buffer, 32768) // Custom 32KB chunks\n```\n\n### Low-Level Tokenizer\n\n```typescript\nimport { Tokenizer } from '@laphilosophia/strime'\n\nconst tokenizer = new Tokenizer()\nconst buffer = new TextEncoder().encode('{\"key\": \"value\"}')\n\n// Zero-allocation callback mode\ntokenizer.processChunk(buffer, (token) =\u003e {\n  console.log(token.type, token.value)\n})\n\n// Iterator mode\nfor (const token of tokenizer.tokenize(buffer)) {\n  console.log(token.type, token.value)\n}\n```\n\n### Telemetry\n\n```typescript\nawait query(stream, '{ id, name }', {\n  sink: {\n    onStats: (stats) =\u003e {\n      console.log(`Throughput: ${stats.throughputMbps.toFixed(2)} Mbps`)\n      console.log(`Skip ratio: ${(stats.skipRatio * 100).toFixed(1)}%`)\n    },\n  },\n})\n```\n\n### Raw Emission\n\n```typescript\nawait query(stream, '{ items { id } }', {\n  emitMode: 'raw',\n  sink: {\n    onRawMatch: (bytes) =\u003e outputStream.write(bytes),\n  },\n})\n```\n\n---\n\n## Documentation\n\n- [Quick Start](docs/quick-start.md) — Getting started\n- [Deployment Matrix](docs/howto-deployment-matrix.md) — Decision guide for API entry points\n- [Real World Use Cases](docs/howto-use-cases.md) — Practical projection scenarios\n- [Query Language](docs/query-language.md) — Syntax reference\n- [API Reference](docs/api-reference.md) — Complete API\n- [CLI Guide](docs/cli-guide.md) — Command-line usage\n- [Performance](docs/performance.md) — Benchmarks and guarantees\n- [Internals](docs/internals.md) — How it works\n\n---\n\n## Licensing\n\nStrime is released under the **Apache 2.0**.\n\n---\n\n## Contact\n\nErdem Arslan\n\u003cme@erdem.work\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaphilosophia%2Fstrime","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaphilosophia%2Fstrime","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaphilosophia%2Fstrime/lists"}