{"id":50777033,"url":"https://github.com/gitmahin/nodejs-web-stream-demo","last_synced_at":"2026-06-12T00:30:40.591Z","repository":{"id":353024542,"uuid":"1217618594","full_name":"gitmahin/nodejs-web-stream-demo","owner":"gitmahin","description":"This repository demonstrates Node.js web streams for beginners. It shows how to scrape thousands of products and send them to clients on demand with newline-delimited JSON (NDJSON) support.","archived":false,"fork":false,"pushed_at":"2026-04-22T06:34:10.000Z","size":176,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-22T07:29:23.727Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gitmahin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-22T04:06:24.000Z","updated_at":"2026-04-22T06:34:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/gitmahin/nodejs-web-stream-demo","commit_stats":null,"previous_names":["gitmahin/nodejs-web-stream-demo"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/gitmahin/nodejs-web-stream-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitmahin%2Fnodejs-web-stream-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitmahin%2Fnodejs-web-stream-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitmahin%2Fnodejs-web-stream-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitmahin%2Fnodejs-web-stream-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gitmahin","download_url":"https://codeload.github.com/gitmahin/nodejs-web-stream-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitmahin%2Fnodejs-web-stream-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34224103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-12T00:30:33.863Z","updated_at":"2026-06-12T00:30:40.579Z","avatar_url":"https://github.com/gitmahin.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🌊 Node.js Web Streams — Live Scraping Demo\n\nA full-stack demonstration of **Node.js Web Streams API** for real-time, memory-efficient data streaming from a scraper to a React UI - with zero buffering on the server side.\n\n---\n\n## What This Demonstrates\n\nThis project shows how to pipe data end-to-end using the **Nodejs Stream**, without waiting for all data to be collected before sending it to the client.\n\n```\nPuppeteer (AsyncGenerator)\n       ↓\n  Readable.from()          - Node.js readable stream from async generator\n       ↓\n  Readable.toWeb()         - Convert to WHATWG ReadableStream\n       ↓\n  TransformStream          - Serialize each page's products to NDJSON\n       ↓\n  Writable.toWeb(res)      - Pipe directly into Express HTTP response\n       ↓  (HTTP / fetch)\n  TextDecoderStream        - Decode binary chunks to string (browser)\n       ↓\n  TransformStream          - Parse NDJSON lines back to JS objects\n       ↓\n  WritableStream           - Write parsed products into React state\n```\n\n---\n\n## Folder Structure\n\n```\n├── react-client/          # Vite + React frontend\n│   └── src/\n│       └── App.jsx        # Stream consumer UI\n│\n└── server/                # Express backend\n    └── index.ts           # Puppeteer scraper + stream pipeline\n```\n\n---\n\n## Key Concepts\n\n### NDJSON (Newline Delimited JSON)\nEach scraped page is serialized as one JSON line followed by `\\n`. This lets the client parse results incrementally - one line = one complete chunk of data - without waiting for the full response.\n\n```\n[{\"title\":\"Bag A\",...}]\\n\n[{\"title\":\"Bag B\",...}]\\n\n```\n\n### Server - `TransformStream` for serialization\n```ts\nnew TransformStream({\n  transform(chunk, controller) {\n    this.buffer += JSON.stringify(chunk) + \"\\n\";\n    let boundary = this.buffer.indexOf(\"\\n\");\n    while (boundary !== -1) {\n      const line = this.buffer.substring(0, boundary);\n      this.buffer = this.buffer.substring(boundary + 1);\n      if (line.trim()) controller.enqueue(line + \"\\n\");\n      boundary = this.buffer.indexOf(\"\\n\");\n    }\n  },\n  flush(controller) {\n    if (this.buffer.trim()) controller.enqueue(this.buffer + \"\\n\");\n  },\n})\n```\n\u003e `buffer` is stored on `this` inside the transformer — it persists across chunks and holds incomplete data between reads.\n\n### Client - `TransformStream` for parsing\n```ts\nnew TransformStream({\n  transform(chunk, controller) {\n    this.buffer += chunk;\n    const lines = this.buffer.split(\"\\n\");\n    this.buffer = lines.pop() ?? \"\";\n    for (const line of lines) {\n      if (!line.trim()) continue;\n      try { controller.enqueue(JSON.parse(line)); } catch {}\n    }\n  },\n})\n```\n\u003e Chunks arriving over HTTP may be split arbitrarily — the buffer ensures we only parse **complete** lines.\n\n### Why `Readable.from()` + `Readable.toWeb()`?\nPuppeteer's scraper is an `AsyncGenerator`. Node's `Readable.from()` wraps it in a Node.js stream, and `Readable.toWeb()` converts it to a [WHATWG](https://streams.spec.whatwg.org/) `ReadableStream` - making it compatible with `.pipeThrough()` and `.pipeTo()`.\n\n---\n\n## Getting Started\n\n### Prerequisites\n- Node.js 18+\n- Google Chrome installed (used by Puppeteer)\n\n### Install \u0026 Run\n\n**Puppeteer browser**\n```bash\nnpx puppeteer browsers install chrome\n```\n\n**Server**\n```bash\ncd server\npnpm install\npnpm run dev        # or: npx ts-node index.ts\n```\n\n**Client**\n```bash\ncd react-client\npnpm install\npnpm run dev\n```\n\nThen open `http://localhost:5173` and click **Start Scraping**.\n\n---\n\n## How It Works - Step by Step\n\n1. A `GET /scrap` request hits the Express server.\n2. Puppeteer launches a headless Chrome and navigates to the target shop page.\n3. An `async generator` (`getProducts`) scrapes each page and `yield`s an array of products, then clicks the pagination \"next\" button and repeats.\n4. The generator is wrapped in a `Readable` stream and converted to a WHATWG `ReadableStream`.\n5. A `TransformStream` serializes each yielded array to an NDJSON line and enqueues it.\n6. The final stream is piped directly into the HTTP response - **data starts flowing to the client before all pages are scraped**.\n7. In the browser, the response body is decoded, split by newlines, parsed as JSON, and appended to React state in real time.\n\n---\n\n## Notes \u0026 Caveats\n\n- The scraper targets `skybuybd.com/shop/purse` - adjust the URL and selectors as needed.\n- Products render **live** as each page is scraped; no need to wait for all 50 pages.\n- The `buffer` property on `TransformStream` is a pattern for stateful transforms - it is not part of the `Transformer` interface spec, so TypeScript requires it to be managed outside the object or cast appropriately.\n- Memory stays low because the generator `yield`s one page at a time, not all pages at once.\n\n---\n\n## Tech Stack\n\n| Layer | Technology |\n|---|---|\n| Scraping | Puppeteer Extra + Stealth Plugin + Adblocker |\n| Server | Node.js, Express, WHATWG Streams (`node:stream/web`) |\n| Client | React (Vite), WHATWG Streams (browser-native) |\n| Data format | NDJSON over HTTP streaming |","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitmahin%2Fnodejs-web-stream-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgitmahin%2Fnodejs-web-stream-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitmahin%2Fnodejs-web-stream-demo/lists"}