{"id":26638036,"url":"https://github.com/spellcraftai/mdstream","last_synced_at":"2025-03-24T17:24:57.625Z","repository":{"id":252732791,"uuid":"841283610","full_name":"SpellcraftAI/mdstream","owner":"SpellcraftAI","description":"Streaming markdown parser. Includes TransformStream utilities and ANSI, HTML, and DOM renderers. Fork of streaming-markdown, most code by @thetarnav.","archived":false,"fork":false,"pushed_at":"2024-09-14T00:50:23.000Z","size":493,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-09-15T09:57:55.321Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SpellcraftAI.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-12T05:12:30.000Z","updated_at":"2024-09-14T00:50:26.000Z","dependencies_parsed_at":"2024-09-12T15:35:47.438Z","dependency_job_id":"860bbddd-bc77-42cc-aa7c-0d6e694e89fb","html_url":"https://github.com/SpellcraftAI/mdstream","commit_stats":null,"previous_names":["spellcraftai/mdstream"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpellcraftAI%2Fmdstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpellcraftAI%2Fmdstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpellcraftAI%2Fmdstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpellcraftAI%2Fmdstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SpellcraftAI","download_url":"https://codeload.github.com/SpellcraftAI/mdstream/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245316754,"owners_count":20595492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-24T17:24:57.013Z","updated_at":"2025-03-24T17:24:57.579Z","avatar_url":"https://github.com/SpellcraftAI.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Streaming *Markdown*\n\n[![version](https://img.shields.io/npm/v/streaming-markdown?logo=npm)](https://www.npmjs.com/package/streaming-markdown) [![github](https://img.shields.io/badge/GitHub-streaming--markdown-orange?logo=github)](https://github.com/thetarnav/streaming-markdown)\n\n**Experiment making a streaming markdown parser *à la ChatGPT*.**\n\n| Version | Size (gzip) |\n|---------|----------------|\n| Full package | 16kB |\n| `browser.js` | 2.5kB |\n\n## INTERNAL FORK\n\n*Original:* https://github.com/thetarnav/streaming-markdown\n\n*Original author and all parsing logic*: @thetarnav\n\n---\n\n## Installation\n\nInstall `mdstream` package from npm.\n\n```bash\nbun i mdstream\n```\n\nOr use the [CDN link](https://www.jsdelivr.com/package/npm/mdstream). The\n`browser` export is a minified **(2.5kB gzipped)** version of the package, with only\nthe parser methods and and DOM renderer exported.\n\n```html\n\u003cscript type=\"module\"\u003e\n    import { parse, finish, createParser, createDOMRenderer } from \"https://cdn.jsdelivr.net/npm/mdstream/dist/browser.js\"\n    // ...\n\u003c/script\u003e\n```\n\nThe package uses ES module exports, so you need to use `type=\"module\"` in your\nscript tag. See usage below.\n\n## Using the parser\n\nFirst create new markdown `Parser` by calling `parser` function. It's single\nargument is a `Renderer` object, which is an interface to render the parsed\nmarkdown tokens to the DOM. The built-in renderers are:\n\n- `DOMRenderer`, which renders by appending to the DOM from a browser client\n  script;\n- `HTMLRenderer`, which renders to raw HTML;\n- `ANSIRenderer`, which renders to ANSI-styled text using `chalk`; and\n- `LogRenderer`, which prints the internal parser methods as they're called.\n\nSee **Examples** below.\n\n### `parse` function\n\nThen, you can start streaming markdown to the `Parser` by calling `parse()`\nfunction with the chunk of markdown string.\n\n```js\nparse(parser, \"# Streaming Markdown\\n\\n\")\n```\n\n*You can write **as many times as you want** to stream the markdown.*\n\nThe parser is optimistic. When it sees the start of an inline code block or code\nblock, it will immediately style the element accordingly.\n\nE.g. `` `print(\"hello wor `` should be rendered as `\u003ccode\u003eprint(\"hello\nwor\u003c/code\u003e`\n\nWhile the text is streamed in, the user should be able to select the text that\nhas already been streamed in and copy it.\n\n*(The parser is only adding new elements to the DOM, not modifying the existing\nones.)*\n\n### `finish` function\n\nFinally, you can end the stream by calling `finish()` function.\n\nIt will reset the `Parser` state and flush the remaining markdown.\n\n```js\nfinish(parser)\n```\n\n## Working with streams: `ReadableStream\u003cUint8Array\u003e`\n\nTo transform a `ReadableStream`, use `MarkdownStream` to create a\n`TransformStream\u003cUint8Array, Uint8Array\u003e`. The built-in renderers come with\nrenderer transforms already (except the DOM renderer, which manipulates the DOM\nat runtime):\n\n- `MarkdownHTMLStream`\n- `MarkdownANSIStream`\n- `MarkdownLogStream`\n\n### Render to DOM using parser\n\n```js\nimport { parse, finish, createParser, createDOMRenderer } from \"mdstream\"\n\nconst response = await fetch(\"readme.md\")\nconst source = await response.text()\n\nconst container = document.getElementById(\"markdown\")\nconst renderer = createDOMRenderer(container)\nconst parser = createParser(renderer)\n\nlet i = 0\nwhile (i \u003c source.length) {\n  const length = Math.floor(Math.random() * 20) + 1\n  const delay = Math.floor(Math.random() * 80) + 10\n  const chunk = source.slice(i, i += length)\n\n  await new Promise(resolve =\u003e setTimeout(resolve, delay))\n  parse(parser, chunk)\n}\n\nfinish(parser)\n```\n\n### Streams: Rendering to HTML (server-side)\n\n```ts\nconst readme = Bun.file(\"readme.md\")\nconst response = new Response(\n  readme.stream().pipeThrough(new MarkdownHTMLStream())\n)\n```\n\n### Streams: Rendering to DOM (client-side)\n\n```ts\nconst container = document.getElementById(\"markdown\")\nconst response = await fetch(\"readme.md\")\nawait response.body\n  .pipeThrough(new MarkdownDOMStream(container))\n  .pipeTo(new WritableStream())\n```\n\n### Extending\n\n`MarkdownStream` can wrap any renderer, for instance `MarkdownLogStream` creates\nthe parser and returns it for its parent to use:\n\n```ts\nexport class MarkdownLogStream extends MarkdownStream {\n  constructor() {\n    const ENCODER = new TextEncoder()\n    \n    super({\n      start: (controller) =\u003e {\n        const renderer = createLogRenderer({\n          render: (chunk) =\u003e controller.enqueue(ENCODER.encode(chunk)),\n        })\n\n        return createParser(renderer)\n      }\n    })\n  }\n}\n```\n\n## Examples\n\n### Render to DOM with `DOMRenderer`\n\nDisplaying this README as a demo with delayed chunks:\n\n```html\n\u003cscript type=\"module\"\u003e\n  import { parse, finish, createParser, createDOMRenderer } from \"mdstream\"\n\n  const response = await fetch(\"readme.md\")\n  const source = await response.text()\n\n  const container = document.getElementById(\"markdown\")\n  const renderer = createDOMRenderer(container)\n  const parser = createParser(renderer)\n\n  let i = 0\n  while (i \u003c source.length) {\n    const length = Math.floor(Math.random() * 20) + 1\n    const delay = Math.floor(Math.random() * 80) + 10\n    const chunk = source.slice(i, i += length)\n\n    await new Promise(resolve =\u003e setTimeout(resolve, delay))\n    parse(parser, chunk)\n  }\n\n  finish(parser)\n\u003c/script\u003e\n```\n\n### Testing\n\n\u003e Blockquotes can span multiple lines.\n\u003e Another line.\n\n## Markdown features\n\n- [x] Paragraphs\n- [x] Line breaks\n    - [x] don't end tokens\n    - [x] Escaping line breaks\n- [x] Trim unnecessary spaces\n- [x] Headers\n    - [ ] ~~Alternate syntax~~ *(not planned)*\n- [x] Code Block with indent\n- [x] Code Block with triple backticks\n    - [x] language attr\n    - [x] with many backticks\n- [x] `` `inline code` `` with backticks\n    - [x] with many backticks\n    - [x] trim spaces ` code `\n- [x] *italic* with single asterisks\n- [x] **Bold** with double asterisks\n- [x] _italic_ with underscores\n- [x] __Bold__ with double underscores\n- [x] Special cases:\n    - [x] **bold*bold\u003eem***\n    - [x] ***bold\u003eem*bold**\n    - [x] *em**em\u003ebold***\n    - [x] ***bold\u003eem**em*\n- [x] \\* or \\_ cannot be surrounded by spaces\n- [x] Strikethrough ~~example~~\n- [x] Escape characters (e.g. \\* or \\_ with \\\\\\* or \\\\\\_)\n- [x] \\[Link\\](url)\n    - [x] `href` attr\n- [ ] Raw URLs\n    - [ ] http://example.com\n    - [ ] https://example.com\n    - [ ] www.example.com\n    - [ ] example@fake.com\n    - [ ] mailto:example@fake.com\n- [x] Autolinks\n    - [ ] www.example.com\n    - [x] http://example.com\n    - [x] https://example.com\n    - [ ] example@fake.com\n    - [ ] mailto:example@fake.com\n- [ ] Reference-style Links\n- [x] Images\n    - [x] `src` attr\n- [x] Horizontal rules\n    - [x] With `---`\n    - [x] With `***`\n    - [x] With `___`\n- [x] Unordered lists\n- [x] Ordered lists\n    - [x] `start` attr\n- [x] Task lists\n- [x] Nested lists\n- [ ] One-line nested lists\n- [ ] Adding Elements in Lists\n- [x] Blockquotes\n- [ ] Tables\n- [ ] Subscript\n- [ ] Superscript\n- [ ] Emoji Shortcodes\n- [ ] Html tags (e.g. `\u003cdiv\u003e`, `\u003cspan\u003e`, `\u003ca\u003e`, `\u003cimg\u003e`, etc.)\n- Code blocks in lists\n\n  ```ts\n  test\n  backticks: `\n  backticks: `\n  ```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspellcraftai%2Fmdstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspellcraftai%2Fmdstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspellcraftai%2Fmdstream/lists"}