Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/romansky/dom-to-semantic-markdown
DOM to Semantic-Markdown for use with LLMs
https://github.com/romansky/dom-to-semantic-markdown
Last synced: 1 day ago
JSON representation
DOM to Semantic-Markdown for use with LLMs
- Host: GitHub
- URL: https://github.com/romansky/dom-to-semantic-markdown
- Owner: romansky
- License: mit
- Created: 2024-07-22T20:38:49.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-10-06T12:23:22.000Z (2 months ago)
- Last Synced: 2024-12-05T18:03:15.931Z (8 days ago)
- Language: TypeScript
- Homepage:
- Size: 279 KB
- Stars: 684
- Watchers: 8
- Forks: 16
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
DOM to Semantic Markdown
[](https://github.com/romansku/dom-to-semantic-markdown/actions/workflows/ci.yml)
[](https://badge.fury.io/js/dom-to-semantic-markdown)
[](https://opensource.org/licenses/MIT)
This library converts HTML DOM to a semantic Markdown format optimized for use with Large Language Models (LLMs). It
preserves the semantic structure of web content, extracts essential metadata, and reduces token usage compared to raw
HTML, making it easier for LLMs to understand and process information.
## Key Features
* **Semantic Structure Preservation:** Retains the meaning of HTML elements like ``, ``, ``, and
more.
* **Metadata Extraction:** Captures important metadata such as title, description, keywords, Open Graph tags, Twitter
Card tags, and JSON-LD data.
* **Token Efficiency:** Optimizes for token usage through URL refification and concise representation of content.
* **Main Content Detection:** Automatically identifies and extracts the primary content section of a webpage.
* **Table Column Tracking:** Adds unique identifiers to table columns, improving LLM's ability to correlate data across
rows.
## Special Feature Examples
Here are examples showcasing the library's special features using the CLI tool:
**1. Simple Content Extraction:**
```bash
npx d2m@latest -u https://xkcd.com
```
This command fetches `https://xkcd.com` and converts it to Markdown
Click to view the output
```markdown
- [Archive](/archive)
- [What If?](https://what-if.xkcd.com/)
- [About](/about)
- [Feed](/atom.xml) • [Email](/newsletter/)
- [TW](https://twitter.com/xkcd/) • [FB](https://www.facebook.com/TheXKCD/)
• [IG](https://www.instagram.com/xkcd/)
- [-Books-](/books/)
- [What If? 2](/what-if-2/)
- [WI?](/what-if/) • [TE](/thing-explainer/) • [HT](/how-to/)
 A webcomic of romance,
sarcasm, math, and language. [Special 10th anniversary edition of WHAT IF?](https://xkcd.com/what-if/) —revised and
annotated with brand-new illustrations and answers to important questions you never thought to ask—coming from
November 2024. Preorder [here](https://bit.ly/WhatIf10th) ! Exam Numbers
- [|<](/1/)
- [< Prev](/2965/)
- [Random](//c.xkcd.com/random/comic/)
- [Next >](about:blank#)
- [>|](/)
- [|<](/1/)
- [< Prev](/2965/)
- [Random](//c.xkcd.com/random/comic/)
- [Next >](about:blank#)
- [>|](/)
Permanent link to this comic: [https://xkcd.com/2966/](https://xkcd.com/2966)
Image URL (for
hotlinking/embedding): [https://imgs.xkcd.com/comics/exam_numbers.png](https://imgs.xkcd.com/comics/exam_numbers.png)
[RSS Feed](/rss.xml) - [Atom Feed](/atom.xml) - [Email](/newsletter/)
Comics I enjoy:
[Three Word Phrase](http://threewordphrase.com/) , [SMBC](https://www.smbc-comics.com/) , [Dinosaur Comics](https://www.qwantz.com/) , [Oglaf](https://oglaf.com/) (
nsfw), [A Softer World](https://www.asofterworld.com/) , [Buttersafe](https://buttersafe.com/) , [Perry Bible Fellowship](https://pbfcomics.com/) , [Questionable Content](https://questionablecontent.net/) , [Buttercup Festival](http://www.buttercupfestival.com/) , [Homestuck](https://www.homestuck.com/) , [Junior Scientist Power Hour](https://www.jspowerhour.com/)
Other things:
[Tips on technology and government](https://medium.com/civic-tech-thoughts-from-joshdata/so-you-want-to-reform-democracy-7f3b1ef10597) ,
[Climate FAQ](https://www.nytimes.com/interactive/2017/climate/what-is-climate-change.html) , [Katharine Hayhoe](https://twitter.com/KHayhoe)
xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3±1 emulated in Javascript on an Apple IIGS
at a screen resolution of 1024x1. Please enable your ad blockers, disable high-heat drying, and remove your device
from Airplane Mode and set it to Boat Mode. For security reasons, please leave caps lock on while browsing. This work is
licensed under
a [Creative Commons Attribution-NonCommercial 2.5 License](https://creativecommons.org/licenses/by-nc/2.5/).
This means you're free to copy and share these comics (but not to sell them). [More details](/license.html).
```
**2. Table Column Tracking:**
```bash
npx d2m@latest -u https://softwareyoga.com/latency-numbers-everyone-should-know/ -t -e
```
This command fetches and converts the main content from to Markdown and adds unique identifiers to table columns, aiding
LLMs in understanding table structure.
Click to view the output
```markdown
# Latency Numbers Everyone Should Know
## Latency
In a computer network, latency is defined as the amount of time it takes for a packet of data to get from one designated
point to another.
In more general terms, it is the amount of time between the cause and the observation of the effect.
As you would expect, latency is important, very important. As programmers, we all know reading from disk takes longer
than reading from memory or the fact that L1 cache is faster than the L2 cache.
But do you know the orders of magnitude by which these aspects are faster/slower compared to others?
## Latency for common operations
Jeff Dean from Google studied exactly that and came up with figures for latency in various situations.
With improving hardware, the latency at the higher ends of the spectrum are reducing, but not enough to ignore them
completely! For instance, to read 1MB sequentially from disk might have taken 20,000,000 ns a decade earlier and with
the advent of SSDs may probably take 1,000,000 ns today. But it is never going to surpass reading directly from memory.
The table below presents the latency for the most common operations on commodity hardware. These data are only
approximations and will vary with the hardware and the execution environment of your code. However, they do serve their
primary purpose, which is to enable us make informed technical decisions to reduce latency.
For better comprehension of  the multi-fold increase in latency, scaled figures in relation to L2 cache are also
provided by assuming that the L1 cache reference is 1 sec.
**Scroll horizontally on the table in smaller screens**
| Operation | Note | Latency | Scaled Latency |
| --- | --- | --- | --- |
| L1 cache reference | Level-1 cache, usually built onto the microprocessor chip itself. | 0.5 ns | Consider L1 cache reference duration is 1 sec |
| Branch mispredict | During the execution of a program, CPU predicts the next set of instructions. Branch misprediction is when it makes the wrong prediction. Hence, the previous prediction has to be erased and new one calculated and placed on the execution stack. | 5 ns | 10 s |
| L2 cache reference | Level-2 cache is memory built on a separate chip. | 7 ns | 14 s |
| Mutex lock/unlock | Simple synchronization method used to ensure exclusive access to resources shared between many threads. | 25 ns | 50 s |
| Main memory reference | Time to reference main memory i.e. RAM. | 100 ns | 3m 20s |
| Compress 1K bytes with Snappy | Snappy is a fast data compression and decompression library written in C++ by Google and used in many Google projects like BigTable, MapReduce and other open source projects. | 3,000 ns | 1h 40 m |
| Send 1K bytes over 1 Gbps network | | 10,000 ns | 5h 33m 20s |
| Read 1 MB sequentially from memory | Read from RAM. | 250,000 ns | 5d 18h 53m 20s |
| Round trip within same datacenter | We can assume that the DNS lookup will be much faster within a datacenter than it is to go over an external router. | 500,000 ns | 11d 13h 46m 40s |
| Read 1 MB sequentially from SSD disk | Assumes SSD disk. SSD boasts random data access times of 100000 ns or less. | 1,000,000 ns | 23d 3h 33m 20s |
| Disk seek | Disk seek is method to get to the sector and head in the disk where the required data exists. | 10,000,000 ns | 231d 11h 33m 20s |
| Read 1 MB sequentially from disk | Assumes regular disk, not SSD. Check the difference in comparison to SSD! | 20,000,000 ns | 462d 23h 6m 40s |
| Send packet CA->Netherlands->CA | Round trip for packet data from U.S.A to Europe and back. | 150,000,000 ns | 3472d 5h 20m |
### References:
1. [Designs, Lessons and Advice from Building Large Distributed Systems](http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf)
2. [Peter Norvig’s post on – Teach Yourself Programming in Ten Years](http://norvig.com/21-days.html#answers)
```
**3. Metadata Extraction (Basic):**
```bash
npx d2m@latest -u https://xkcd.com -meta basic
```
This command extracts basic metadata (title, description, keywords) and includes it in the Markdown output.
Click to view the output
```markdown
---
title: "xkcd: Exam Numbers"
---
- [Archive](/archive)
- [What If?](https://what-if.xkcd.com/)
- [About](/about)
- [Feed](/atom.xml) • [Email](/newsletter/)
- [TW](https://twitter.com/xkcd/) • [FB](https://www.facebook.com/TheXKCD/)
• [IG](https://www.instagram.com/xkcd/)
- [-Books-](/books/)
- [What If? 2](/what-if-2/)
- [WI?](/what-if/) • [TE](/thing-explainer/) • [HT](/how-to/)
 A webcomic of romance,
sarcasm, math, and language. [Special 10th anniversary edition of WHAT IF?](https://xkcd.com/what-if/) —revised and
annotated with brand-new illustrations and answers to important questions you never thought to ask—coming from
November 2024. Preorder [here](https://bit.ly/WhatIf10th) ! Exam Numbers
- [|<](/1/)
- [< Prev](/2965/)
- [Random](//c.xkcd.com/random/comic/)
- [Next >](about:blank#)
- [>|](/)
- [|<](/1/)
- [< Prev](/2965/)
- [Random](//c.xkcd.com/random/comic/)
- [Next >](about:blank#)
- [>|](/)
Permanent link to this comic: [https://xkcd.com/2966/](https://xkcd.com/2966)
Image URL (for
hotlinking/embedding): [https://imgs.xkcd.com/comics/exam_numbers.png](https://imgs.xkcd.com/comics/exam_numbers.png)
[RSS Feed](/rss.xml) - [Atom Feed](/atom.xml) - [Email](/newsletter/)
Comics I enjoy:
[Three Word Phrase](http://threewordphrase.com/) , [SMBC](https://www.smbc-comics.com/) , [Dinosaur Comics](https://www.qwantz.com/) , [Oglaf](https://oglaf.com/) (
nsfw), [A Softer World](https://www.asofterworld.com/) , [Buttersafe](https://buttersafe.com/) , [Perry Bible Fellowship](https://pbfcomics.com/) , [Questionable Content](https://questionablecontent.net/) , [Buttercup Festival](http://www.buttercupfestival.com/) , [Homestuck](https://www.homestuck.com/) , [Junior Scientist Power Hour](https://www.jspowerhour.com/)
Other things:
[Tips on technology and government](https://medium.com/civic-tech-thoughts-from-joshdata/so-you-want-to-reform-democracy-7f3b1ef10597) ,
[Climate FAQ](https://www.nytimes.com/interactive/2017/climate/what-is-climate-change.html) , [Katharine Hayhoe](https://twitter.com/KHayhoe)
xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3±1 emulated in Javascript on an Apple IIGS
at a screen resolution of 1024x1. Please enable your ad blockers, disable high-heat drying, and remove your device
from Airplane Mode and set it to Boat Mode. For security reasons, please leave caps lock on while browsing. This work is
licensed under
a [Creative Commons Attribution-NonCommercial 2.5 License](https://creativecommons.org/licenses/by-nc/2.5/).
This means you're free to copy and share these comics (but not to sell them). [More details](/license.html).
```
**4. Metadata Extraction (Extended):**
```bash
npx d2m@latest -u https://xkcd.com -meta extended
```
This command extracts extended metadata, including Open Graph, Twitter Card tags, and JSON-LD data, and includes it in
the Markdown output.
Click to view the output
```markdown
---
title: "xkcd: Exam Numbers"
openGraph:
site_name: "xkcd"
title: "Exam Numbers"
url: "https://xkcd.com/2966/"
image: "https://imgs.xkcd.com/comics/exam_numbers_2x.png"
twitter:
card: "summary_large_image"
---
- [Archive](/archive)
- [What If?](https://what-if.xkcd.com/)
- [About](/about)
- [Feed](/atom.xml) • [Email](/newsletter/)
- [TW](https://twitter.com/xkcd/) • [FB](https://www.facebook.com/TheXKCD/)
• [IG](https://www.instagram.com/xkcd/)
- [-Books-](/books/)
- [What If? 2](/what-if-2/)
- [WI?](/what-if/) • [TE](/thing-explainer/) • [HT](/how-to/)
 A webcomic of romance,
sarcasm, math, and language. [Special 10th anniversary edition of WHAT IF?](https://xkcd.com/what-if/) —revised and
annotated with brand-new illustrations and answers to important questions you never thought to ask—coming from
November 2024. Preorder [here](https://bit.ly/WhatIf10th) ! Exam Numbers
- [|<](/1/)
- [< Prev](/2965/)
- [Random](//c.xkcd.com/random/comic/)
- [Next >](about:blank#)
- [>|](/)
- [|<](/1/)
- [< Prev](/2965/)
- [Random](//c.xkcd.com/random/comic/)
- [Next >](about:blank#)
- [>|](/)
Permanent link to this comic: [https://xkcd.com/2966/](https://xkcd.com/2966)
Image URL (for
hotlinking/embedding): [https://imgs.xkcd.com/comics/exam_numbers.png](https://imgs.xkcd.com/comics/exam_numbers.png)
[RSS Feed](/rss.xml) - [Atom Feed](/atom.xml) - [Email](/newsletter/)
Comics I enjoy:
[Three Word Phrase](http://threewordphrase.com/) , [SMBC](https://www.smbc-comics.com/) , [Dinosaur Comics](https://www.qwantz.com/) , [Oglaf](https://oglaf.com/) (
nsfw), [A Softer World](https://www.asofterworld.com/) , [Buttersafe](https://buttersafe.com/) , [Perry Bible Fellowship](https://pbfcomics.com/) , [Questionable Content](https://questionablecontent.net/) , [Buttercup Festival](http://www.buttercupfestival.com/) , [Homestuck](https://www.homestuck.com/) , [Junior Scientist Power Hour](https://www.jspowerhour.com/)
Other things:
[Tips on technology and government](https://medium.com/civic-tech-thoughts-from-joshdata/so-you-want-to-reform-democracy-7f3b1ef10597) ,
[Climate FAQ](https://www.nytimes.com/interactive/2017/climate/what-is-climate-change.html) , [Katharine Hayhoe](https://twitter.com/KHayhoe)
xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3±1 emulated in Javascript on an Apple IIGS
at a screen resolution of 1024x1. Please enable your ad blockers, disable high-heat drying, and remove your device
from Airplane Mode and set it to Boat Mode. For security reasons, please leave caps lock on while browsing. This work is
licensed under
a [Creative Commons Attribution-NonCommercial 2.5 License](https://creativecommons.org/licenses/by-nc/2.5/).
This means you're free to copy and share these comics (but not to sell them). [More details](/license.html).
```
## Installation
### Using npm
```bash
npm install dom-to-semantic-markdown
```
### Using npx (CLI)
```bash
> npx d2m@latest -h
Usage: d2m [options]
Convert DOM to Semantic Markdown
Options:
-V, --version output the version number
-i, --input Input HTML file
-o, --output Output Markdown file
-e, --extract-main Extract main content
-u, --url URL to fetch HTML content from
-t, --track-table-columns Enable table column tracking for improved LLM data correlation
-meta, --include-meta-data <"basic" | "extended"> Include metadata extracted from the HTML head
-h, --help display help for command
```
## Usage
### Browser
```javascript
import {convertHtmlToMarkdown} from 'dom-to-semantic-markdown';
const markdown = convertHtmlToMarkdown(document.body);
console.log(markdown);
```
### Node.js
```javascript
import {convertHtmlToMarkdown} from 'dom-to-semantic-markdown';
import {JSDOM} from 'jsdom';
const html = '
Hello, World!
This is a test.
';
const dom = new JSDOM(html);
const markdown = convertHtmlToMarkdown(html, {overrideDOMParser: new dom.window.DOMParser()});
console.log(markdown);
```
### CLI
```bash
d2m -i input.html -o output.md # Convert input.html to output.md
d2m -u https://example.com -o output.md # Fetch and convert a webpage to Markdown
d2m -i input.html -e # Extract main content from input.html
d2m -i input.html -t # Enable table column tracking
d2m -i input.html -meta basic # Include basic metadata
d2m -i input.html -meta extended # Include extended metadata
```
## API
### `convertHtmlToMarkdown(html: string, options?: ConversionOptions): string`
Converts an HTML string to semantic Markdown.
### `convertElementToMarkdown(element: Element, options?: ConversionOptions): string`
Converts an HTML Element to semantic Markdown.
### `ConversionOptions`
* `websiteDomain?: string`: The domain of the website being converted.
* `extractMainContent?: boolean`: Whether to extract only the main content of the page.
* `refifyUrls?: boolean`: Whether to convert URLs to reference-style links.
* `debug?: boolean`: Enable debug logging.
* `overrideDOMParser?: DOMParser`: Custom DOMParser for Node.js environments.
* `enableTableColumnTracking?: boolean`: Adds unique identifiers to table columns.
* `overrideElementProcessing?: (element: Element, options: ConversionOptions, indentLevel: number) => SemanticMarkdownAST[] | undefined`:
Custom processing for HTML elements.
* `processUnhandledElement?: (element: Element, options: ConversionOptions, indentLevel: number) => SemanticMarkdownAST[] | undefined`:
Handler for unknown HTML elements.
* `overrideNodeRenderer?: (node: SemanticMarkdownAST, options: ConversionOptions, indentLevel: number) => string | undefined`:
Custom renderer for AST nodes.
* `renderCustomNode?: (node: CustomNode, options: ConversionOptions, indentLevel: number) => string | undefined`:
Renderer for custom AST nodes.
* `includeMetaData?: 'basic' | 'extended'`: Controls whether to include metadata extracted from the HTML head.
- `'basic'`: Includes standard meta tags like title, description, and keywords.
- `'extended'`: Includes basic meta tags, Open Graph tags, Twitter Card tags, and JSON-LD data.
## Using the Output with LLMs
The semantic Markdown produced by this library is optimized for use with Large Language Models (LLMs). To use it effectively:
1. Extract the Markdown content using the library.
2. Start with a brief instruction or context for the LLM.
3. Wrap the extracted Markdown in triple backticks (```).
4. Follow the Markdown with your question or prompt.
Example:
````
The following is a semantic Markdown representation of a webpage. Please analyze its content:
```markdown
{paste your extracted markdown here}
```
{your question, e.g., "What are the main points discussed in this article?"}
````
This format helps the LLM understand its task and the context of the content, enabling more accurate and relevant responses to your questions.
## Contributing
Contributions are welcome! See the [CONTRIBUTING.md](CONTRIBUTING.md) file for details.
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.