https://github.com/hyperphantasia/bookmark-harvester
Bookmark content extractor: fast, concurrent Go tool to parse bookmarks, fetch page content, and export as clean CSV.
https://github.com/hyperphantasia/bookmark-harvester
bookmark bookmark-extractor bookmarks concurrent content-extraction data-collection data-hoarding go golang goquery parallel-processing web-archiving web-crawling web-scraper web-scraping
Last synced: 4 days ago
JSON representation
Bookmark content extractor: fast, concurrent Go tool to parse bookmarks, fetch page content, and export as clean CSV.
- Host: GitHub
- URL: https://github.com/hyperphantasia/bookmark-harvester
- Owner: hyperphantasia
- License: unlicense
- Created: 2026-03-13T17:28:50.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-04-11T14:35:21.000Z (3 months ago)
- Last Synced: 2026-05-16T14:54:02.018Z (2 months ago)
- Topics: bookmark, bookmark-extractor, bookmarks, concurrent, content-extraction, data-collection, data-hoarding, go, golang, goquery, parallel-processing, web-archiving, web-crawling, web-scraper, web-scraping
- Language: Go
- Homepage:
- Size: 4.76 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# Bookmark content extractor
**Harvesting your own knowledge base.**
*LLM-crafted diorama paying homage to the Library of Alexandria, inspired by a 19th-century [artwork](https://commons.wikimedia.org/wiki/File:Ancientlibraryalex.jpg).*
## What ?
A Go CLI that reads bookmarks, fetches pages concurrently, extracts key content, and exports the results to CSV.
## Genesis
> This is the first part of a broader project that aims to turn a messy bookmark collection into an analyzable CSV of content useful for organizing, summarizing, research, or simply archive bookmarks contents.
## Features
- Parse bookmarks from Netscape-style HTML and nested JSON export formats.
- Remove duplicate links.
- Concurrent HTTP fetching with worker limit and timeouts.
- Extract headers (h1–h6), paragraphs, and meta description.
- Skip common media file links.
- Write results to CSV with UTF-8 encoding.
- Progress logging and basic error counts.
## Table of contents
Contents - click to expand
- [Bookmark content extractor](#bookmark-content-extractor)
- [What ?](#what-)
- [Genesis](#genesis)
- [Features](#features)
- [Table of contents](#table-of-contents)
- [Requirements](#requirements)
- [Installation](#installation)
- [Install Golang](#install-golang)
- [Usage](#usage)
- [Configuration](#configuration)
- [Input files](#input-files)
- [HTML format](#html-format)
- [HTML Sample](#html-sample)
- [JSON format](#json-format)
- [JSON Sample](#json-sample)
- [Output](#output)
- [Output format](#output-format)
- [CSV columns](#csv-columns)
- [CSV sample](#csv-sample)
- [Improvements](#improvements)
- [Limitations](#limitations)
- [Contributing](#contributing)
- [License](#license)
- [Acknowledgements](#acknowledgements)
## Requirements
- Go 1.23.6+ (or latest stable supported by your environment).
- External dependency: *github.com/PuerkitoBio/goquery*
## Installation
### Install Golang
1. Download the [installer](https://go.dev/dl/) for your OS.
2. Follow the platform instructions:
- **macOS**: use the pkg installer or Homebrew (`brew install go`).
- **Windows**: run the MSI installer.
- **Linux**: extract the tarball to `/usr/local` and add `/usr/local/go/bin` to your `PATH`.
```bash
rm -rf /usr/local/go && tar -C /usr/local -xzf goX.XX.X.linux-amd64.tar.gz
```
3. Add `GOPATH/bin` to your environment:
```bash
export PATH=$PATH:$(go env GOPATH)/bin
```
- *Optionally*, set `GOPROXY` to speed module downloads:
```bash
export GOPROXY=https://proxy.golang.org,direct
```
4. Verify installation:
```bash
go version
```
Expect an output like `go version goX.XX.X linux/amd64`.
5. Clone the repository:
```bash
git clone https://github.com/brooks-code/bookmark-harvester.git
cd bookmark-harvester
```
6. Initialize modules (if not already initialized) and download dependencies:
```bash
go mod download
```
or
```bash
go mod tidy
```
7. Build the binary:
```bash
go build -o bookmarks ./cmd
```
## Usage
Default behavior expects a folder named `bookmarks` with input bookmark files stored.
Run the program:
```bash
./bookmarks
```
## Configuration
| **Constant** | **Default value** | **Description** |
| --- | --- | --- |
| Input directory | `bookmarks` | directory for `.html` and `.json` exports |
| CSV Output | `bookmarks_dataset.csv` | default output file for CSV data (written in working directory) |
| Worker/Concurrency | 20 | maximum number of concurrent workers (`maxWorkers`) |
| Connection establishment timeout | 5 | seconds to establish a connection (`HTTPClient.Timeout`) |
| Request/response timeout | 4 | seconds to get a response from the server (`Dialer.Timeout`)|
| Keep connection timeout | 30 | seconds for keeping the connection alive (`Dialer.KeepAlive`)|
### Input files
The script accepts two types of bookmark files.
#### HTML format
The usual Netscape format used by Firefox as well. More [details](https://gist.github.com/apfelchips/749b121f27d62d3cef78a45342c5c967).
##### HTML Sample
Click to expand
```html
Bookmarks
Bookmarks Menu
-
Mozilla Firefox
-
Bookmarks Toolbar
#### JSON format
A JSON file that represents a bookmark data structure in the Firefox object formats.
One bookmark record exists for each bookmark item, where an item may actually be a folder or a separator. More [details](https://mozilla-services.readthedocs.io/en/latest/sync/objectformats.html#bookmarks)
##### JSON Sample
Click to expand
```json
{
"guid": "root________",
"title": "",
"index": 0,
"dateAdded": 1548500403406000,
"lastModified": 1577923579000000,
"id": 1,
"typeCode": 2,
"type": "text/x-moz-place-container",
"root": "placesRoot",
"children": [
{
"guid": "menu________",
"title": "Menu des marque-pages",
"index": 0,
"dateAdded": 1548500403406000,
"lastModified": 1534696399488000,
"id": 2,
"typeCode": 2,
"type": "text/x-moz-place-container",
"root": "bookmarksMenuFolder"
},
{
"guid": "toolbar_____",
"title": "Barre personnelle",
"index": 1,
"dateAdded": 1548500403406000,
"lastModified": 1548500454183000,
"id": 3,
"typeCode": 2,
"type": "text/x-moz-place-container",
"root": "toolbarFolder"
},
{
"guid": "unfiled_____",
"title": "Autres marque-pages",
"index": 3,
"dateAdded": 1548500403406000,
"lastModified": 1577923579000000,
"id": 5,
"typeCode": 2,
"type": "text/x-moz-place-container",
"root": "unfiledBookmarksFolder",
"children": [
{
"guid": "Efhs0vI9z8za",
"title": "Salvador Dali On Creativity « Creative Creativity",
"index": 0,
"dateAdded": 1548361994582000,
"lastModified": 1548361994582000,
"id": 8,
"typeCode": 1,
"type": "text/x-moz-place",
"uri": "https://creativecreativity.com/2007/05/27/salvador_dali_o/"
}
]
}
]
}
```
## Output
After completion, open `bookmarks_dataset.csv`.
### Output format
All text fields are cleaned to remove excess whitespace and newlines.
#### CSV columns
| **Field** | **Type** | **Description** |
| --- | --- | --- |
| **title** | string | page title or link text |
| **link** | string (URL) | URL |
| **add_date** | integer (timestamp) | original bookmark dateAdded/add_date if present |
| **last_modified** | integer (timestamp) | original lastModified/last_modified if present |
| **last_checked** | date (YYYY-MM-DD) | date the program last checked the URL |
| **active** | boolean | "yes" or "no" depending on fetch and status |
| **h_tag_content** | string | concatenated header text *(h1–h6)* |
| **p**| string | concatenated paragraph text |
| **meta_description** | string | content of `meta[name="description"]` (cleaned) |
#### CSV sample
```csv
"title","link","add_date","last_modified","last_checked","active","h_tag_content","p","meta_description"
"On Set VFX Tips: Cross Polarization Photography and Skin Texures","http://onsetvfxtips.blogspot.com/2009/06/cross-polarization-photography-and-skin.html","1225932202224000","1225942202246000","2025-01-14","yes","On Set VFX Tips Tuesday, June 30, 2009 Cross Polarization Photography and Skin Texures 2 comments: Search This Blog Subscribe To VFX Tips Blog Archive Labels About Me LInks","A site dedicated to the art of On Set VFX Supervision. I will be sharing all the tips and tricks I learn along the way. This site will include interviews of on set VFX supervisors, equipment reviews, and other fun stuff. Interesting stuff! Thanks for posting. Yes, great. Thanks! Note: Only a member of this blog may post a comment.|A site dedicated to the art of On Set VFX Supervision. I will be sharing all the tips and tricks I learn along the way. This site will include interviews of on set VFX supervisors, equipment reviews, and other fun stuff. Interesting stuff! Thanks for posting. Yes, great. Thanks! Note: Only a member of this blog may post a comment. Note: Only a member of this blog may post a comment.",""
```
## Improvements
1. Implement command-line options:
- `--bookmarks-dir PATH` : path to bookmarks folder (*default: ./bookmarks*)
- `--output-csv PATH` : output CSV file path (*default: bookmarks_dataset.csv*)
- `--workers N` : maximum concurrent workers (*default: 20*)
- `--http-timeout SECONDS` : HTTP client timeout (*default: 4*)
- `--request-timeout SECONDS` : per-request overall timeout (*default: 30*)
- `--user-agent STRING` : user-Agent header to use for requests
2. Validation and sanitization:
- More complete validation and sanitization of URLs: skip unsupported schemes (*file:, javascript:, data:*)
> [!IMPORTANT]
> Respect `robots.txt` and crawl politeness if running at scale.
## Limitations
- HTTP client is global and reused configuring Transport options like `MaxIdleConns`, `IdleConnTimeout`, and `MaxIdleConnsPerHost` should be considered.
- Concurrency is limited by a buffered semaphore and `sync.WaitGroup`. Consider refactoring to use `errgroup` and propagate contexts.
- Timeouts: HTTP client timeout is separate from per-request context timeout.
## Contributing
Contributions are **welcome**. You can follow the usual steps detailed in the [CONTRIBUTING](/CONTRIBUTING.md) section.
## License
The source code is provided under the [Unlicense](https://unlicense.org) license. See the [LICENSE](/LICENSE) file for details.
### Acknowledgements
- *[Martin Angers](https://www.github.com/PuerkitoBio)* for its `goquery` module.
🀐