{"id":51747829,"url":"https://github.com/hyperphantasia/bookmark-harvester","last_synced_at":"2026-07-18T21:42:51.766Z","repository":{"id":344283325,"uuid":"1181043789","full_name":"hyperphantasia/bookmark-harvester","owner":"hyperphantasia","description":"Bookmark content extractor: fast, concurrent Go tool to parse bookmarks, fetch page content, and export as clean CSV.","archived":false,"fork":false,"pushed_at":"2026-04-11T14:35:21.000Z","size":4987,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-16T14:54:02.018Z","etag":null,"topics":["bookmark","bookmark-extractor","bookmarks","concurrent","content-extraction","data-collection","data-hoarding","go","golang","goquery","parallel-processing","web-archiving","web-crawling","web-scraper","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hyperphantasia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-13T17:28:50.000Z","updated_at":"2026-04-11T14:35:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hyperphantasia/bookmark-harvester","commit_stats":null,"previous_names":["brooks-code/bookmark-harvester","hyperphantasia/bookmark-harvester"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/hyperphantasia/bookmark-harvester","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fbookmark-harvester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fbookmark-harvester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fbookmark-harvester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fbookmark-harvester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hyperphantasia","download_url":"https://codeload.github.com/hyperphantasia/bookmark-harvester/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fbookmark-harvester/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35632492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-18T02:00:07.223Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bookmark","bookmark-extractor","bookmarks","concurrent","content-extraction","data-collection","data-hoarding","go","golang","goquery","parallel-processing","web-archiving","web-crawling","web-scraper","web-scraping"],"created_at":"2026-07-18T21:42:50.892Z","updated_at":"2026-07-18T21:42:51.756Z","avatar_url":"https://github.com/hyperphantasia.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bookmark content extractor\n\n**Harvesting your own knowledge base.**\n\n![Banner Image](/img/lego_antiq.jpg \"Lego diorama of the library of Alexandria.\")\n\u003cbr\u003e*LLM-crafted diorama paying homage to the Library of Alexandria, inspired by a 19th-century [artwork](https://commons.wikimedia.org/wiki/File:Ancientlibraryalex.jpg).*\n\n## What ?\n\nA Go CLI that reads bookmarks, fetches pages concurrently, extracts key content, and exports the results to CSV.\n\n## Genesis\n\n\u003e This is the first part of a broader project that aims to turn a messy bookmark collection into an analyzable CSV of content useful for organizing, summarizing, research, or simply archive bookmarks contents.\n\n## Features\n\n- Parse bookmarks from Netscape-style HTML and nested JSON export formats.\n- Remove duplicate links.\n- Concurrent HTTP fetching with worker limit and timeouts.\n- Extract headers (h1–h6), paragraphs, and meta description.\n- Skip common media file links.\n- Write results to CSV with UTF-8 encoding.\n- Progress logging and basic error counts.\n\n## Table of contents\n\n\u003cdetails\u003e\n\u003csummary\u003eContents - click to expand\u003c/summary\u003e\n\n- [Bookmark content extractor](#bookmark-content-extractor)\n  - [What ?](#what-)\n  - [Genesis](#genesis)\n  - [Features](#features)\n  - [Table of contents](#table-of-contents)\n  - [Requirements](#requirements)\n  - [Installation](#installation)\n    - [Install Golang](#install-golang)\n  - [Usage](#usage)\n  - [Configuration](#configuration)\n    - [Input files](#input-files)\n      - [HTML format](#html-format)\n        - [HTML Sample](#html-sample)\n      - [JSON format](#json-format)\n        - [JSON Sample](#json-sample)\n  - [Output](#output)\n    - [Output format](#output-format)\n      - [CSV columns](#csv-columns)\n      - [CSV sample](#csv-sample)\n  - [Improvements](#improvements)\n  - [Limitations](#limitations)\n  - [Contributing](#contributing)\n  - [License](#license)\n    - [Acknowledgements](#acknowledgements)\n\n\u003c/details\u003e\n\n## Requirements\n\n- Go 1.23.6+ (or latest stable supported by your environment).\n- External dependency: *github.com/PuerkitoBio/goquery*\n\n## Installation\n\n### Install Golang\n\n1. Download the [installer](https://go.dev/dl/) for your OS.\n\n2. Follow the platform instructions:\n   - **macOS**: use the pkg installer or Homebrew (`brew install go`).\n   - **Windows**: run the MSI installer.\n   - **Linux**: extract the tarball to `/usr/local` and add `/usr/local/go/bin` to your `PATH`.\n\n   ```bash\n    rm -rf /usr/local/go \u0026\u0026 tar -C /usr/local -xzf goX.XX.X.linux-amd64.tar.gz\n   ```\n\n3. Add `GOPATH/bin` to your environment:\n\n     ```bash\n     export PATH=$PATH:$(go env GOPATH)/bin\n     ```\n\n   - *Optionally*, set `GOPROXY` to speed module downloads:\n\n     ```bash\n     export GOPROXY=https://proxy.golang.org,direct\n     ```\n\n4. Verify installation:\n\n   ```bash\n   go version\n   ```\n\n   Expect an output like `go version goX.XX.X linux/amd64`.\n\n5. Clone the repository:\n\n   ```bash\n   git clone https://github.com/brooks-code/bookmark-harvester.git\n   cd bookmark-harvester\n   ```\n\n6. Initialize modules (if not already initialized) and download dependencies:\n\n   ```bash\n   go mod download\n   ```\n\n   or\n\n   ```bash\n   go mod tidy\n   ```\n\n7. Build the binary:\n\n   ```bash\n   go build -o bookmarks ./cmd\n   ```\n\n## Usage\n\nDefault behavior expects a folder named `bookmarks` with input bookmark files stored.\n\nRun the program:\n\n```bash\n./bookmarks\n```\n\n## Configuration\n\n| **Constant** | **Default value** | **Description** |\n| --- | --- | --- |\n| Input directory | `bookmarks` | directory for `.html` and `.json` exports |\n| CSV Output | `bookmarks_dataset.csv` | default output file for CSV data (written in working directory) |\n| Worker/Concurrency | 20 | maximum number of concurrent workers (`maxWorkers`) |\n| Connection establishment timeout | 5 | seconds to establish a connection (`HTTPClient.Timeout`) |\n| Request/response timeout | 4 | seconds to get a response from the server (`Dialer.Timeout`)|\n| Keep connection timeout | 30 | seconds for keeping the connection alive (`Dialer.KeepAlive`)|\n\n### Input files\n\nThe script accepts two types of bookmark files.\n\n#### HTML format\n\nThe usual Netscape format used by Firefox as well. More [details](https://gist.github.com/apfelchips/749b121f27d62d3cef78a45342c5c967).\n\n##### HTML Sample\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand\u003c/summary\u003e\n  \n```html\n\u003c!DOCTYPE NETSCAPE-Bookmark-file-1\u003e\n\u003c!-- This is an automatically generated file.\n     It will be read and overwritten.\n     DO NOT EDIT! --\u003e\n\u003cMETA HTTP-EQUIV=\"Content-Type\" CONTENT=\"text/html; charset=UTF-8\"\u003e\n\u003cmeta http-equiv=\"Content-Security-Policy\"\n      content=\"default-src 'self'; script-src 'none'; img-src data: *; object-src 'none'\"\u003e\u003c/meta\u003e\n\u003cTITLE\u003eBookmarks\u003c/TITLE\u003e\n\u003cH1\u003eBookmarks Menu\u003c/H1\u003e\n\n\u003cDL\u003e\u003cp\u003e\n    \u003cDT\u003e\u003cH3 ADD_DATE=\"1702154335\" LAST_MODIFIED=\"1702217811\"\u003eMozilla Firefox\u003c/H3\u003e\n    \u003cDL\u003e\u003cp\u003e\n    \u003c/DL\u003e\u003cp\u003e\n    \u003cDT\u003e\u003cH3 ADD_DATE=\"1702154335\" LAST_MODIFIED=\"1718594944\" PERSONAL_TOOLBAR_FOLDER=\"true\"\u003eBookmarks Toolbar\u003c/H3\u003e\n    \u003cDL\u003e\u003cp\u003e\n        \u003cDT\u003e\u003cA HREF=\"https://learn.microsoft.com/en-us/windows/wsl/setup/environment?source=recommendations\" ADD_DATE=\"1702155216\" LAST_MODIFIED=\"1702155216\"\u003eSet up a WSL development environment | Microsoft Learn\u003c/A\u003e\n```\n\u003c/details\u003e\n\n#### JSON format\n\nA JSON file that represents a bookmark data structure in the Firefox object formats.\nOne bookmark record exists for each bookmark item, where an item may actually be a folder or a separator. More [details](https://mozilla-services.readthedocs.io/en/latest/sync/objectformats.html#bookmarks)\n\n##### JSON Sample\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n```json\n{\n  \"guid\": \"root________\",\n  \"title\": \"\",\n  \"index\": 0,\n  \"dateAdded\": 1548500403406000,\n  \"lastModified\": 1577923579000000,\n  \"id\": 1,\n  \"typeCode\": 2,\n  \"type\": \"text/x-moz-place-container\",\n  \"root\": \"placesRoot\",\n  \"children\": [\n    {\n      \"guid\": \"menu________\",\n      \"title\": \"Menu des marque-pages\",\n      \"index\": 0,\n      \"dateAdded\": 1548500403406000,\n      \"lastModified\": 1534696399488000,\n      \"id\": 2,\n      \"typeCode\": 2,\n      \"type\": \"text/x-moz-place-container\",\n      \"root\": \"bookmarksMenuFolder\"\n    },\n    {\n      \"guid\": \"toolbar_____\",\n      \"title\": \"Barre personnelle\",\n      \"index\": 1,\n      \"dateAdded\": 1548500403406000,\n      \"lastModified\": 1548500454183000,\n      \"id\": 3,\n      \"typeCode\": 2,\n      \"type\": \"text/x-moz-place-container\",\n      \"root\": \"toolbarFolder\"\n    },\n    {\n      \"guid\": \"unfiled_____\",\n      \"title\": \"Autres marque-pages\",\n      \"index\": 3,\n      \"dateAdded\": 1548500403406000,\n      \"lastModified\": 1577923579000000,\n      \"id\": 5,\n      \"typeCode\": 2,\n      \"type\": \"text/x-moz-place-container\",\n      \"root\": \"unfiledBookmarksFolder\",\n      \"children\": [\n        {\n          \"guid\": \"Efhs0vI9z8za\",\n          \"title\": \"Salvador Dali On Creativity « Creative Creativity\",\n          \"index\": 0,\n          \"dateAdded\": 1548361994582000,\n          \"lastModified\": 1548361994582000,\n          \"id\": 8,\n          \"typeCode\": 1,\n          \"type\": \"text/x-moz-place\",\n          \"uri\": \"https://creativecreativity.com/2007/05/27/salvador_dali_o/\"\n        }\n      ]\n    }\n  ]\n}\n```\n\n\u003c/details\u003e\n\n## Output\n\nAfter completion, open `bookmarks_dataset.csv`.\n\n### Output format\n\nAll text fields are cleaned to remove excess whitespace and newlines.\n\n#### CSV columns\n\n| **Field** | **Type** | **Description** |\n| --- | --- | --- |\n| **title** | string | page title or link text |\n| **link** | string (URL) | URL |\n| **add_date** | integer (timestamp) | original bookmark dateAdded/add_date if present |\n| **last_modified** | integer (timestamp) | original lastModified/last_modified if present |\n| **last_checked** | date (YYYY-MM-DD) | date the program last checked the URL |\n| **active** | boolean | \"yes\" or \"no\" depending on fetch and status |\n| **h_tag_content** | string | concatenated header text *(h1–h6)* |\n| **p**| string | concatenated paragraph text |\n| **meta_description** | string | content of `meta[name=\"description\"]` (cleaned) |\n\n#### CSV sample\n\n```csv\n\"title\",\"link\",\"add_date\",\"last_modified\",\"last_checked\",\"active\",\"h_tag_content\",\"p\",\"meta_description\"\n\"On Set VFX Tips: Cross Polarization Photography and Skin Texures\",\"http://onsetvfxtips.blogspot.com/2009/06/cross-polarization-photography-and-skin.html\",\"1225932202224000\",\"1225942202246000\",\"2025-01-14\",\"yes\",\"On Set VFX Tips Tuesday, June 30, 2009 Cross Polarization Photography and Skin Texures 2 comments: Search This Blog Subscribe To VFX Tips Blog Archive Labels About Me LInks\",\"A site dedicated to the art of On Set VFX Supervision. I will be sharing all the tips and tricks I learn along the way. This site will include interviews of on set VFX supervisors, equipment reviews, and other fun stuff. Interesting stuff! Thanks for posting. Yes, great. Thanks! Note: Only a member of this blog may post a comment.|A site dedicated to the art of On Set VFX Supervision. I will be sharing all the tips and tricks I learn along the way. This site will include interviews of on set VFX supervisors, equipment reviews, and other fun stuff. Interesting stuff! Thanks for posting. Yes, great. Thanks! Note: Only a member of this blog may post a comment. Note: Only a member of this blog may post a comment.\",\"\"\n```\n\n## Improvements\n\n1. Implement command-line options:\n\n- `--bookmarks-dir PATH` :     path to bookmarks folder (*default: ./bookmarks*)\n- `--output-csv PATH` :        output CSV file path (*default: bookmarks_dataset.csv*)\n- `--workers N` :              maximum concurrent workers (*default: 20*)\n- `--http-timeout SECONDS` :   HTTP client timeout (*default: 4*)\n- `--request-timeout SECONDS` : per-request overall timeout (*default: 30*)\n- `--user-agent STRING` :      user-Agent header to use for requests\n\n2. Validation and sanitization:\n\n- More complete validation and sanitization of URLs: skip unsupported schemes (*file:, javascript:, data:*)\n\n\u003e [!IMPORTANT]\n\u003e Respect `robots.txt` and crawl politeness if running at scale.\n\n## Limitations\n\n- HTTP client is global and reused configuring Transport options like `MaxIdleConns`, `IdleConnTimeout`, and `MaxIdleConnsPerHost` should be considered.\n- Concurrency is limited by a buffered semaphore and `sync.WaitGroup`. Consider refactoring to use `errgroup` and propagate contexts.\n- Timeouts: HTTP client timeout is separate from per-request context timeout.\n\n## Contributing\n\nContributions are **welcome**. You can follow the usual steps detailed in the [CONTRIBUTING](/CONTRIBUTING.md) section.\n\n## License\n\nThe source code is provided under the [Unlicense](https://unlicense.org) license. See the [LICENSE](/LICENSE) file for details.\n\n### Acknowledgements\n\n- *[Martin Angers](https://www.github.com/PuerkitoBio)* for its `goquery` module.\n🀐\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperphantasia%2Fbookmark-harvester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhyperphantasia%2Fbookmark-harvester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperphantasia%2Fbookmark-harvester/lists"}