{"id":13592291,"url":"https://github.com/alphapapa/org-web-tools","last_synced_at":"2026-01-27T03:36:53.323Z","repository":{"id":45937816,"uuid":"97963919","full_name":"alphapapa/org-web-tools","owner":"alphapapa","description":"View, capture, and archive Web pages in Org-mode","archived":false,"fork":false,"pushed_at":"2024-02-10T08:03:29.000Z","size":154,"stargazers_count":657,"open_issues_count":23,"forks_count":34,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-02-24T23:29:37.767Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Emacs Lisp","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alphapapa.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-21T15:46:40.000Z","updated_at":"2025-02-21T15:53:52.000Z","dependencies_parsed_at":"2022-08-12T12:40:20.987Z","dependency_job_id":"881bce7f-2c27-4c32-935a-16b6f91d823a","html_url":"https://github.com/alphapapa/org-web-tools","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/alphapapa/org-web-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphapapa%2Forg-web-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphapapa%2Forg-web-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphapapa%2Forg-web-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphapapa%2Forg-web-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alphapapa","download_url":"https://codeload.github.com/alphapapa/org-web-tools/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphapapa%2Forg-web-tools/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28799841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T01:07:07.743Z","status":"online","status_checked_at":"2026-01-27T02:00:07.755Z","response_time":168,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T16:01:07.719Z","updated_at":"2026-01-27T03:36:53.309Z","avatar_url":"https://github.com/alphapapa.png","language":"Emacs Lisp","funding_links":[],"categories":["Web capture","Emacs Lisp"],"sub_categories":[],"readme":"#+TITLE: org-web-tools\n#+PROPERTY: LOGGING nil\n\n[[https://melpa.org/#/org-web-tools][file:https://melpa.org/packages/org-web-tools-badge.svg]] [[https://stable.melpa.org/#/org-web-tools][file:https://stable.melpa.org/packages/org-web-tools-badge.svg]]\n\nThis file contains library functions and commands useful for retrieving web page content and processing it into Org-mode content.\n\nFor example, you can copy a URL to the clipboard or kill-ring, then run a command that downloads the page, isolates the \"readable\" content with =eww-readable=, converts it to Org-mode content with Pandoc, and displays it in an Org-mode buffer.  Another command does all of that but inserts it as an Org entry instead of displaying it in a new buffer.\n\n* Installation                                                   :noexport_1:\n\n** Requirements\n\n+ Emacs 27.1 or later.\n+ Commands that process HTML into Org require [[https://pandoc.org/][Pandoc]].  *Note:* The output of current Pandoc versions differs substantially from versions that may still be present in stable Linux distros.  If you encounter any issues, please install a more recent version of Pandoc.\n\n** MELPA\n\nAfter installing from MELPA, just run one of the [[*Usage][commands]] below.  If you want to use any of the functions in your own code, you should ~(require 'org-web-tools)~.\n\n* Usage                                                          :noexport_1:\n\n** Commands\n\n+  =org-web-tools-insert-link-for-url=: Insert an Org-mode link to the URL in the clipboard or kill-ring.  Downloads the page to get the HTML title.\n+  =org-web-tools-insert-web-page-as-entry=: Insert the web page for the URL in the clipboard or kill-ring as an Org-mode entry, as a sibling heading of the current entry.\n+  =org-web-tools-read-url-as-org=: Display the web page for the URL in the clipboard or kill-ring as Org-mode text in a new buffer, processed with =eww-readable=.\n+  =org-web-tools-convert-links-to-page-entries=: Convert all URLs and Org links in current Org entry to Org headings, each containing the web page content of that URL, converted to Org-mode text and processed with =eww-readable=.  This should be called on an entry that solely contains a list of URLs or links.\n+  ~org-web-tools-archive-attach~: Download archive of page at URL and attach with =org-attach=.  If =CHOOSE-FN= is non-nil (interactively, with universal prefix), prompt for the archive function to use.  If =VIEW= is non-nil (interactively, with two universal prefixes), view the archive immediately after attaching.  (See also [[https://github.com/scallywag/org-board][org-board]]).\n+  ~org-web-tools-archive-view~: Open Zip file archive of web page. Extracts to a temp directory and opens with ~browse-url-default-browser~.  Note: the extracted files are left on-disk in the temp directory.\n\n** Functions\n\n These are used in the commands above and may be useful in building your own commands.\n\n+  =org-web-tools--dom-to-html=: Return parsed HTML DOM as an HTML string. Note: This is an approximation and is not necessarily correct HTML (e.g. IMG tags may be rendered with a closing \"\u003c/img\u003e\" tag).\n+  =org-web-tools--eww-readable=: Return \"readable\" part of HTML with title.\n+  =org-web-tools--get-url=: Return content for URL as string.\n+  =org-web-tools--html-to-org-with-pandoc=: Return string of HTML converted to Org with Pandoc.  When SELECTOR is non-nil, the HTML is filtered using =esxml-query= SELECTOR and re-rendered to HTML with =org-web-tools--dom-to-html=, which see.\n+  =org-web-tools--url-as-readable-org=: Return string containing Org entry of URL's web page content.  Content is processed with =eww-readable= and Pandoc.  Entry will be a top-level heading, with article contents below a second-level \"Article\" heading, and a timestamp in the first-level entry for writing comments.\n+  =org-web-tools--demote-headings-below=: Demote all headings in buffer so the highest level is below LEVEL.\n+  =org-web-tools--get-first-url=: Return URL in clipboard, or first URL in the kill-ring, or nil if none.\n+  ~org-web-tools--read-url~: Return a URL by searching at point, then in clipboard, then in kill-ring, and finally prompting the user.\n+  =org-web-tools--read-org-bracket-link=: Return (TARGET . DESCRIPTION) for Org bracket LINK or next link on current line.\n+  =org-web-tools--remove-dos-crlf=: Remove all DOS CRLF (^M) in buffer.\n\n* Changelog                                                      :noexport_1:\n\n** 1.3\n\n*Changes*\n+ Errors from Pandoc are now displayed.  ([[https://github.com/alphapapa/org-web-tools/pull/47][#47]].  Thanks to [[https://github.com/c1-g][c1-g]].)\n\n*Fixes*\n+ Default options to Wget (see [[https://github.com/alphapapa/org-web-tools/issues/35][#35]]).\n+ Finding URL in clipboard on MacOS and Windows.  (See [[https://github.com/alphapapa/org-web-tools/pull/66][#66]].  Thanks to [[https://github.com/askdkc][@askdkc]].)\n+ Org timestamp format when inserting pages.  ([[https://github.com/alphapapa/org-web-tools/pull/54][#54]].  Thanks to [[https://github.com/p4v4n][p4v4n]] for reporting.)\n\n*Internal*\n+ Use ~plz~ HTTP library and make various related optimizations.\n\n*Removed*\n+ Internal function ~org-web-tools--html-title~.  (If your program used this function, it's trivially reimplemented; see source code.)\n\n** 1.2\n\n*Improvements*\n+ Archiving tools:\n  - Can use multiple functions to attempt archiving.\n  - Associated options control retry attempts, delays, and fallbacks to other functions.\n  - Functions to archive Web pages with =wget= and =tar=:\n    + Function ~org-web-tools-archive--wget-tar~ archives a URL's Web page, including page resources.\n    + Function =org-web-tools-archive--wget-tar-html-only= archives a URL's HTML only.\n  - Command ~org-web-tools-archive-view~ handles both =zip= and =tar= archives.\n  - The default settings use =wget= and =tar= to archive pages (because the ~archive.today~ service has not worked reliably with external tools for a long time).\n\n*Changes*\n+ Option ~org-web-tools-archive-fn~ defaults to using ~wget~ and ~tar~ to archive pages to XZ archives with HTML and page resources.  (The ~archive.is~ service has not worked reliably with other tools for a long time.)\n\n*Fixes*\n+ =org-web-tools--org-link-for-url= now returns the URL if the HTML page has no title tag.  This avoids an error, e.g. when used in an Org capture template.\n\n*Compatibility*\n+ Emacs 27.1 or later is now required.\n+ Updated for Org 9.3's changes to ~org-bracket-link-regexp~.  (Thanks to [[https://github.com/bcc32][Aaron Zeng]] and [[https://github.com/akirak][Akira Komamura]].)\n+ Activate ~org-mode~ in temporary buffer for ~org-web-tools--html-to-org-with-pandoc~.  ([[https://github.com/alphapapa/org-web-tools/issues/56][#56]].  Thanks to [[https://github.com/mooseyboots][mooseyboots]].)\n+ Use ~compat~ library.\n\n** 1.1.2\n\n*Fixed*\n+  Only test non-nil items in ~org-web-tools--get-first-url~.  This makes it work properly in non-GUI Emacs sessions.  (Thanks to [[https://github.com/bsima][Ben Sima]] for reporting.)\n\n** 1.1.1\n\n*Fixed*\n+  Require ~org-attach~.\n\n** 1.1\n\n*Additions*\n+  Command ~org-web-tools-attach-url-archive~.\n+  Command ~org-web-tools-view-archive~.\n+  Function ~org-web-tools--read-url~.\n\n** 1.0.1\n\n*Changes*\n+  Remove all property drawers that contain the =CUSTOM_ID= property from Pandoc output.\n\n** 1.0\n\n+ First declared stable release.\n\n* Development                                                    :noexport_1:\n\nContributions and suggestions are welcome.\n\n* License                                                          :noexport:\n\nGPLv3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falphapapa%2Forg-web-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falphapapa%2Forg-web-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falphapapa%2Forg-web-tools/lists"}