{"id":13681374,"url":"https://github.com/lapwat/papeer","last_synced_at":"2026-04-23T10:02:15.617Z","repository":{"id":43212378,"uuid":"406510993","full_name":"lapwat/papeer","owner":"lapwat","description":"Scrape the web in the eink era. Convert websites into ebooks and markdown.","archived":false,"fork":false,"pushed_at":"2026-03-23T10:01:51.000Z","size":682,"stargazers_count":335,"open_issues_count":4,"forks_count":25,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-03-24T07:42:13.603Z","etag":null,"topics":["cli","command-line","command-line-tool","ebook","eink","epub","ereader","kindle","markdown","mobi","remarkable","remarkable-tablet","scraper"],"latest_commit_sha":null,"homepage":"https://papeer.tech","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lapwat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-09-14T20:23:33.000Z","updated_at":"2026-03-23T10:01:55.000Z","dependencies_parsed_at":"2022-09-10T11:32:00.865Z","dependency_job_id":"a00771e7-3720-42de-8400-409a9cb4be50","html_url":"https://github.com/lapwat/papeer","commit_stats":{"total_commits":53,"total_committers":2,"mean_commits":26.5,"dds":"0.018867924528301883","last_synced_commit":"4521497d1229358b7a4f5249ff6ec417d30e9d23"},"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"purl":"pkg:github/lapwat/papeer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapwat%2Fpapeer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapwat%2Fpapeer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapwat%2Fpapeer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapwat%2Fpapeer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lapwat","download_url":"https://codeload.github.com/lapwat/papeer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lapwat%2Fpapeer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32175040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-23T02:19:40.750Z","status":"ssl_error","status_checked_at":"2026-04-23T02:17:55.737Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","command-line","command-line-tool","ebook","eink","epub","ereader","kindle","markdown","mobi","remarkable","remarkable-tablet","scraper"],"created_at":"2024-08-02T13:01:30.016Z","updated_at":"2026-04-23T10:02:15.612Z","avatar_url":"https://github.com/lapwat.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cimg src=\"logo.png\" alt=\"Papeer\"\u003e\n  \u003cbr\u003e\n  Papeer\n\u003c/h1\u003e\n\n\u003ch4 align=\"center\"\u003eWeb scraper for ereaders\u003c/h4\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#features\"\u003eFeatures\u003c/a\u003e •\n  \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e •\n  \u003ca href=\"#how-to-use\"\u003eHow To Use\u003c/a\u003e\n\u003c/p\u003e\n\n  \u003cimg src=\"terminal.gif\" alt=\"Papeer\"\u003e\n\n\n## Features\n\n* Scrape websites and RSS feeds\n* Keep relevant content only\n  - Formatted text (bold, italic, links)\n  - Images\n* Save websites as Markdown, HTML, EPUB or MOBI files\n* Use it as a an HTTP proxy\n* Cross platform\n  - Windows, MacOS and Linux ready\n\n# Installation\n\n## From source\n\n```sh\ngo install github.com/lapwat/papeer@latest\n```\n\n## From binary\n\nDownload [latest release](https://github.com/lapwat/papeer/releases/latest) for Windows, MacOS (darwin) and Linux.\n\n## MOBI support\n\n\u003e Kindle e-readers now support EPUB format\n\nInstall kindlegen to export websites to Kindle compatible ebooks, Linux only.\n\n```sh\nTMPDIR=$(mktemp -d -t papeer-XXXXX)\ncurl -L https://github.com/lapwat/papeer/releases/download/kindlegen/kindlegen_linux_2.6_i386_v2_9.tar.gz \u003e $TMPDIR/kindlegen.tar.gz\ntar xzvf $TMPDIR/kindlegen.tar.gz -C $TMPDIR\nchmod +x $TMPDIR/kindlegen\nsudo mv $TMPDIR/kindlegen /usr/local/bin\nrm -rf $TMPDIR\n```\n\nNow you can use `--format=mobi` in your `get` command.\n\n## How To Use\n\n### Scrape a single page\n\n```sh\npapeer get URL\n```\n\nThe `get` command let's you retrieve the content of a web page.\n\nIt removes ads and menus with `go-readability`, keeping only formatted text and images.\n\nYou can chain URLs.\n\n**Options**\n\n```sh\n-a, --author string      book author\n    --delay int          time in milliseconds to wait before downloading next chapter, use with depth/selector (default -1)\n-d, --depth int          scraping depth\n-f, --format string      file format [md, json, html, epub, mobi] (default \"md\")\n-h, --help               help for get\n    --images             retrieve images only\n-i, --include            include URL as first chapter, use with depth/selector\n-l, --limit int          limit number of chapters, use with depth/selector (default -1)\n-n, --name string        book name (default: page title)\n-o, --offset int         skip first chapters, use with depth/selector\n    --output string      file name (default: book name)\n    --print-url          print url after chapter title\n-q, --quiet              hide progress bar\n-r, --reverse            reverse chapter order\n-s, --selector strings   table of contents CSS selector\n    --stdout             print to standard output\n-t, --threads int        download concurrency, use with depth/selector (default -1)\n    --use-link-name      use link name for chapter title\n```\n\n### Scrape a whole website recursively\n\n**Display the table of contents**\n\nBefore scraping a whole website, it is a good idea to use the `list` command. This command is like a _dry run_, **which lets you visualize the content before retrieving it**.\n\nYou can use several options to customize the table of contents extraction, such as `selector`, `limit`, `offset`, `reverse` and `include`. Type `papeer list --help` for more information about those options.\n\nThe selector option should point to **`\u003ca\u003e` HTML tags**. If you don't specify it, the `selector` will be automatically determined based on the links present on the page.\n\n```sh\npapeer list https://12factor.net/ --selector='section.concrete\u003earticle\u003eh2\u003ea'\n```\n\n```\n #  NAME                    URL                                    \n 1  I. Codebase             https://12factor.net/codebase          \n 2  II. Dependencies        https://12factor.net/dependencies      \n 3  III. Config             https://12factor.net/config            \n 4  IV. Backing services    https://12factor.net/backing-services  \n 5  V. Build, release, run  https://12factor.net/build-release-run \n 6  VI. Processes           https://12factor.net/processes         \n 7  VII. Port binding       https://12factor.net/port-binding      \n 8  VIII. Concurrency       https://12factor.net/concurrency       \n 9  IX. Disposability       https://12factor.net/disposability     \n10  X. Dev/prod parity      https://12factor.net/dev-prod-parity   \n11  XI. Logs                https://12factor.net/logs              \n12  XII. Admin processes    https://12factor.net/admin-processes\n```\n\n**Scrape the content**\n\nOnce you are satisfied with the table of contents listed by the `list` command, you can scrape the content of those pages with the `get` command. You can use the same options that you specified for the `list` command.\n\n```sh\npapeer get https://12factor.net/ --selector='section.concrete\u003earticle\u003eh2\u003ea'\n```\n\n```\n[===\u003e-----------------------------] Chapters 7 / 12\n[=================================] 1. I. Codebase\n[=================================] 2. II. Dependencies\n[=================================] 3. III. Config\n[=================================] 4. IV. Backing services\n[=================================] 5. V. Build, release, run\n[=================================] 6. VI. Processes\n[=================================] 7. VII. Port binding\n[---------------------------------] 8. VIII. Concurrency\n[---------------------------------] 9. IX. Disposability\n[---------------------------------] 10. X. Dev/prod parity\n[---------------------------------] 11. XI. Logs\n[---------------------------------] 12. XII. Admin processes\nMarkdown saved to \"The_Twelve-Factor_App.md\"\n```\n\n**Recursive mode options**\n\nIf a navigation menu is present on a website, you can scrape the content of each subpage.\n\nYou can activate this mode by using the `depth` or `selector` options.\n\n**`depth`**\n\nThis option defaults to 0, `papeer` will grab only the main page.\n\nThis option defaults to 1 if the `limit` option is specified.\n\nIf you specify a value greater than 0, `papeer` will grab pages as deep as the value you specify.\n\n**`selector`**\n\nIf this option is not specified, `papeer` will grab only the one page.\n\nIf this option is specified, `papeer` will select the links (a HTML tag) present on the main page, then grab each one of them.\n\nYou can chain this option to grab several level of pages with different selectors for each level.\n\n**`include`**\n\nUsing this option will include all intermediary levels into the book.\n\n**`delay` `threads`**\n\nBy default, it will grab all the pages asynchronously.\n\nUse those options to tweak the synchronicity of scrape requests.\n\n**Automatic table of contents extraction**\n\nIf you have a `depth` greater than 1 with no `selector`, it will be automatically determined based on the links present on the parent page.\n\n# Proxy\n\nYou can use the `proxy` command to act like proxy. It can serve HTML or Markdown content based on the `--output` option.\n\n```sh\npapeer proxy --output=md\n# Proxy listening on port 8080...\n```\n\nYou can call the endpoint with `curl` and the `--proxy` option.\n\n```sh\ncurl --insecure --location --proxy localhost:8080 http://www.brainjar.com/java/host/test.html\n# This is a very simple HTML file.\n```\n\n# Autocompletion\n\nExecute this command in your current shell, or add it to your `.bashrc`.\n\n```sh\n. \u003c(papeer completion bash)\n```\n\nYou can replace `bash` by your own shell (zsh, fish or powershell).\n\nType `papeer completion bash -h` for more information.\n\n# Dependencies\n\n- `cobra` command line interface\n- `go-readability` extract content from HTML\n- `html-to-markdown` convert HTML to Markdown\n- `go-epub` convert HTML to EPUB\n- `colly` query HTML trees\n- `uiprogress` display progress bars\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flapwat%2Fpapeer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flapwat%2Fpapeer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flapwat%2Fpapeer/lists"}