{"id":13587106,"url":"https://github.com/hipstermojo/paperoni","last_synced_at":"2025-04-07T19:30:39.651Z","repository":{"id":47559047,"uuid":"260118457","full_name":"hipstermojo/paperoni","owner":"hipstermojo","description":"An article extractor in Rust","archived":false,"fork":false,"pushed_at":"2022-02-01T17:17:13.000Z","size":844,"stargazers_count":128,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-04-14T11:57:28.936Z","etag":null,"topics":["article-extractor","readability","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hipstermojo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-30T05:06:05.000Z","updated_at":"2024-04-14T11:57:28.937Z","dependencies_parsed_at":"2022-08-25T20:20:50.928Z","dependency_job_id":null,"html_url":"https://github.com/hipstermojo/paperoni","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hipstermojo%2Fpaperoni","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hipstermojo%2Fpaperoni/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hipstermojo%2Fpaperoni/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hipstermojo%2Fpaperoni/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hipstermojo","download_url":"https://codeload.github.com/hipstermojo/paperoni/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247716068,"owners_count":20984167,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["article-extractor","readability","rust"],"created_at":"2024-08-01T15:06:01.510Z","updated_at":"2025-04-07T19:30:39.330Z","avatar_url":"https://github.com/hipstermojo.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"./paperoni-dark.png\"\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003ci\u003eSalami not included\u003c/i\u003e\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://crates.io/crates/paperoni\"\u003e\n        \u003cimg alt=\"crates.io version\" src=\"https://img.shields.io/crates/v/paperoni.svg\"\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\nPaperoni is a CLI tool made in Rust for downloading web articles as EPUB or HTML files. There is provisional\u003csup\u003e\u003ca href=\"#pdf-exports\"\u003e\\*\u003c/a\u003e\u003c/sup\u003e support for exporting to PDF as well.\n\n\u003e This project is in an alpha release so it might crash when you use it. Please open an [issue on Github](https://github.com/hipstermojo/paperoni/issues/new) if it does crash.\n\n## Installation\n\n### Precompiled binaries\n\nCheck the [releases](https://github.com/hipstermojo/paperoni/releases) page for precompiled binaries. Currently there are only builds for Debian and Arch.\n\n### Installing from crates.io\n\nPaperoni is published on [crates.io](https://crates.io). If you have [cargo](https://github.com/rust-lang/cargo) installed, then run:\n\n```sh\ncargo install paperoni --version 0.6.1-alpha1\n```\n\n_Paperoni is still in alpha so the `version` flag has to be passed._\n\n### Building from source\n\nThis project uses `async/.await` so it should be compiled using a minimum Rust version of 1.33. Preferrably use the latest version of Rust.\n\n```sh\ngit clone https://github.com/hipstermojo/paperoni.git\ncd paperoni\n## You can build and install paperoni locally\ncargo install --path .\n## or use it from within the project\ncargo run -- # pass your url here\n```\n\n## Usage\n\n```\nUSAGE:\n    paperoni [OPTIONS] [urls]...\n\nOPTIONS:\n        --export \u003ctype\u003e\n            Specify the file type of the export. The type must be in lower case. [default: epub]  [possible values:\n            html, epub]\n    -f, --file \u003cfile\u003e\n            Input file containing links\n\n    -h, --help\n            Prints help information\n\n        --inline-images\n            Inlines the article images when exporting to HTML using base64.\n            This is used when you do not want a separate folder created for images during HTML export.\n            NOTE: It uses base64 encoding on the images which results in larger HTML export sizes as each image\n            increases in size by about 25%-33%.\n        --inline-toc\n            Add an inlined Table of Contents page at the start of the merged article. This does not affect the Table of Contents navigation\n        --log-to-file\n            Enables logging of events to a file located in .paperoni/logs with a default log level of debug. Use -v to\n            specify the logging level\n        --max-conn \u003cmax-conn\u003e\n            The maximum number of concurrent HTTP connections when downloading articles. Default is 8.\n            NOTE: It is advised to use as few connections as needed i.e between 1 and 50. Using more connections can end\n            up overloading your network card with too many concurrent requests.\n        --no-css\n            Removes the stylesheets used in the EPUB generation.\n            The EPUB file will then be laid out based on your e-reader's default stylesheets.\n            Images and code blocks may overflow when this flag is set and layout of generated\n            PDFs will be affected. Use --no-header-css if you want to only disable the styling on headers.\n        --no-header-css\n            Removes the header CSS styling but preserves styling of images and codeblocks. To remove all the default\n            CSS, use --no-css instead.\n        --merge \u003coutput-name\u003e\n            Merge multiple articles into a single epub that will be given the name provided\n\n    -o, --output-dir \u003coutput_directory\u003e\n            Directory to store output epub documents\n\n    -V, --version\n            Prints version information\n\n    -v\n            This takes upto 4 levels of verbosity in the following order.\n            - Error (-v)\n            - Warn (-vv)\n            - Info (-vvv)\n            - Debug (-vvvv)\n            When this flag is passed, it disables the progress bars and logs to stderr.\n            If you would like to send the logs to a file (and enable progress bars), pass the log-to-file flag.\n\nARGS:\n    \u003curls\u003e...\n            Urls of web articles\n\n```\n\nTo download a single article pass in its URL\n\n```sh\npaperoni https://en.wikipedia.org/wiki/Pepperoni\n```\n\nPaperoni also supports passing multiple links as arguments.\n\n```sh\npaperoni https://en.wikipedia.org/wiki/Pepperoni https://en.wikipedia.org/wiki/Salami\n```\n\nAlternatively, if you are on a Unix-like OS, you can simply do something like this:\n\n```sh\ncat links.txt | xargs paperoni\n```\n\nThese can also be read from a file using the `-f/--file` flag.\n\n```sh\npaperoni -f links.txt\n```\n\n### Exporting articles\n\nBy default, Paperoni exports to EPUB files but you can change to HTML by passing the `--export html` flag.\n\n```sh\npaperoni https://en.wikipedia.org/wiki/Pepperoni --export html\n```\n\nHTML exports allow you to read the articles as plain HTML documents on your browser but can also be used to convert to PDF as explained [here](#).\n\nWhen exporting to HTML, Paperoni will download the article's images to a folder named similar to the article. Therefore the folder structure would look like this for the command ran above:\n\n```\n.\n├── Pepperoni - Wikipedia\n│   ├── 1a9f886e9b58db72e0003a2cd52681d8.png\n│   ├── 216f8a4265a1ceb3f8cfba4c2f9057b1.jpeg\n│   ...\n└── Pepperoni - Wikipedia.html\n```\n\nIf you would instead prefer to have the images inlined directly to the HTML export, pass the `inline-images` flag, i.e.:\n\n```sh\npaperoni https://en.wikipedia.org/wiki/Pepperoni --export html --inline-images\n```\n\nThis is especially useful when exporting multiple links.\n\n**NOTE**: The inlining of images for HTML exports uses base64 encoding which is known to increase the overall size of images by about 25% to 33%.\n\n### Disabling CSS\n\nThe `no-css` and `no-header-css` flags can be used to remove the default styling added by Paperoni. Refer to `--help` to see the usage of the flags.\n\n### Merging articles\n\nBy default, Paperoni generates an epub file for each link. You can also merge multiple links\ninto a single epub using the `merge` flag and specifying the output file.\n\n```sh\npaperoni -f links.txt --merge out.epub\n```\n\n### Logging events\n\nLogging is disabled by default. This can be activated by either using the `-v` flag or `--log-to-file` flag. If the `--log-to-file` flag is passed the logs are sent to a file in the default Paperoni directory `.paperoni/logs` which is on your home directory. The `-v` flag configures the verbosity levels such that:\n\n```\n-v Logs only the error level\n-vv Logs only the warn level\n-vvv Logs only the info level\n-vvvv Logs only the debug level\n```\n\nIf only the `-v` flag is passed, the progress bars are disabled. If both `-v` and `--log-to-file` are passed then the progress bars will still be shown.\n\n## How it works\n\nThe URL passed to Paperoni is fetched and the returned HTML response is passed to the extractor.\nThis extractor retrieves a possible article using a [custom port](https://github.com/hipstermojo/paperoni/blob/master/src/moz_readability/mod.rs) of the [Mozilla Readability algorithm](https://github.com/mozilla/readability). This article is then saved in an EPUB.\n\n\u003e The port of the algorithm is still unstable as well so it is not fully compatible with all the websites that can be extracted using Readability.\n\n## How it (currently) doesn't work\n\nThis program is still in alpha so a number of things won't work:\n\n- Websites that only run with JavaScript cannot be extracted.\n- Website articles that cannot be extracted by Readability cannot be extracted by Paperoni either.\n- Code snippets on Medium articles that are lazy loaded will not appear in the EPUB.\n\nThere are also web pages it won't work on in general such as Twitter and Reddit threads.\n\n## PDF exports\n\nPDF conversion can be done using a third party tool. There are 2 options to do so:\n\n### EPUB to PDF\n\nThis requires that you install [Calibre](https://calibre-ebook.com/) which comes with a ebook conversion. You can convert the epub to a pdf through the terminal with `ebook-convert`:\n\n```sh\n# Assuming the downloaded epub was called foo.epub\nebook-convert foo.epub foo.pdf\n```\n\nAlternatively, you can use the Calibre GUI to do the file conversion.\n\n### HTML to PDF\n\nThe recommended approach is to use [Weasyprint](https://weasyprint.org/start/), a free and open-source tool that converts HTML documents to PDF. It is available on Linux, MacOS and Windows. Using the CLI, it can be done as follows:\n\n```sh\npaperoni https://en.wikipedia.org/wiki/Pepperoni --export html\nweasyprint \"Pepperoni - Wikipedia.html\" Pepperoni.pdf\n```\n\nInlining images is not mandatory as Weasyprint will be able to find the files on its own.\n\n### Comparison of PDF conversion methods\n\nEither of the conversion methods is sufficient for most use cases. The main differences are listed below:\n| | EPUB to PDF | HTML to PDF |\n|----------------------|----------------------------|------------------|\n| Wrapping code blocks | Yes | No |\n| CSS customization | No | Yes |\n| Generated file size | Slightly larger | Slightly smaller |\n\nThe difference in file size is due to the additional fonts added to the PDF file by `ebook-convert`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhipstermojo%2Fpaperoni","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhipstermojo%2Fpaperoni","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhipstermojo%2Fpaperoni/lists"}