{"id":16557657,"url":"https://github.com/patrickkfkan/xenforo-dl","last_synced_at":"2025-03-04T23:40:28.186Z","repository":{"id":227930859,"uuid":"772722972","full_name":"patrickkfkan/xenforo-dl","owner":"patrickkfkan","description":"XenForo forum downloader","archived":false,"fork":false,"pushed_at":"2024-03-15T19:03:07.000Z","size":70,"stargazers_count":2,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-15T10:02:42.748Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/patrickkfkan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"ko_fi":"patrickkfkan"}},"created_at":"2024-03-15T19:00:01.000Z","updated_at":"2025-01-24T13:33:55.000Z","dependencies_parsed_at":"2024-03-15T21:32:37.515Z","dependency_job_id":"77d18d49-6510-40e3-8e46-2d3d4b7f6395","html_url":"https://github.com/patrickkfkan/xenforo-dl","commit_stats":null,"previous_names":["patrickkfkan/xenforo-dl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickkfkan%2Fxenforo-dl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickkfkan%2Fxenforo-dl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickkfkan%2Fxenforo-dl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrickkfkan%2Fxenforo-dl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/patrickkfkan","download_url":"https://codeload.github.com/patrickkfkan/xenforo-dl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241940529,"owners_count":20045878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T20:08:10.078Z","updated_at":"2025-03-04T23:40:28.165Z","avatar_url":"https://github.com/patrickkfkan.png","language":"TypeScript","funding_links":["https://ko-fi.com/patrickkfkan","https://ko-fi.com/C0C5RGOOP'"],"categories":[],"sub_categories":[],"readme":"\u003ca href='https://ko-fi.com/C0C5RGOOP' target='_blank'\u003e\u003cimg height='36' style='border:0px;height:36px;' src='https://storage.ko-fi.com/cdn/kofi2.png?v=3' border='0' alt='Buy Me a Coffee at ko-fi.com' /\u003e\u003c/a\u003e\n\n# xenforo-dl\n\nA [XenForo](https://xenforo.com/) forum downloader written in [Node.js](https://nodejs.org):\n\n- Scrapes content from forum pages\n- For each thread, downloads attachments and saves messages in text files\n- Supports downloading a single thread or all threads in a forum\n- Supports continuing from previous download\n\nSince the downloader works through scraping, it is not guaranteed to work with all XenForo forums. I created the downloader for my data-hoarding needs targeting a handful of sites, so it might be limited in what it can scrape. But feel free to raise issues.\n\n\n## Installation\n\nFirst, install [Node.js](https://nodejs.org/).\n\nThen, in a terminal, run the following command:\n\n```\nnpm i -g xenforo-dl\n```\n\n## Usage\n\n```\n$ xenforo-dl [OPTION]... URL\n```\n\n### URL\n\n#### Thread URLs\n\nPattern: `\u003cforum_site_url\u003e/threads/\u003ctitle_slug\u003e.\u003cthread_id\u003e[/page-\u003cnum\u003e]`\n\nDownload all messages and attachments shown on page. If content spans multiple pages, download from subsequent pages as well.\n\nIf `page-\u003cnum\u003e` is present in URL, then download will begin with the specified page.\n\n#### Forum URLs\n\nPattern: `\u003cforum_site_url\u003e/forums/\u003ctitle_slug\u003e.\u003cforum_id\u003e[/page-\u003cnum\u003e]`\n\nDownload all threads listed on page. If the forum has threads spanning multiple pages, download from subsequent pages as well.\n\nIf `page-\u003cnum\u003e` is present in URL, then download will begin with the specified page.\n\n#### Other URLs\n\nFor URLs not matching the above patterns, `xenforo-dl` will scrape for forum links and download from them. It is your responsibility to ensure the given URL is a valid XenForo link.\n\n### Options\n\n| Option    | Description |\n|-----------|-------------|\n| `-h`, `--help` | Display usage guide |\n| `-k`, `--cookie` | (string) Cookie to set in requests. See [Cookies](#cookies). |\n| `-o`, `--out-dir` | (string) Path of save directory. Default: current working directory. |\n| `-d`, `--dir-structure` | Combination of flags controlling the output directory structure of downloaded threads: \u003cul\u003e\u003cli\u003e`s`: Include directory for the forum site.\u003c/li\u003e\u003cli\u003e`pl`: Include directory for each category or forum leading up to the target thread.\u003c/li\u003e\u003cli\u003e`pi`: Include directory for the immediate section or forum containing the target thread.\u003c/li\u003e\u003cli\u003e`t`: Include directory for the target thread itself.\u003c/li\u003e\u003cli\u003e`a`: Include directory for attachments.\u003c/li\u003e\u003cli\u003e`-`: No directory structure. Everything will be saved directly to --out-dir.\u003c/li\u003e\u003c/ul\u003e\u003cp\u003eDefault: `splta`\u003c/p\u003e|\n| `-w`, `--overwrite` | Overwrite existing attachment files |\n| `-l`, `--log-level` | Log level: `info`, `debug`, `warn` or `error`; set to `none`` to disable logging. Default: `info` |\n| `-s`, `--log-file` | (string) Save logs to specified path |\n| `-r`, `--max-retries` | (number) Maximum retry attempts when a download fails. Default: 3 |\n| `-c`, `--max-concurrent`| (number) Maximum number of concurrent downloads for attachments. Default: 10 |\n| `-p`, `--min-time-page` | (number) Minimum time, in milliseconds, to wait between page fetch requests. Default: 500 |\n| `-i`, `--min-time-image` | (number) Minimum time, in milliseconds, to wait between download requests for attachments. Default: 200 |\n| `--continue` | Continue from previous download |\n| `-y`, `--no-prompt` | Do not prompt for confirmation to proceed |\n\n### Cookies\n\nCookies allow you to download content that would otherwise be inaccessible due to lack of user credentials. To obtain a cookie for passing to `xenforo-dl` through the `--cookie` option, do the following:\n\n1. In a browser, sign in to the target forum site.\n2. Press `F12` to bring up Developer Tools.\n3. Select `Network` tab, followed by `HTML` filter.\n4. Press `F5` to refresh the page. Select one of the entries that appear under the `Network` tab.\n5. Under `Headers` -\u003e `Request Headers`, you should see the `Cookie` entry. Copy the value of that entry and pass it to `xenforo-dl`.\n\nCookies should remain valid until they expire or you sign out of the forum site.\n\n## Changelog\n\nv1.0.0\n- Initial release\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatrickkfkan%2Fxenforo-dl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatrickkfkan%2Fxenforo-dl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatrickkfkan%2Fxenforo-dl/lists"}