{"id":28260876,"url":"https://github.com/sctg-development/wp-go-downloader","last_synced_at":"2025-06-26T09:33:40.206Z","repository":{"id":292167345,"uuid":"980019165","full_name":"sctg-development/wp-go-downloader","owner":"sctg-development","description":"A Go-based tool for creating offline copies of WordPress websites. Downloads complete sites with assets, handles link conversion, and preserves structure. Requires customization for specific WordPress implementations.","archived":false,"fork":false,"pushed_at":"2025-05-10T14:03:44.000Z","size":43,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-20T05:12:10.392Z","etag":null,"topics":["golang","pishing","wordpress"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sctg-development.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["sctg-development"]}},"created_at":"2025-05-08T12:40:07.000Z","updated_at":"2025-05-13T09:32:36.000Z","dependencies_parsed_at":"2025-05-08T13:45:41.176Z","dependency_job_id":"fc0751c0-e366-4b9d-b9e2-dafcf8ade8f1","html_url":"https://github.com/sctg-development/wp-go-downloader","commit_stats":null,"previous_names":["sctg-development/wp-go-downloader"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/sctg-development/wp-go-downloader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sctg-development%2Fwp-go-downloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sctg-development%2Fwp-go-downloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sctg-development%2Fwp-go-downloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sctg-development%2Fwp-go-downloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sctg-development","download_url":"https://codeload.github.com/sctg-development/wp-go-downloader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sctg-development%2Fwp-go-downloader/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260028221,"owners_count":22947900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","pishing","wordpress"],"created_at":"2025-05-20T05:11:11.003Z","updated_at":"2025-06-15T18:30:50.869Z","avatar_url":"https://github.com/sctg-development.png","language":"Go","funding_links":["https://github.com/sponsors/sctg-development"],"categories":[],"sub_categories":[],"readme":"# WordPress Website Downloader\n\nA robust Go-based tool for downloading WordPress websites for offline viewing. This project creates a complete offline copy of WordPress websites while maintaining structure, assets, and navigation.\n\n## Disclaimer\n\nThis tool was originally designed for downloading a specific WordPress website but has been generalized to work with many WordPress sites.  \n**Important:** This downloader will almost certainly need to be forked and customized for each website you plan to download. WordPress sites vary significantly in their structure and implementation.  \n**Important:** Only use this tool on websites you own or have explicit permission to download. Always respect terms of service and copyright.\n\n## Star the project\n\n**If you find this tool useful, please consider giving it a star! 🤩**\n\n## When to Fork This Project\n\nYou should consider forking this project when:\n\n- The target website has custom WordPress plugins that alter HTML structure\n- You need to handle specific URL patterns unique to your website\n- You encounter parsing errors with the default implementation\n- You need to add custom post-processing for particular content types\n- The website uses non-standard JavaScript implementations\n\n## Features\n\n- Complete website downloads with all assets (images, CSS, JavaScript, fonts)\n- CSS processing to extract and download referenced resources\n- WordPress-specific URL pattern handling and cache-busting parameter removal\n- Multilingual site support with proper language directory handling\n- Tracking script removal (Google Analytics, Google Tag Manager)\n- Absolute to relative URL conversion for offline navigation\n- Original directory structure preservation\n- Support for responsive images (`srcset` attributes)\n- Recursive HTML page processing\n- **Configurable concurrent downloads**\n- **URL filtering with include/exclude patterns**\n- **Automatic retry on download failures**\n- **Custom output directory support**\n- **Adjustable HTTP timeout settings**\n- **Detailed logging options**\n\n## Installation\n\n### Prerequisites\n\n- Go 1.20 or higher\n- Git\n\n### Building from Source\n\n```bash\n# Clone the repository\ngit clone https://github.com/sctg-development/wp-go-downloader.git\ncd wp-go-downloader\n\n# Build the project\ngo build ./wp-go-downloader.go\n```\n\n## Usage\n\n### Basic Usage\n\n```bash\n# With default options (downloads example.com)\n./wp-go-downloader\n\n# Specify a custom WordPress site URL\n./wp-go-downloader -url https://mywordpresssite.com\n```\n\n### Advanced Options\n\n```bash\n# Set concurrency, output directory, and timeout\n./wp-go-downloader -url https://mywordpresssite.com -concurrency 15 -outdir my-site-backup -timeout 60\n\n# Filter URLs to download only specific files\n./wp-go-downloader -url https://mywordpresssite.com -include \"\\.(jpg|png|gif)$\" -exclude \"/wp-admin/\"\n```\n\n### Command Line Reference\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `-url` | Website URL to download | https://www.example.com/ |\n| `-concurrency` | Maximum parallel downloads | 10 |\n| `-outdir` | Custom output directory | Domain name of the website |\n| `-include` | Only process URLs matching this regex | None (process all URLs) |\n| `-exclude` | Skip URLs matching this regex | None (process all URLs) |\n| `-timeout` | HTTP request timeout in seconds | 30 |\n| `-verbose` | Enable detailed logging | false |\n\n## Process Overview\n\nThe tool performs the following steps:\n\n1. Downloads the initial HTML page\n2. Parses HTML to extract all URLs (scripts, images, CSS, links)\n3. Downloads all assets concurrently within the concurrency limit\n4. Recursively processes any HTML pages found during download\n5. Post-processes HTML and CSS files for offline functionality\n6. Removes tracking scripts and optimizes files\n7. Saves the complete website to the specified output directory\n\n## Limitations\n\n- No support for content loaded dynamically via AJAX\n- Cannot execute JavaScript to render dynamic content\n- Not compatible with single-page applications (SPAs)\n- Limited functionality with complex frontend frameworks\n- May require adjustments for heavily customized WordPress sites\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\nThis tool was designed for specific use cases and may need adjustments for different WordPress installations. You're welcome to open Pull Requests to improve the tool or add new features to make it more versatile.\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE.md file for details.\n\n## Credits\n\nCreated by Ronan Le Meillat.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsctg-development%2Fwp-go-downloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsctg-development%2Fwp-go-downloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsctg-development%2Fwp-go-downloader/lists"}