{"id":31767902,"url":"https://github.com/webdevtodayjason/clone-it","last_synced_at":"2025-10-10T01:48:50.532Z","repository":{"id":309465880,"uuid":"1032668060","full_name":"webdevtodayjason/Clone-It","owner":"webdevtodayjason","description":"CLI tool to grab an entire website and clone it to html for Site backups.","archived":false,"fork":false,"pushed_at":"2025-08-05T16:47:27.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-06T01:30:30.666Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/webdevtodayjason.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-05T16:33:53.000Z","updated_at":"2025-08-05T16:47:30.000Z","dependencies_parsed_at":"2025-08-12T04:30:17.567Z","dependency_job_id":null,"html_url":"https://github.com/webdevtodayjason/Clone-It","commit_stats":null,"previous_names":["webdevtodayjason/clone-it"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/webdevtodayjason/Clone-It","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FClone-It","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FClone-It/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FClone-It/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FClone-It/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/webdevtodayjason","download_url":"https://codeload.github.com/webdevtodayjason/Clone-It/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webdevtodayjason%2FClone-It/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002400,"owners_count":26083375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-10T01:48:43.939Z","updated_at":"2025-10-10T01:48:50.527Z","avatar_url":"https://github.com/webdevtodayjason.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🌐 Clone-It - Website Cloning Tool\n\nA user-friendly command-line tool for cloning websites using wget with an interactive menu interface.\n\n## Features\n\n- **Interactive Menu**: Easy-to-use CLI interface with colored output\n- **Domain Validation**: Ensures valid domain format before proceeding\n- **Project Organization**: Creates organized folder structure in `projects/domain.com`\n- **Live Output**: Shows real-time wget progress and output\n- **Post-Clone Options**: View files, open folder, or clone another site\n- **Error Handling**: Graceful error handling with helpful messages\n\n## Prerequisites\n\n- **wget**: The tool requires wget to be installed\n  - macOS: `brew install wget`\n  - Ubuntu/Debian: `sudo apt-get install wget`\n  - CentOS/RHEL: `sudo yum install wget`\n\n## Installation\n\n1. Clone or download the script to your desired location\n2. Make it executable: `chmod +x clone-it.sh`\n3. (Optional) Run the installer for easier access: `./install.sh`\n\n## Usage\n\n### Direct execution:\n```bash\n./clone-it.sh\n```\n\n### After installation:\n```bash\nclone-it\n```\n\n## How It Works\n\n1. **Menu Interface**: The script presents a clean, colored menu asking \"What are we cloning today?\"\n2. **Domain Input**: Enter the domain you want to clone (e.g., `example.com`)\n3. **Validation**: The script validates the domain format\n4. **Link Conversion Option**: Choose whether to convert internal links to .html extension\n5. **Directory Creation**: Creates `projects/domain.com/` folder structure\n6. **Cloning Process**: Runs the wget command with optimal parameters:\n   ```bash\n   wget --mirror -w 2 -p --html-extension --convert-links https://domain.com/\n   ```\n7. **Link Processing**: If enabled, converts internal links to work with .html extensions\n8. **Live Output**: Shows the wget output in real-time within a bordered window\n9. **Completion Menu**: Offers options to:\n   - Clone another website\n   - Open the project folder\n   - View project contents\n   - Exit\n\n## wget Parameters Explained\n\n- `--mirror`: Creates a complete mirror of the site\n- `-w 2`: Waits 2 seconds between downloads (respectful crawling)\n- `-p`: Downloads all page prerequisites (images, CSS, etc.)\n- `--html-extension`: Adds .html extension to files\n- `--convert-links`: Converts links for offline browsing\n\n## Project Structure\n\n```\nCLONE-IT/\n├── clone-it.sh        # Main cloning script\n├── fix-links.sh       # Link conversion utility\n├── install.sh         # Installation helper\n├── README.md          # This file\n└── projects/          # Created when first used\n    └── domain.com/    # Individual site folders\n        └── domain.com/  # Actual site files\n```\n\n## Error Handling\n\n- Checks for wget installation before running\n- validates domain format\n- Handles existing directories with user confirmation\n- Reports wget exit codes and errors\n- Graceful handling of user cancellations\n\n## Colors and UI\n\nThe script uses ANSI color codes for better user experience:\n- 🔵 Blue: Process information\n- 🟢 Green: Success messages\n- 🟡 Yellow: Warnings and prompts\n- 🔴 Red: Error messages\n- 🔷 Cyan: Headers and menus\n\n## Link Conversion Feature\n\nThe tool includes a link conversion feature to fix a common issue with cloned websites:\n\n**The Problem**: When wget clones a site, it adds `.html` extensions to files that didn't originally have them. This breaks internal links like `/office-visits` which become `/office-visits.html` but the HTML still links to the original path without the extension.\n\n**The Solution**: When you choose \"Y\" for \"Convert all internal links to .html?\", the script will:\n- Scan all HTML files in the cloned site\n- Convert internal links from `/page-name` to `/page-name.html`\n- Handle both absolute and relative links\n- Preserve existing `.html` links unchanged\n- Create backups when using the standalone fix-links utility\n\n### Standalone Link Fixer\n\nFor sites already cloned without link conversion, use the separate utility:\n\n```bash\n./fix-links.sh              # Interactive mode\n./fix-links.sh example.com  # Direct mode\n```\n\n## Tips\n\n- The script is respectful to servers with a 2-second delay between requests\n- Large sites may take considerable time to clone completely\n- Check robots.txt and site terms of service before cloning\n- The cloned site will work offline with converted links\n- Use link conversion if the original site used clean URLs without extensions\n\n## Troubleshooting\n\n**\"wget not found\"**: Install wget using your package manager\n**\"Invalid domain\"**: Ensure domain format like `example.com` (no http://)\n**\"Permission denied\"**: Make sure the script is executable (`chmod +x`)\n**Slow cloning**: This is normal for large sites due to the respectful 2-second delay\n\n## License\n\nFree to use and modify as needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebdevtodayjason%2Fclone-it","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwebdevtodayjason%2Fclone-it","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebdevtodayjason%2Fclone-it/lists"}