Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/shurco/goClone
🌱 goClone - clone websites in seconds
https://github.com/shurco/goClone
cloner cloning crawler go goclone golang hacktoberfest scraping scrapper website-cloner website-scraper wp2static
Last synced: 3 months ago
JSON representation
🌱 goClone - clone websites in seconds
- Host: GitHub
- URL: https://github.com/shurco/goClone
- Owner: shurco
- License: mit
- Created: 2023-07-08T12:36:12.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-17T11:43:21.000Z (5 months ago)
- Last Synced: 2024-06-22T14:12:07.283Z (5 months ago)
- Topics: cloner, cloning, crawler, go, goclone, golang, hacktoberfest, scraping, scrapper, website-cloner, website-scraper, wp2static
- Language: Go
- Homepage:
- Size: 7.09 MB
- Stars: 54
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
- jimsghstars - shurco/goClone - 🌱 goClone - clone websites in seconds (Go)
README
# 🌱 goClone
goClone is a powerful utility that enables you to effortlessly download entire websites from the Internet and save them to your local directory. With goClone, you can easily obtain HTML, CSS, JavaScript, images, and other associated files directly from the server and store them on your computer.
One of the standout features of goClone is its ability to accurately preserve the original website's relative link structure. This means that when you open any page of the "mirrored" website in your browser, you can seamlessly navigate through the site by following links just as if you were browsing it online.
goClone empowers you to have offline access to websites, making it convenient for various purposes such as research, archiving, or simply enjoying a website without an internet connection.
So go ahead, give goClone a try and experience the freedom of having your favorite websites at your fingertips, even when you're offline!
```shell
$ brew install shurco/tap/goclone
```
Alternately, you can configure the tap and install the package separately:
``` shell
$ brew tap shurco/tap
$ brew install goclone
```
## Manual
```bash
# go get :)
go get github.com/shurco/goClone
# change to project directory using your GOPATH
cd $GOPATH/src/github.com/shurco/goClone/cmd
# build and install application
go install
```
## Examples
```bash
# goclone
goclone https://domain.com
```
## Usage
```
Usage:
goclone [flags]
Flags:
-b, --browser_endpoint string chrome headless browser WS endpoint
-c, --cookie if set true, cookies won't send
-h, --help help for goclone
-o, --open automatically open project in default browser
-p, --proxy_string string proxy connection string
-r, --robots disable robots.txt checks
-s, --serve serve the generated files using gofiber
-P, --servePort int serve port number (default 8088)
-u, --user_agent string custom User-Agent (default "goclone")
-v, --version version for goclone
```
## Making JS Rendered Requests
JS Rendered requests can be made using ```-b``` flag. For example start image :
``` bash
docker run -d -p 9222:9222 --rm --name headless-shell chromedp/headless-shell
```
then run goclone:
```bash
goclone -b "ws://localhost:9222" https://domain.com
```