https://github.com/aaronpk/spiderpig

🕷🐷 reduces a dynamic website into a pile of smoldering html
https://github.com/aaronpk/spiderpig

Last synced: 11 days ago
JSON representation

🕷🐷 reduces a dynamic website into a pile of smoldering html

• Host: GitHub
• URL: https://github.com/aaronpk/spiderpig
• Owner: aaronpk
• Created: 2015-02-27T00:01:51.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2016-12-21T16:08:52.000Z (almost 8 years ago)
• Last Synced: 2024-04-15T09:13:21.227Z (7 months ago)
• Language: JavaScript
• Homepage:
• Size: 6.84 KB
• Stars: 20
• Watchers: 4
• Forks: 6
• Open Issues: 6
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
Spiderpig

=========

Spiderpig is a website crawler designed to archive your own sites as flat HTML files. It works great for Wordpress or other dynamic sites, collapsing the site into a folder of HTML files, images and JS and CSS assets.

## Usage

```

./spider.js example.com

```

This will crawl example.com, look for any links in <a> tags, and download all pages it finds to a folder called "example.com". It will also find images, Javascript and CSS files that are linked from any HTML pages or even other CSS files. However any content loaded via Javascript won't be discovered by the script.

## Why not just use `wget --mirror`?

Here is what Spiderpig does that Wget does not:

* Ensures all filenames on disk are called "index.html" so that the URLs to the content don't change (aside from sometimes adding a trailing slash).

* Ensures URLs such as `/example` and `/example/1` do not conflict with each other, which they would if you were to write a file to disk named `/example` because then `/example` would be a filename and it would try to write a file named `1` into the folder `/example` which is a problem.

* Spiderpig will parse CSS files looking for images referenced from the CSS file and download those images.

* Keeps track of HTTP redirects it encounters, saving them to a file, so you can generate .htaccess or nginx rewrite rules to ensure all your existing redirects stay intact.

* Query strings are ignored completely, since URLs with query strings can't be reliably served from disk. Also, cool URLs don't have query strings.

## License

Copyright 2014 Esri, Inc

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

> http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

A copy of the license is available in the repository's [LICENSE.txt] file.