https://github.com/yookoala/crawler

A library to fetch and catch HTTP results.
https://github.com/yookoala/crawler

Last synced: 5 days ago
JSON representation

A library to fetch and catch HTTP results.

• Host: GitHub
• URL: https://github.com/yookoala/crawler
• Owner: yookoala
• Created: 2015-01-18T08:35:20.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2015-02-12T11:44:43.000Z (almost 10 years ago)
• Last Synced: 2024-10-23T22:53:01.728Z (about 2 months ago)
• Language: Go
• Size: 3.17 MB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
crawler

=======

A library to fetch and cache HTTP results.

What it does?

-------------

This is a library to fetch and cache HTTP results. It allows easy comparison between different version of the same site. Suitable for crawler

You may extend the way to diff versions with the `Diff` interface. That means you decide what it means to be different and how you want to keep revisions of Raw HTTP results.

The default cache uses sql database. You may extend by implementing the `Cacher` interface.

Licence

-------

This file is part of crawler.

crawler is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

crawler is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public Licensefor more details.

You should have received a copy of the GNU Lesser General Public License along with crawler. If not, see http://www.gnu.org/licenses/lgpl.html.