https://github.com/flexoid/web-crawler-clojure

https://github.com/flexoid/web-crawler-clojure

Last synced: about 1 year ago
JSON representation

• Host: GitHub
• URL: https://github.com/flexoid/web-crawler-clojure
• Owner: flexoid
• License: epl-1.0
• Created: 2014-10-29T06:24:08.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2014-12-08T23:13:21.000Z (over 11 years ago)
• Last Synced: 2024-10-12T07:44:26.402Z (over 1 year ago)
• Language: Clojure
• Size: 376 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
## Simple web crawler

### Examples

    $ cat samples/sites.txt

    http://stackoverflow.com

    http://github.com

    http://redis.io

    $ lein run samples/sites.txt 2

      ......

### Run specs

    $ lein test

### TODO

  * Add support of relative URLs

  * Do not process repeating links