https://github.com/emcecs/jobot
Useless internet crawler
https://github.com/emcecs/jobot
Last synced: over 1 year ago
JSON representation
Useless internet crawler
- Host: GitHub
- URL: https://github.com/emcecs/jobot
- Owner: EMCECS
- Created: 2019-02-07T14:07:54.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-03-07T11:30:34.000Z (over 7 years ago)
- Last Synced: 2025-02-08T18:45:41.887Z (over 1 year ago)
- Language: Java
- Size: 62.5 KB
- Stars: 0
- Watchers: 7
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Jobot
A useless, quick and dirty internet crawler which extracts the hyperlinks from the web pages. Jobot also saves the
metadata for each processed URL into the file `/.jobot//links.txt`. The saved metadata is a list of the
URLs extracted from the corresponding HTML page.
# Build
```bash
./gradlew clean jar
```
# Run
```bash
java -jar build/libs/jobot.jar https://en.wikipedia.org/wiki/\(486958\)_2014_MU69
```
# Implementation Notes
Jobot modifies and filters the URLs upon processing:
* URL should not be equal to any of the last 1,000,000 extracted URLs
* Anchor and query parts of the URL are being dropped
* URL should begin with `http`, otherwise it's dropped entirely
* The corresponding HTTP response content type should begin with `text`, otherwise the content is dropped
The URL processing queue size is 1,000,000.