https://github.com/ghostwords/surveyor

Run a regex against the HTML of Alexa Global Top 1,000,000 Sites
https://github.com/ghostwords/surveyor

cli multiprocessing requests

Last synced: 3 months ago
JSON representation

Run a regex against the HTML of Alexa Global Top 1,000,000 Sites

• Host: GitHub
• URL: https://github.com/ghostwords/surveyor
• Owner: ghostwords
• License: mpl-2.0
• Created: 2016-09-20T20:06:58.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2016-11-23T01:53:46.000Z (over 9 years ago)
• Last Synced: 2025-03-06T12:27:16.291Z (over 1 year ago)
• Topics: cli, multiprocessing, requests
• Language: Python
• Homepage:
• Size: 24.4 KB
• Stars: 2
• Watchers: 5
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Surveyor

Like `grep` for website sources.

Requires `top-1m.csv`. Get it from http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

See http://stackoverflow.com/questions/4265748/search-in-html-source-with-google for alternatives.

## Usage

```

usage: survey.py [-h] [-Q] [-s SKIP] [-l LIMIT] [-n NUM_PROCESSES]

                 [-t SECONDS] [-d]

                 PATTERN

positional arguments:

  PATTERN               the regex pattern to search for

optional arguments:

  -h, --help            show this help message and exit

  -Q, --literal         treat PATTERN as literal string, not a regex

  -s SKIP, --skip SKIP  skip this many hostnames from the start

  -l LIMIT, --limit LIMIT

                        stop after this many hostnames

  -n NUM_PROCESSES      use this many processes in parallel (default: 20)

  -t SECONDS, --timeout SECONDS

                        wait this many seconds to connect and again to read

                        before timing out (default: 3.4)

  -d, --debug           enable debugging output

```

## Code license

Mozilla Public License Version 2.0