Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/steder/spinneret
Spinneret is a web spider and dynamic load generator for testing, benchmarking, and warming caches.
https://github.com/steder/spinneret
gevent python traffic-generation web-crawler
Last synced: about 2 months ago
JSON representation
Spinneret is a web spider and dynamic load generator for testing, benchmarking, and warming caches.
- Host: GitHub
- URL: https://github.com/steder/spinneret
- Owner: steder
- Created: 2012-07-26T02:06:11.000Z (over 12 years ago)
- Default Branch: master
- Last Pushed: 2023-03-22T00:40:35.000Z (almost 2 years ago)
- Last Synced: 2024-10-15T21:38:25.510Z (2 months ago)
- Topics: gevent, python, traffic-generation, web-crawler
- Language: Python
- Homepage:
- Size: 110 KB
- Stars: 4
- Watchers: 4
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.rst
Awesome Lists containing this project
README
Spinneret
-----------------------------
Spinneret is a web spider and dynamic load generator for testing,
benchmarking, and warming caches.
Setup
-----------------------------
::
git clone [email protected]:steder/spinneret.git
cd spinneret
mkdir -p ~/.virtualenvs
virtualenv ~/.virtualenvs/spinneret
python setup.py develop
Generating a sitemap
-----------------------------
::
spinneret spider --base_url http://localhost:8000
Because this can take a while I've included a sample `sitemap.yaml`
that can be used to immediately generate traffic.
Using that sitemap to generate requests against a site
------------------------------------------------------------
::
spinneret traffic --base_url http://localhost:8000 --inflight 100