Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zaycev/cbg-scrapy
Simple HTTP server for asynchronous scrapping data from Twitter API using Twisted library
https://github.com/zaycev/cbg-scrapy
Last synced: 21 days ago
JSON representation
Simple HTTP server for asynchronous scrapping data from Twitter API using Twisted library
- Host: GitHub
- URL: https://github.com/zaycev/cbg-scrapy
- Owner: zaycev
- Created: 2013-01-23T03:38:19.000Z (almost 12 years ago)
- Default Branch: master
- Last Pushed: 2013-05-13T19:58:33.000Z (over 11 years ago)
- Last Synced: 2024-08-05T17:42:26.989Z (4 months ago)
- Language: Python
- Size: 133 KB
- Stars: 7
- Watchers: 3
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-hacking-lists - zaycev/cbg-scrapy - Simple HTTP server for asynchronous scrapping data from Twitter API using Twisted library (Python)
README
# CBG Scrapy
CBG Scrapy – is a simple HTTP server for asynchronous scrapping data from Twitter API using Twisted library.
## Installation and running
```bash
$ python scrapy.py [-p ] [-l ]
```
## HTTP API
* ###Adding scrapers
Adds (activates) new scrapers.
URI: `/add/`
GET parameters:
```json
data:
[
{
"name": "LA Scraper",
"oauth": {
"token": "",
"secret": ""
},
"filter": {
"id": "Some integer, unique for each scraper",
"location": [-122.75, 36.8, -121.75, 37.8],
}
}
]
```
Response:
```js
{
"error": true | false,
"message": "Error message"
}
```
* ### Listing scrapers
Returnes state of active scrapers.
URI: `/list/`
GET parameters:
```
none
```
Response:
```js
[
{
"name": "LA scraper",
"token": "",
"status": "connecting" | "connected" | "failed",
"ts_start": "2012.12.12T12:12:00",
"received": 10000,
"total_received": 100000,
"limits": 5000,
"total_limits": 60000,
"rate": 10.4,
"last_received": "2012.12.12T12:12:00",
"filter": {
"track": ["#Python", "#Haskell"],
"follow": [1, 2, 4],
"locations" [0, 0, 0, 0]
},
"errors": [
{
"message": "error message",
"ts": "2012.12.12T12:12:00"
}
]
}
]
```
* ### Removing scrapers
Stops and removes active scrapers.
URI: `/remove/`
GET parameters:
```js
data:
[
""
]
```
Response:
```js
{
"error": true | false,
"message": "Error message"
}
```
* ### Ping
Returns string `pong`.
URI: `/ping/`
GET parameters:
```
none
```
Response:
```
pong
```
* ### Log
Returns log string.
URI: `/log/`
GET parameters:
```
none
```