Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/skorfmann/spidergraph
A Scraper leveraging GraphQL
https://github.com/skorfmann/spidergraph
graphql scraper
Last synced: 23 days ago
JSON representation
A Scraper leveraging GraphQL
- Host: GitHub
- URL: https://github.com/skorfmann/spidergraph
- Owner: skorfmann
- License: mit
- Created: 2018-02-19T21:09:45.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-01-03T16:01:09.000Z (almost 6 years ago)
- Last Synced: 2024-04-25T13:01:35.024Z (7 months ago)
- Topics: graphql, scraper
- Language: JavaScript
- Homepage:
- Size: 2.44 MB
- Stars: 7
- Watchers: 3
- Forks: 0
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# SpiderGraph
GraphQL based web scraper / crawler / spider
You can also find me on twitter at [@skorfmann](https://twitter.com/skorfmann).
[](https://circleci.com/gh/skorfmann/spidergraph)
## Example
### CLI
#### Scrape
```
spidergraph scrape https://www.immobilienscout24.de/expose/54174187,https://www.immonet.de/angebot/33189259 | jq
[
{
"info": {
"query": "immobilienscout24",
"url": "https://www.immobilienscout24.de/expose/54174187"
},
"data": {
"realEstate": [
{
"__typename": "House",
"title": "Einfamilienhaus, \"Exclusive Stadtvilla\" HH Poppenbüttel",
"price": "2.700 €"
}
]
}
},
{
"info": {
"query": "immonet",
"url": "https://www.immonet.de/angebot/33189259"
},
"data": {
"realEstate": [
{
"__typename": "House",
"title": "2 Zimmer Maisonettewohnung - Neubau im Universitätsviertel",
"price": "1390.0 €"
}
]
}
}
]
```
#### Crawl
```
spidergraph crawl https://www.immobilienscout24.de/Suche/S-T/Wohnung-Miete/Hamburg/Hamburg
[{"info":{"query":"immobilienscout24","url":"https://www.immobilienscout24.de/expose/89156209"},"data":{"realEstate":[{"__typename":"House","title":"Wellingsbüttel - 2 Zimmer","price":"1.093,63 €"}]}}]
[{"info":{"query":"immobilienscout24","url":"https://www.immobilienscout24.de/expose/102498094"},"data":{"realEstate":[{"__typename":"House","title":"Ruhiges Wohnen mit Kamin in zentraler Lage","price":"1.863 €"}]}}]
[{"info":{"query":"immobilienscout24","url":"https://www.immobilienscout24.de/expose/99130149"},"data":{"realEstate":[{"__typename":"House","title":"Hochwertig sanierte Wohnung im Zweifamilienhaus mit Garten","price":"3.800 €"}]}}]
...
```
## License
MIT License. See the [LICENSE](LICENSE) file.