Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/viclafouch/pe-crawler
📌 An automated system that serves data extracted from the Google Help Center
https://github.com/viclafouch/pe-crawler
crawler javascript nodejs postgresql sequelize
Last synced: about 1 month ago
JSON representation
📌 An automated system that serves data extracted from the Google Help Center
- Host: GitHub
- URL: https://github.com/viclafouch/pe-crawler
- Owner: viclafouch
- Created: 2019-03-10T16:14:00.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2023-12-10T11:01:15.000Z (about 1 year ago)
- Last Synced: 2023-12-10T12:22:01.642Z (about 1 year ago)
- Topics: crawler, javascript, nodejs, postgresql, sequelize
- Language: JavaScript
- Homepage:
- Size: 9 MB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# PE Crawler
PE Crawler is an automated API system for retrieving threads from various Google products.
Moreover, it also allows to retrieve data from the help center of different products (e.g.: YouTube, Chrome,...).
The data are then served on an express server.
# Features
* No registration
* Zero-config
* Basic API
* "Has many" relationships
* Filters and nested resources
* Supports GET, POST
* HTTPS
* Compatible with React, Angular, Vue, Ember, ...
## Install
```bash
$ npm install
$ npm run dev
```
Might need to run
```
npm audit fix
```
### End-points
#### For getting all products:
```
GET /products
```
#### For getting all languages:
```
GET /languages
```
#### For getting number of answers in total:
```
GET /answers
```
#### For getting answers by product code (e.g: 'youtube') :
```
GET /answers/youtube
```
#### For getting answers by product code (e.g: 'youtube') and language locale (e.g: 'fr'):
```
GET /answers/youtube?hl=fr
```
#### For getting a pagination for answers by filtering by language, search value, and products id:
```
POST /answers/youtube
```
```js
body {
hl: 'fr',
page: 2,
search: 'foo',
products_id: [3, 9]
}
```
#### For getting number of threads in total:
```
GET /threads
```
#### For getting threads by product code (e.g: 'youtube') :
```
GET /threads/youtube
```
#### For getting threads by product code (e.g: 'youtube') and language locale (e.g: 'fr'):
```
GET /threads/youtube?hl=fr
```
## Contributing
Any contributions and/or pull requests would be welcome.
## License
MIT License.