Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/evenchange4/micro-website-api

An API microservice that crawls dynamic website powered by puppeteer.
https://github.com/evenchange4/micro-website-api

puppeteer

Last synced: 13 days ago
JSON representation

An API microservice that crawls dynamic website powered by puppeteer.

Host: GitHub
URL: https://github.com/evenchange4/micro-website-api
Owner: evenchange4
License: mit
Created: 2017-11-07T13:22:20.000Z (about 7 years ago)
Default Branch: master
Last Pushed: 2018-01-15T01:35:15.000Z (almost 7 years ago)
Last Synced: 2024-10-10T21:19:06.584Z (about 1 month ago)
Topics: puppeteer
Language: JavaScript
Homepage: https://micro-website-api.now.sh/
Size: 323 KB
Stars: 29
Watchers: 3
Forks: 9
Open Issues: 13
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE

Awesome Lists containing this project

awesome-puppeteer - micro-website-api

README

        # Micro-website-api

> An API microservice that crawls dynamic website powered by

> [puppeteer](https://github.com/GoogleChrome/puppeteer).

[![Travis][travis-badge]][travis] [![Codecov Status][codecov-badge]][codecov]

[![Dependency Status][dependency-badge]][dependency]

[![devDependency Status][devdependency-badge]][devdependency]

[![peerDependency Status][peerdependency-badge]][peerdependency]

[![Greenkeeper badge][greenkeeper-badge]][greenkeeper]

[![prettier][prettier-badge]][prettier] [![license][license-badge]][license]

[![](./docs/screenshot.png)](https://micro-website-api.now.sh)

## How To Use

### a. Deploy to Now.sh

> One-click deploy to △ now

[![Deploy to now](https://deploy.now.sh/static/button.svg)](https://deploy.now.sh/?repo=https://github.com/evenchange4/micro-website-api&env=TIMEOUT&docker=true)

### b. Docker image

```

$ docker build -t micro-website-api .

$ docker run --rm -it \

  -p 3000:3000 \

  -e "TIMEOUT=300000" \

  micro-website-api

```

## API

### Environment variables

| **ENV**           | **Required** | **Default** | **Description**   |

| ----------------- | ------------ | ----------- | ----------------- |

| `TIMEOUT`         | V            |             | Wait for timeout. |

## Demo

> Note: You should deploy your own service for production usage.

1. Prerender a SPA.

   [\[WebUI\]](https://micro-website-api.now.sh/?cache=true&format=raw&selector=body&url=https%3A%2F%2Fmichaelhsu.tw)

   [\[API\]](https://micro-website-api.now.sh/api?cache=true&format=raw&selector=body&url=https%3A%2F%2Fmichaelhsu.tw)

2. With target `selector`.

   [\[WebUI\]](https://micro-website-api.now.sh/?cache=true&format=raw&selector=%23ctl00_PlaceHolderEmptyMain_PlaceHolderMain_fecurrentid_gvResult&url=http%3A%2F%2Fwww.tcb-bank.com.tw%2Ffinance_info%2FPages%2Fforeign_spot_rate.aspx)

   [\[API\]](https://micro-website-api.now.sh/api?cache=true&format=raw&selector=%23ctl00_PlaceHolderEmptyMain_PlaceHolderMain_fecurrentid_gvResult&url=http%3A%2F%2Fwww.tcb-bank.com.tw%2Ffinance_info%2FPages%2Fforeign_spot_rate.aspx)

3. `Click` to send an ajax and wait for a `selector`.

   [\[WebUI\]]()

   [\[API\]]()

4. `Click` to send an ajax and wait for a long `timeout`.

   [\[WebUI\]]()

   [\[API\]]()

5. Respond `JSON` format.

   [\[WebUI\]](https://micro-website-api.now.sh/?cache=true&format=json&selector=%23root%20%3E%20div&url=https%3A%2F%2Fmichaelhsu.tw%2F)

   [\[API\]](https://micro-website-api.now.sh/api?cache=true&format=json&selector=%23root%20%3E%20div&url=https%3A%2F%2Fmichaelhsu.tw%2F)

6. Multiple actions.

   [\[WebUI\]]()

   [\[API\]]()

## Developer Guide

### Requirements

* node >= 9.1.0

* npm >= 5.5.1

* yarn >= 1.3.2

```

$ git clone https://github.com/evenchange4/micro-website-api.git

$ yarn install --pure-lockfile

$ yarn run dev # dev server

$ yarn start   # prod server

```

### Test

```

$ yarn run format

$ yarn run eslint

$ yarn run test:watch

$ yarn run flow

```

## Inspiration

* https://wrapapi.com/

* https://github.com/ebidel/try-puppeteer

* https://github.com/zenato/puppeteer-renderer

## CONTRIBUTING

* ⇄ Pull requests and ★ Stars are always welcome.

* For bugs and feature requests, please create an issue.

* Pull requests must be accompanied by passing automated tests (`$ yarn run

  test`).

## [CHANGELOG](CHANGELOG.md)

## [LICENSE](LICENSE)

MIT: [http://michaelhsu.mit-license.org](http://michaelhsu.mit-license.org)

[travis-badge]: https://img.shields.io/travis/evenchange4/micro-website-api/master.svg?style=flat-square

[travis]: https://travis-ci.org/evenchange4/micro-website-api

[codecov-badge]: https://img.shields.io/codecov/c/github/evenchange4/micro-website-api.svg?style=flat-square

[codecov]: https://codecov.io/github/evenchange4/micro-website-api?branch=master

[dependency-badge]: https://david-dm.org/evenchange4/micro-website-api.svg?style=flat-square

[dependency]: https://david-dm.org/evenchange4/micro-website-api

[devdependency-badge]: https://david-dm.org/evenchange4/micro-website-api/dev-status.svg?style=flat-square

[devdependency]: https://david-dm.org/evenchange4/micro-website-api#info=devDependencies

[peerdependency-badge]: https://david-dm.org/evenchange4/micro-website-api/peer-status.svg?style=flat-square

[peerdependency]: https://david-dm.org/evenchange4/micro-website-api#info=peerDependencies

[license-badge]: https://img.shields.io/github/license/evenchange4/micro-website-api.svg?style=flat-square

[license]: http://michaelhsu.mit-license.org/

[greenkeeper-badge]: https://badges.greenkeeper.io/evenchange4/micro-website-api.svg

[greenkeeper]: https://greenkeeper.io/

[prettier-badge]: https://img.shields.io/badge/styled_with-prettier-ff69b4.svg?style=flat-square

[prettier]: https://github.com/prettier/prettier