https://github.com/financial-times/ft-poller
Scheduled, asynchronous JSON fetching for Node.js applications
https://github.com/financial-times/ft-poller
component customer-products platforms-customer-products
Last synced: 8 months ago
JSON representation
Scheduled, asynchronous JSON fetching for Node.js applications
- Host: GitHub
- URL: https://github.com/financial-times/ft-poller
- Owner: Financial-Times
- Created: 2014-07-28T09:06:00.000Z (almost 12 years ago)
- Default Branch: main
- Last Pushed: 2024-12-03T16:17:24.000Z (over 1 year ago)
- Last Synced: 2024-12-03T17:29:09.208Z (over 1 year ago)
- Topics: component, customer-products, platforms-customer-products
- Language: JavaScript
- Homepage:
- Size: 937 KB
- Stars: 12
- Watchers: 93
- Forks: 5
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Codeowners: CODEOWNERS
Awesome Lists containing this project
README
Scheduled, asynchronous JSON fetching for Node.js applications. [Background](#background);
### installation
```
npm install ft-poller
```
### API
You can create an instance of Poller like so,
```
var Poller = require('ft-poller');
var p = new Poller(config);
```
Where `config` is an object with the following properties
* `url` [required]: Url to fetch data from
* `defaultData` [recommended]: Data to return if the poller is yet to make a successful request. Typically this will be an empty object of the same type/structure as a successful response e.g. if a successful response would give you an array of users then set `defaultData: []`
* `options` [optional]: options object to pass to `fetch`. If options is not defined or doesn't contain a `signal` property, the request timeout will be set to 4000ms by default. If `retry` is specified then the fetch will be retried the given number of times before erroring
* `refreshInterval` [default: 60000]: Number of milliseconds to wait between request for data
* `autostart` [default: false]: Whether to start the poller automatically when the instance is created
* `parseData` [optional]: function to post-process the data returned by the request. Should return the post-processed data e.g
```
parseData: function (data) {
return data.rows;
}
```
`parseData` can be any function you like and there's nothing to stop you using it to mutate any other variables in scope.
#### methods
* `start()` - Starts the poller. If passed an object `{ initialRequest: true }` it will send its first request immediately, otherwise it will wait until `config.refreshInterval` milliseconds. Returns a promise for the result of the first request (if `initialRequest` is true), or an empty resolved Promise otherwise
* `stop()` - Stops polling
* `getData()` - Returns the last set of data retrieved from the server (post-processed if `parseData` function exists). This will throw an `HttpError` if the most recent fetch received an error.
#### Events
* `error` - emits an error whenever a request returns with an error
* `ok` - emits the response whenever a request returns successfully
### Background
The classic request cycle for a web application follows a call from a client
to the server, which in turn makes one or more further requests to some
underlying service(s).
+---> Web service 1 --> Data
|
Client ---> Presentation tier --|---> Web service 2 --> Data
|
+---> Web service 3 --> Data
Once the data has been retrieved the response makes its way back through the
various layers to the client.
This causes two problems.
Firstly, your response is dependent on the slowest service to respond. If every
request is hanging around waiting for 'the slow one' your performance is pegged
to the worst performing part of your application.
Secondly, by far the slowest thing in this type of architecture is the
round-trip between the presentation tier and the service(s). The more of these
open connections you have hanging around, waiting to close, the greater the
burden you place on your server.
#### Async
Often though, and this is especially true of News sites, the data doesn't
change radically from second to second so this round trip is wasted effort.
It's much more efficient for each presentation tier server to periodically
fetch the data it needs (or listen for a message to signal when new content is
available), stash it in memory, then use that to service any incoming requests.
This suits a [microservice
architecture](http://martinfowler.com/articles/microservices.html), where many
discrete modules, APIs etc. need to be assembled by a presentation tier before
being rendered out to the client (as HTML, JSON etc.).
This pattern (of asynchronous fetching) allows the presentation tier to focus on
building a response from existing data (in memory) and sending it back out the
front door as quickly as possible.