https://github.com/openactive/harvesting-utils
Utils library for harvesting RPDE feeds
https://github.com/openactive/harvesting-utils
Last synced: 10 months ago
JSON representation
Utils library for harvesting RPDE feeds
- Host: GitHub
- URL: https://github.com/openactive/harvesting-utils
- Owner: openactive
- License: mit
- Created: 2024-02-20T11:49:48.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-11-13T14:38:26.000Z (over 1 year ago)
- Last Synced: 2024-12-10T00:55:24.290Z (over 1 year ago)
- Language: JavaScript
- Size: 293 KB
- Stars: 0
- Watchers: 7
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# harvesting-utils
Utils library for harvesting [RPDE feeds](https://openactive.io/realtime-paged-data-exchange/).
## Version 0.X.X
This library is currently in version 0.X.X, which means that the API will not be stable until 1.0.0.
## Install
This library can be installed as an npm package using the following command:
```
$ npm install git://github.com/openactive/harvesting-utils.git
```
## Usage
```js
const { harvestRPDE } = require('@openactive/harvesting-utils')
harvestRPDE({
baseUrl: '...',
/* ...relevant parameters here */
});
```
## Examples
A very simple example of `harvestRPDE` can be found in `examples/simple-rpde-harvester.js`. For more information on this script see [here](./examples/README.md).
## API Reference
### harvestRPDE
Indefinitely harvests an RPDE feed, following the ["expected consumer behaviour" described in the RPDE spec](https://openactive.io/realtime-paged-data-exchange/#expected-consumer-behaviour).
**N.B.** This function will run indefinitely, and only return if a fatal error occurs. For this reason, you will generally **not** want to run `await harvestRPDE(..)`.
#### Parameters
The parameters for this function are typed and documented in [HarvestRpde.d.ts](./src/models/HarvestRpde.d.ts).
### harvestRpdeLossless
`harvestRpdeLossless` has the same function signature as `harvestRpde`. However it is capable of handling `modified` values that are too large for JavaScript numbers to handle natively ie > 2^53. This is handled by storing them as strings in memory.
For more guidance on how to handle these values, see [here](https://developer.openactive.io/using-data/harvesting-opportunity-data/large-integers-in-javascript).
## Developing
### TypeScript
The code is written in native JS, but uses TypeScript to check for type errors. TypeScript uses JSDoc annotations to determine types (See: [Type Checking JavaScript Files](https://www.typescriptlang.org/docs/handbook/type-checking-javascript-files.html)) from our native .js files.
In order for these types to be used by other projects, they must be saved to [TypeScript Declaration files](https://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html). This is enabled by our tsconfig.json, which specifies that declaration files are to be generated and saved to `built-types/` (As an aside, the reason that the package's types must be saved to .d.ts files is due to TypeScript not automatically using JS defined types from libraries. There is a good reason for this and proposals to allow it to work at least for certain packages. See some of the discussion here: https://github.com/microsoft/TypeScript/issues/33136).
For this reason, TypeScript types should be generated after code changes to make sure that consumers of this library can use the new types. The openactive-test-suite project does this automatically in its pre-commit hook, which calls `npm run gen-types`
TypeScript-related scripts:
- `check-types`: This uses the `tsconfig.check.json` config, which does not emit any TS declaration files - all it does is check that there are no type errors. This is used for code tests.
- `gen-types`: This uses the `tsconfig.gen.json` config, which emits TS declaration files into `built-types/`.
Additionally, it copies programmer-created `.d.ts` files from our source code (e.g. `src/types/Criteria.d.ts`) into `built-types/`. This is because our code references these types, so they must be in the `built-types/` directory so that the relative paths match (e.g. so that `import('../types/Criteria').Criteria` works).