{"id":23009867,"url":"https://github.com/openactive/harvesting-utils","last_synced_at":"2026-06-07T22:33:14.562Z","repository":{"id":223711408,"uuid":"760422768","full_name":"openactive/harvesting-utils","owner":"openactive","description":"Utils library for harvesting RPDE feeds","archived":false,"fork":false,"pushed_at":"2025-05-12T15:41:34.000Z","size":2374,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-10-12T09:54:11.960Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openactive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-20T11:49:48.000Z","updated_at":"2025-05-12T15:41:38.000Z","dependencies_parsed_at":"2024-11-08T17:34:16.849Z","dependency_job_id":null,"html_url":"https://github.com/openactive/harvesting-utils","commit_stats":null,"previous_names":["openactive/harvesting-utils"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openactive/harvesting-utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openactive%2Fharvesting-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openactive%2Fharvesting-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openactive%2Fharvesting-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openactive%2Fharvesting-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openactive","download_url":"https://codeload.github.com/openactive/harvesting-utils/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openactive%2Fharvesting-utils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34041087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-07T02:00:07.652Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-15T09:16:14.584Z","updated_at":"2026-06-07T22:33:14.544Z","avatar_url":"https://github.com/openactive.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# harvesting-utils\nUtils library for harvesting [RPDE feeds](https://openactive.io/realtime-paged-data-exchange/).\n\n## Version 0.X.X\n\nThis library is currently in version 0.X.X, which means that the API will not be stable until 1.0.0.\n\n## Install\nThis library can be installed as an npm package using the following command:\n\n```\n$ npm install git://github.com/openactive/harvesting-utils.git\n```\n\n## Usage\n\n```js\nconst { harvestRPDE } = require('@openactive/harvesting-utils')\n\nharvestRPDE({\n  baseUrl: '...',\n  /* ...relevant parameters here */\n});\n```\n\n## Examples\nA very simple example of `harvestRPDE` can be found in `examples/simple-rpde-harvester.js`. For more information on this script see [here](./examples/README.md).\n\n## API Reference\n### harvestRPDE\n\nIndefinitely harvests an RPDE feed, following the [\"expected consumer behaviour\" described in the RPDE spec](https://openactive.io/realtime-paged-data-exchange/#expected-consumer-behaviour).\n\n**N.B.** This function will run indefinitely, and only return if a fatal error occurs. For this reason, you will generally **not** want to run `await harvestRPDE(..)`.\n\n#### Parameters\n\nThe parameters for this function are typed and documented in [HarvestRpde.d.ts](./src/models/HarvestRpde.d.ts).\n\n### harvestRpdeLossless\n`harvestRpdeLossless` has the same function signature as `harvestRpde`. However it is capable of handling `modified` values that are too large for JavaScript numbers to handle natively ie \u003e 2^53. This is handled by storing them as strings in memory. \n\nFor more guidance on how to handle these values, see [here](https://developer.openactive.io/using-data/harvesting-opportunity-data/large-integers-in-javascript).\n\n## Developing\n### TypeScript\n\nThe code is written in native JS, but uses TypeScript to check for type errors. TypeScript uses JSDoc annotations to determine types (See: [Type Checking JavaScript Files](https://www.typescriptlang.org/docs/handbook/type-checking-javascript-files.html)) from our native .js files.\n\nIn order for these types to be used by other projects, they must be saved to [TypeScript Declaration files](https://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html). This is enabled by our tsconfig.json, which specifies that declaration files are to be generated and saved to `built-types/` (As an aside, the reason that the package's types must be saved to .d.ts files is due to TypeScript not automatically using JS defined types from libraries. There is a good reason for this and proposals to allow it to work at least for certain packages. See some of the discussion here: https://github.com/microsoft/TypeScript/issues/33136).\n\nFor this reason, TypeScript types should be generated after code changes to make sure that consumers of this library can use the new types. The openactive-test-suite project does this automatically in its pre-commit hook, which calls `npm run gen-types`\n\nTypeScript-related scripts:\n\n- `check-types`: This uses the `tsconfig.check.json` config, which does not emit any TS declaration files - all it does is check that there are no type errors. This is used for code tests.\n- `gen-types`: This uses the `tsconfig.gen.json` config, which emits TS declaration files into `built-types/`.\n\n  Additionally, it copies programmer-created `.d.ts` files from our source code (e.g. `src/types/Criteria.d.ts`) into `built-types/`. This is because our code references these types, so they must be in the `built-types/` directory so that the relative paths match (e.g. so that `import('../types/Criteria').Criteria` works).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenactive%2Fharvesting-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenactive%2Fharvesting-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenactive%2Fharvesting-utils/lists"}