{"id":16701817,"url":"https://github.com/michaelnisi/manger","last_synced_at":"2025-07-21T10:05:31.237Z","repository":{"id":11460243,"uuid":"13923630","full_name":"michaelnisi/manger","owner":"michaelnisi","description":"Cache RSS and Atom feeds","archived":false,"fork":false,"pushed_at":"2022-12-07T17:39:42.000Z","size":1879,"stargazers_count":5,"open_issues_count":15,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-29T11:02:44.025Z","etag":null,"topics":["atom","caching","feeds","rss"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michaelnisi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-10-28T11:11:36.000Z","updated_at":"2021-04-13T19:33:58.000Z","dependencies_parsed_at":"2023-01-11T20:16:40.896Z","dependency_job_id":null,"html_url":"https://github.com/michaelnisi/manger","commit_stats":null,"previous_names":[],"tags_count":72,"template":false,"template_full_name":null,"purl":"pkg:github/michaelnisi/manger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fmanger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fmanger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fmanger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fmanger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michaelnisi","download_url":"https://codeload.github.com/michaelnisi/manger/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fmanger/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266278376,"owners_count":23904041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atom","caching","feeds","rss"],"created_at":"2024-10-12T18:45:49.091Z","updated_at":"2025-07-21T10:05:31.207Z","avatar_url":"https://github.com/michaelnisi.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://secure.travis-ci.org/michaelnisi/manger.svg)](http://travis-ci.org/michaelnisi/manger)\n[![Coverage Status](https://coveralls.io/repos/github/michaelnisi/manger/badge.svg?branch=master)](https://coveralls.io/github/michaelnisi/manger?branch=master)\n\n# manger - cache feeds\n\nThe Manger [Node.js](http://nodejs.org/) package provides caching for RSS and Atom formatted XML feeds, it implements an interface to query entries by feed and time. The obvious challenge here is to build a resilient system facing potentially misconfigured servers and malformed feeds. Most of Manger’s API is implemented as streams.\n\nManger leverages the lexicographical key sort order of [LevelDB](http://leveldb.org/). The keys are designed to stream feeds or entries in time ranges between now and some user defined point in the past.\n\n## REPL\n\nThere’s a REPL for exploring the API.\n\n```\n$ npm start\n\n\u003e manger@8.0.0 start /Users/michael/node/manger\n\u003e ./repl.js\n\nmanger\u003e const feeds = cache.feeds()\nmanger\u003e feeds.write('http://rss.art19.com/the-daily')\ntrue\nmanger\u003e read(feeds, 'title')\nmanger\u003e 'The Daily'\nmanger\u003e\n```\n\n## Data and Types\n\n### void()\n\n`null | undefined` Absence of any object value, intentional or not.\n\n### str()\n\n`String() | void()` An optional [`String()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String).\n\n### html()\n\nA sanitized HTML `String()` with a limited set of tags: `'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'blockquote', 'p', 'a', 'ul', 'ol', 'li', 'b', 'i', 'strong', 'em', 'code', 'br', 'div', 'pre'`.\n\n### enclosure()\n\nA related resource of an `entry()`.\n\n- `href` `str()`\n- `length` `Number()`\n- `type` `str()`\n\n### entry()\n\nAn individual entry.\n\n- `author` `str()`\n- `duration` `Number() | null` The value of the `\u003citunes:duration\u003e` tag in seconds or `null`.\n- `enclosure` `enclosure() | void()`\n- `id` `String()` A globally unique, not the original, (SHA-1) identifier for this entry.\n- `image` `str()`\n- `link` `str()`\n- `originalURL` `str()` The originally requested URL.\n- `subtitle` `str()`\n- `summary` `html() | void()`\n- `title` `str()`\n- `updated` `str()`\n- `url` `str()` The URL of this entry’s feed.\n\nWhy SHA-1, cryptographic hashing, to produce the `id` property?\n\n\u003e Having a good hash is good for being able to trust your data, it happens to have some other good features, too, it means when we hash objects, we know the hash is well distributed and we do not have to worry about certain distribution issues.\n\nRead more [here](https://stackoverflow.com/questions/28792784/why-does-git-use-a-cryptographic-hash-function).\n\n### feed()\n\nOne metadata object per XML feed.\n\n- `author` `str()`\n- `copyright` `str()`\n- `id` `str()`\n- `image` `str()`\n- `language` `str()`\n- `link` `str()`\n- `originalURL` `str()`\n- `payment` `str()`\n- `subtitle` `str()`\n- `summary` `str()`\n- `title` `str()`\n- `ttl` `str()`\n- `updated` `Number()`\n- `url` `str()`\n\n## Creating a Cache\n\n```js\nconst {createLevelDB, Manger, Opts} = require('manger');\n```\n\nWe need a [Level](https://github.com/Level/level) database for creating a cache. Additionally, we can pass options or use defaults.\n\n### Opts\n\nOptions for a `Manger` instance.\n\n- `cacheSize = 16 * 1024 * 1024` [LevelDB](https://github.com/google/leveldb) cache size for uncompressed blocks.\n- `counterMax = 500` Limits the items in the ranks counter.\n- `failures = { set, get, has }` LRU cache for failures.\n- `force = false` A flag to bypass cached data entirely.\n- `highWaterMark` Buffer level when `stream.write()` starts returning `false`.\n- `isEntry = function (entry) { return true }` A function to validate entries.\n- `isFeed = function (feed) { return true }` A function to validate feeds.\n- `objectMode = false` Read `Object()` instead of `Buffer()`.\n- `redirects = { set, get, has }` LRU cache for redirects.\n\n```js\nconst db = createLevelDB('/tmp/manger');\nconst opts = {}; // | new Opts()\nconst cache = new Manger(db, opts);\n```\n\n## Querying the Manger Cache\n\nA Manger cache object provides multiple streams to access data. Selecting data works by writing query objects to these streams.\n\n```js\nconst {Queries, Query} = require('manger');\n```\n\n### Query\n\nA Query to get a feed or entries of a feed in a time range between `Date.now()` and `since`. Conceptually consequent, but semantically inaccurate, the `since` date is exclusive. If you pass the `updated` date of the latest entry received, this entry will not be included in the response.\n\n- `url` `String()`\n- `since` [`Date()`](https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Date) `| void()`\n- `etag` `String() | void()` An [entity tag](http://en.wikipedia.org/wiki/HTTP_ETag)\n- `force` [`Boolean()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Boolean)`| false` Force update ignoring cache\n\nSourced with inaccurate URLs the `Query` constructor throws.\n\n### Queries\n\nFor convenience, the `Queries` class transforms JSON to queries which can be piped to `feeds()` and `entries()`. The expected JSON input format:\n\n```js\n[\n  { \"url\": \"http://feeds.5by5.tv/directional\" },\n  { \"url\": \"http://www.newyorker.com/feed/posts\",\n    \"since\": 1433083971124 },\n  { \"url\": \"https://www.joyent.com/blog/feed\",\n    \"since\": \"May 2015\" },\n  ...\n]\n```\n\nWhere `\"since\"` can be anything `Date()` is able to parse.\n\n## Querying Feeds\n\nThe distinction between feed and entries might be unclear: a feed models the metadata of an RSS or Atom feed (title, author, published, etc.), while entries are the actual items in the feed. These are detached to not repeatedly transmit feed metadata—after all Manger tries to reduce round-trips.\n\n### cache.feeds()\n\nReturns a [Transform](http://nodejs.org/api/stream.html#stream_class_stream_transform) stream that transforms queries or URL strings to feeds.\n\n- `write(query() | String())`\n- `read()` `Buffer() | String() | Object() | feed()`\n\n```js\nconst feeds = cache.feeds();\n\nfeeds.on('readable', () =\u003e {\n  let data;\n\n  while ((data = this.read())) {\n    console.log(data);\n  }\n});\n\nfeeds.write(new Query({url: 'http://rss.art19.com/the-daily'}));\n```\n\n## Querying Entries\n\nAccess granular ranges of items within multiple feeds at once – using a single stream.\n\n### cache.entries()\n\nReturns a [Transform](http://nodejs.org/api/stream.html#stream_class_stream_transform) stream that transforms queries or URLs to entries.\n\n- `write(Buffer() | String() | Object() | query())`\n- `read()` `Buffer() | entry()`\n\n```js\nconst entries = cache.entries();\n\nentries.on('readable', () =\u003e {\n  let data;\n\n  while ((data = this.read())) {\n    console.log(data);\n  }\n});\n\nentries.write(new Query({url: 'http://rss.art19.com/the-daily'}));\n```\n\n## Updating the Cache\n\nIt’s common to update all cached feeds on a regular basis. Manger uses a counter to reason about which feeds to update first.\n\n### cache.flushCounter(cb)\n\n- `cb` `Function(error, count) | void()` The callback\n  - `error` `Error() | void()` The possible error\n  - `count` `Number()` Total number of cached feeds\n\nA Manger cache keeps an in-memory count of how many times feeds have been accessed. This function flushes the counter to disk, updating the ranks index.\n\n### cache.update()\n\nUpdates all ranked feeds and returns a stream that emits feed URLs of updated feeds. This, of course, is a **resource heavy long-running** operation! Feeds are updated ordered by their popularity, using the rank index, therefor `flushCounter` must have been invoked before this method takes any effect.\n\n- `read()` `str()`\n\n## Removing Feeds from the Cache\n\nSometimes feeds contain redundant or unwanted items, so you might want to remove them from the cache.\n\n### cache.remove(url, cb)\n\nAttempts to remove a feed matching the `url` from the cache and applies callback without `error` if this succeeds.\n\n- `url` `String()` The URL of the feed\n- `cb` `Function(error) | void()` The callback\n  - `error` `Error() | void()` The possible error\n\n## Observing Cache Contents\n\n### cache.list()\n\nA [Readable](http://nodejs.org/api/stream.html#stream_class_stream_readable_1) stream of URLs of all feeds currently cached.\n\n- `read()` `Buffer() | str()`\n\n### cache.has(url, cb)\n\nApplies callback `cb` without arguments if a feed with this `url` is cached.\n\n- `url` `String()` The URL of the feed\n- `cb` `Function(error) | void()` The callback\n\n  - `error` `Error() | void()` The possible error\n\n### cache.ranks(limit)\n\nA stream of URLs sorted by rank (highest first).\n\n- `limit` `Number()` Optionally, limit the number of URLs.\n\nThis stream lets you:\n\n- `read()` `Buffer() | str()`\n\n### cache.resetRanks(cb)\n\nResets the ranks index.\n\n- `cb` [`Function`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function)`(error)` The callback\n  - `error` [`Error()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) `| void()` The possible error\n\n### Event: 'hit'\n\n- `query()` The query that hit the cache.\n\nFor each cache hit the Manger cache emits a `'hit'` event.\n\n## Installation\n\nWith [npm](https://npmjs.org/package/manger), do:\n\n```\n$ npm install manger\n```\n\n## License\n\n[MIT License](https://raw.github.com/michaelnisi/manger/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelnisi%2Fmanger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelnisi%2Fmanger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelnisi%2Fmanger/lists"}