{"id":13432514,"url":"https://github.com/npm/registry-follower-tutorial","last_synced_at":"2025-10-07T04:30:24.489Z","repository":{"id":65992611,"uuid":"61908242","full_name":"npm/registry-follower-tutorial","owner":"npm","description":"write you a registry follower for great good","archived":true,"fork":false,"pushed_at":"2020-07-17T05:07:49.000Z","size":41,"stargazers_count":58,"open_issues_count":0,"forks_count":21,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-04-14T06:52:37.905Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/npm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-06-24T19:41:17.000Z","updated_at":"2024-03-19T02:47:38.000Z","dependencies_parsed_at":"2023-03-10T15:15:58.384Z","dependency_job_id":null,"html_url":"https://github.com/npm/registry-follower-tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npm%2Fregistry-follower-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npm%2Fregistry-follower-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npm%2Fregistry-follower-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npm%2Fregistry-follower-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/npm","download_url":"https://codeload.github.com/npm/registry-follower-tutorial/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235586137,"owners_count":19014029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T02:01:12.662Z","updated_at":"2025-10-07T04:30:19.222Z","avatar_url":"https://github.com/npm.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# so you want to write a follower\n\u003e ch-ch-ch-ch-changes\n\nThis tutorial will teach you how to write a generic boilerplate\nNodeJS application that can manipulate, respond to, broadcast, analyze,\nand otherwise play with package metadata as it changes in the npm registry.\n\nWait...what? Why?\n\nHere's the deal: do you want to have some fun with the `package.json` data\nfrom every version of every package in the npm registry? Some neat ideas:\n\n- Find all the package `README`s that mention dogs\n- Discover how many package authors are named \"Kate\"\n- Calculate how many dependency changes occur on average in a major version\n   bump\n\nAnd more! So stop waiting and write a follower!\n\n## prerequisites\n\nIn order to follow along with this tutorial you'll need [NodeJS] and\n[npm]. I recommend installing these using a version manager; I use [nvm].\n\n[NodeJS]: https://nodejs.org\n[npm]: https://www.npmjs.com/\n[nvm]: https://github.com/creationix/nvm\n\n## application setup\n\nLet's set up our application:\n\n1. Create a directory called `follower-tutorial`. \n  (`mkdir follower-tutorial`)\n2. Move into that directory (`cd follower-tutorial`).\n3. Initialize an npm project by typing `npm init --yes`. This will create a\n  `package.json` with default values.\n4. Create a file called `.gitignore` and add the line `node_modules` to it.\n\nOur application currently looks like this:\n\n```\n+ follower-tutorial\n  |- .gitignore\n  |- package.json\n```\n\n## dependencies\n\nOur application is going to depend on a couple super helpful npm packages:\n\n- [`changes-stream`]: This package gives us access to a [stream] of changes\n  from the npm registry's CouchDB. We'll listen for and respond to events\n  from this stream in our app. These events represent changes in the npm\n  registry.\n- [`request`]: This package allows us to make HTTP requests. We'll use this\n  to retrieve the current total number of changes currently in the \n  database so that we can optionally end our progarm when it has received\n  all the current changes.\n\n[`changes-stream`]: https://www.npmjs.com/package/changes-stream\n[stream]: https://nodejs.org/api/stream.html\n[`request`]: https://www.npmjs.com/package/request\n\nTo install these packages we'll type:\n\n```sh\nnpm install changes-stream request --save\n```\n... which will add both of our dependencies to our `package.json`.\n\n## set up a changes stream\n\nLet's start writing our application now! We'll be writing our application\nin an `index.js` file at the root of our `follower-tutorial` application\ndirectory:\n\n```\n+ follower-tutorial\n  |- .gitignore\n  |- index.js       // \u003c-- here's where our app goes\n  |- package.json\n```\n\nThe first thing we'll do inside our `index.js` is use the `changes-stream`\npackage to create a new `ChangesStream` to listen to listen to the npm\nregistry. To do so we'll write:\n\n```js\n1  const ChangesStream = require('changes-stream');\n2 \n3  const db = 'https://replicate.npmjs.com';\n4 \n5  var changes = new ChangesStream({\n6    db: db\n7  });\n``` \n\nLet's talk about what's happening here:\n\n- On line 1, we require the `changes-stream` package\n- On line 3, we save the URL of the npm registry db\n- On lines 5-7, we create a new ChangesStream instance, passing it an\n  options object that points to our db\n\nNow that we've created a changes stream, let's listen to it! To do this, we\nwrite:\n\n```js\n9  changes.on('data', function(change) {\n10   console.log(change);\n11 });\n```\n\nLet's test it out: Run this application by typing:\n\n```sh\nnode index.js\n```\n\nIf everything is working correctly, you'll see something like this start\n**streaming** through your console:\n\n```sh\n{ seq: 445,\n  id: 'CompoundSignal',\n  changes: [ { rev: '5-a0695c30fdaa3471246ef0cd6c8a476d' } ] }\n{ seq: 446,\n  id: 'amphibian',\n  changes: [ { rev: '5-1a864e76d844e90bf6c63cb94303b593' } ] }\n{ seq: 447,\n  id: 'aop',\n  changes: [ { rev: '9-9acc0139df57a1db2604f13f12b500f2' } ] }\n{ seq: 448,\n  id: 'dynamo-schema',\n  changes: [ { rev: '5-bf8052c0d4b6e80e6664625137efd610' } ] }\n{ seq: 451,\n  id: 'password-reset',\n  changes: [ { rev: '21-948e6633799ffd56a993c3fb144d1728' } ] }\n```\n\nIf you don't see that, and/or got an error, reread the sample code in this\nsection and be sure you don't have any typos! If you continue having\ntrouble, [file an issue on this repo].\n\n[file an issue on this repo]: /npm/registry/issues/new\n\nOtherwise... Congrats! You have a successful registry follower. Hurry up\nand hit `ctrl-c` - this stream won't ever exit the way we've written it\nnow!\n\n## moar data please\n\nSo our follower works! But it's not that great right now because we don't \nreally have all that much interesting data. Let's look at what we have\nright now:\n\n```sh\n{ seq: 446,\n  id: 'amphibian',\n  changes: [ { rev: '5-1a864e76d844e90bf6c63cb94303b593' } ] }\n```\n\n- `seq`: the package's order in the sequence of change events\n- `id`: the name of the `package` (sometimes this is something else! we'll\n  get to that in a bit tho, it doesn't matter too much right now.)\n- `changes`: an array containing a single object, with a single key `rev`\n  that point to a change id\n\nLet's be real: this data is not *that* interesting. Where's the good stuff?\nIt turns out that the fun data is in a key called `doc` that we need to\ntell our `ChangesStream` instance, `changes`, to specifically grab. To do\nthis, we'll add `include_docs: true` to the options object we pass to the\n`ChangesStream` constructor.\n\nOnce we've told our ChangesStream to `include_docs`, we get some new\nawesome data.  This new data lives off of a key called `doc` on the `change`\nobject we received from the stream.\n\nThe two changes we make to our code look like this:\n\n```js\n5  var changes = new ChangesStream({\n6    db: db,\n7    include_docs: true            // \u003c- this is the thing we're adding\n8  })\n9\n10 changes.on('data', function(change) {\n11   console.log(change.doc)      // \u003c- so that we can add `.doc` here\n12 });\n```\n\nLet's test it out by running `node index.js`. Assuming you've made all the\nchanges we described above you should be seeing a LOT more data. Here's a\nsummary of what you get:\n\n- `_id`: the name of the package\n- `_rev`: the revision id\n- `name`: the name of the package\n- `description`: the package description\n- `'dist-tags'`: an object with all dist-tag names and versions\n- `versions`: a nested object where every version is a key, and an object of\n  all of the package metadata for that key is the value (this includes: \n  `main`, `directories`, `dependencies`, `scripts`, `engines`, `bin`, \n    `devDependencies`, and more)\n- `maintainers`: an array of objects, each representing info about a \n  maintainer (name, email, website)\n- `time`: timestamps for every version of the package published, plus\n  `created` and `modified`\n- `author`: an object representing the author of the package (name, email\n  website)\n- `repository`: an object representing the location of the package code,\n  e.g. `{ type: 'git', url: 'git://github.com/my/gitrepo.git' }`\n\nTake a moment and play around with your application by exploring the\ndifferent pieces of data you can get from this stream. You may notice that\nsome nested structures appear like `[Object]` in your console. You can\nprint those out by adding `JSON.stringify` to your log, like this:\n\n```js\nconsole.log(JSON.stringify (change.doc,null,' '));\n```\n\n...which will ensure that the nested objects aren't flattened (e.g. appear\nlike `[Object]`).\n\nNote: you'll have to `ctrl-C` out of your application every time you run\nit. It's still a never ending stream! In the next section we'll explain how\nto make it stop. \n\n## a never ending stream\n\nAs we mentioned in the previous section, our follower currently won't ever\nstop! Let's dive a little deeper into why that's the case:\n\nOur application functions by listening for an event called `data` from\na stream coming out of npm's registry, each event hands us an object that\nrepresents a `change` in the registry... and the registry is changing all\nthe time!\n\nLuckily, our db endpoint gives us some useful info to help us consume just\nthe current changes as they exist at the time of access, and then stop the \nprocess.\n\nNavigate your browser to our db url:\n\n```\nhttps://replicate.npmjs.com\n```\n\n...you should see something that looks like this:\n\n```json\n{\n  \"db_name\": \"registry\",\n  \"doc_count\": 345391,\n  \"doc_del_count\": 355,\n  \"update_seq\": 2496579,\n  \"purge_seq\": 0,\n  \"compact_running\": false,\n  \"disk_size\": 1713074299,\n  \"data_size\": 1320944467,\n  \"instance_start_time\": \"1466084344558224\",\n  \"disk_format_version\": 6,\n  \"committed_update_seq\": 2496579\n}\n```\n\nLet's talk about what some of these mean:\n\n- `doc_count`: is the number of documents that the db contains\n- `update_seq`: is the number of changes that are stored in the db\n\n`update_seq` is the important bit of information here. As stated, it \nrepresents the number of `change` resources contained in the db. Remember\nthe `seq` attribute on the `change` object we received from the stream?\nThat number counts up until `update_seq`! This means that we can use this\nnumber to tell our follower to stop when `change.seq` meets or exceeds the\n`update_seq` value, signifying that it has processed all the changes in the\ndb up until the time we accessed the `update_seq` value.\n\nThat was a lot of words, let's take a look at what this would look like in\ncode.\n\n```js\n2  const Request = require('request');\n...\n11 Request.get(db, function(err, req, body) {        // \u003c- make a request to the db\n12   var end_sequence = JSON.parse(body).update_seq; // \u003c- grab the update_seq value\n13   changes.on('data', function(change) {\n14     if (change.seq \u003e= end_sequence) {               // \u003c- if we're at the last change\n15       process.exit(0);                            // \u003c- end the program successfully (\"0\")\n16     }\n17     console.log(change.doc);\n18   }) \n19 });\n```\n\nLet's walk through what this code is doing:\n\n- On line 2, we are require the `request` package\n- On line 11, we are making a request to our db using the `request` package\n- On line 12, we parse the response from our request and grab the `update_seq`\n  value.\n- On line 13, on every `data` event, we check to see if the `change.seq`\n  value we get is greater than or equal to `update_seq`. Why `\u003e=` and\n  not just equal? Remember that the registry is *always* changing, and there's\n  a good chance it will change while we are following it! Using `\u003e=` means that\n  we can account for the change that happens while our application is running. \n- On line 15, we end our program. We send the value `0` to `process.exit` to\n  indicate that we are ending the program successfully, i.e. not with an error.\n\nOk! Given this code, our application will now run for all the current changes in the\nregistry and then exit. Take a moment and give it a go! Note: There are a lot of\nchanges, so this can take up to an hour.\n\n## clean up\n\nSo our follower is pretty much done! However, there's a few things that ain't quite\nright about our data. Let's do that now so we can finish up.\n\nFirstly, remember the `id`/`_id` key we receive from our changes stream? We had\nidentified that as being the name of the package, but that was a generalization.\nIt turns out that there are actually 2 types of things in the changes db: changes\nand \"design docs\".\n\n\"Design docs\"? What? Right. To understand this requires understand how CouchDB works\na bit. One way to program with CouchDB is to write an application **within** the db.\nAt this point, npm is moving away from this structure, but at one point (and still!)\nthe registry was/is written as a CouchDB application. These applications exist as \n\"design docs\" inside the db, so when receive data from the db, *sometimes* we receive\nthese design docs. If you watched your follower closely, you'd notice that \n*sometimes* it's logging `undefined`. Those are the \"design docs\".\n\nTo ignore these files, we can just check to see if a `change` is an actual package\nby checking if it has a `name`. We can accomplish this by checking if \nchange.doc.name` has a value before we do anything with the `change` data. In our\ncode, this looks like this:\n\n```js\n...\n17 if (change.doc.name) {             // \u003c-- make sure the change is a change\n18   console.log(change.doc);\n19 }\n``` \n\nOk, so we're **almost** done. Actually, we are totally done. But there's one last\nthing we can do to make our data even better: we can normalize our data so that\nit is nearly exactly the same as the CLI uses and is returned by http://registry.npmjs.com.\n\nTo do this, we'll add *one more* dependency to our application: [`normalize-registry-metadata`].\n\n[`normalize-registry-metadata`]: https://github.com/npm/normalize-registry-metadata\n\nFirst things first: let's install this package and save it to our `package.json`:\n\n```sh\nnpm install normalize-registry-metadata --save\n```\n\nNext, we require in our `index.js`:\n\n```js\n3  const Normalize = require('normalize-registry-metadata');\n```\n\nLastly, let's call `Normalize()` on the `change` data before we log it to the console:\n\n```js\n...\n18   console.log(Normalize(change.doc));        // \u003c-- we only have to change this line!\n...\n```\n\nAnd we're done! You can double check that your code is correct by looking at the \ncomplete code [here].\n\n[here]: https://github.com/ashleygwilliams/registry-follower-tutorial/blob/master/index.js\n\n## forever follower\n\nDon't want to stop? Want to write a persistent follower? You can move\nforward with our app as it is currently written, however you'll likely have\na better experience replacing `changes-stream` with \n[`concurrent-couch-follower`] which is safer for operations that may \nrequire async (like a file write!). [`concurrent-couch-follower`] remembers\nthe last change you processed and can start back from there if at some point\nyou need to restart the program.\n\n[`concurrent-couch-follower`]: https://github.com/npm/concurrent-couch-follower\n\n## a few notes on performance\n\nThe vast majority of useful registry followers won't ever have\nany kind of follower-side performance problem.  In keeping with\n[long-established wisdom][wisdom], you probably shouldn't even _think_\nabout this section until you hit a bottleneck in use and confirm it\nby measurement.  Logging Node.js [cpuUsage()] and [memoryUsage()],\n[heap analysis], and the built-in [profiler] are great places to start.\n\n[cpuUsage()]: https://nodejs.org/api/process.html#process_process_cpuusage_previousvalue\n\n[memoryUsage()]: https://nodejs.org/api/process.html#process_process_memoryusage\n\n[heap analysis]: https://www.npmjs.com/package/heapdump\n\n[profiler]: https://nodejs.org/en/docs/guides/simple-profiling/\n\n[wisdom]: http://c2.com/cgi/wiki?PrematureOptimization\n\nUnder the hood, libraries like [`changes-stream`] `GET` the\nregistry's CouchDB-style HTTPS replication endpoint, which streams\nnewline-deliminted JSON objects, one per database update, over\nlong-lived responses.  These are the object chunks you receive from\nthe stream.\n\nMost registry update objects are manageably small, but the deviation\nis great, with a few updates weighing in close to 5 MB.  The bulk\nof this is often (highly repetitive) `README` file strings, one\nper version in `chunk.doc.versions`.  Some packages have thousands\nof versions.  And every once in a while, some fiendish jokester\npublishes a \"novelty\" package that \"depends on\" every other package\nin the registry, as if they were the first to think of it.\n\nEspecially if you're using a pipeline of many object-mode streams\nto process the chunks, you may have high memory usage with Node.js'\ndefault maximum stream internal buffer size, `highWaterMark`, of 16.\nMultiple buffers of 16 objects each, plus lingering data not yet\npicked up by the garbage collector, can eat your RAM lunch quick.\nTo reduce this number:\n\n```\nnew ChangesStream({\n  db: 'https://replicate.npmjs.com',\n  include_docs: true,\n  highWaterMark: 4\n})\n```\n\nMost tried-and-true stream packages, like those in the [Mississippi\nStreams Collection][mississippi], take optional options-object\narguments that get passed along to the core [readable-stream]\nconstructors.  You can set `{highWaterMark: Number}` in those\narguments.\n\n[mississippi]: https://www.npmjs.com/package/mississippi\n\n[readable-stream]: https://www.npmjs.com/package/readable-stream\n\n## go forth and make something awesome!\n\nWe're seriously excited about what you'll build. Please share with us on\ntwitter ([@npmjs])! And please don't hesitate to ask for help in the issues\non this repo :)\n\n[@npmjs]: https://twitter.com/npmjs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnpm%2Fregistry-follower-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnpm%2Fregistry-follower-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnpm%2Fregistry-follower-tutorial/lists"}