{"id":16578048,"url":"https://github.com/glynnbird/cloudant-timeseries","last_synced_at":"2026-04-24T05:37:06.979Z","repository":{"id":57200709,"uuid":"122194241","full_name":"glynnbird/cloudant-timeseries","owner":"glynnbird","description":"Cloudant helper library for managing time-series data stored in monthly databases","archived":false,"fork":false,"pushed_at":"2018-03-27T13:46:11.000Z","size":175,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2026-04-21T03:25:31.526Z","etag":null,"topics":["cloudant","nodejs","time-series"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/glynnbird.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-20T12:17:41.000Z","updated_at":"2020-01-28T19:44:53.000Z","dependencies_parsed_at":"2022-09-16T15:11:58.123Z","dependency_job_id":null,"html_url":"https://github.com/glynnbird/cloudant-timeseries","commit_stats":null,"previous_names":["ibm-watson-data-lab/cloudant-timeseries"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/glynnbird/cloudant-timeseries","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fcloudant-timeseries","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fcloudant-timeseries/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fcloudant-timeseries/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fcloudant-timeseries/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/glynnbird","download_url":"https://codeload.github.com/glynnbird/cloudant-timeseries/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glynnbird%2Fcloudant-timeseries/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32211381,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T03:15:14.334Z","status":"ssl_error","status_checked_at":"2026-04-24T03:15:11.608Z","response_time":64,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudant","nodejs","time-series"],"created_at":"2024-10-11T22:12:58.787Z","updated_at":"2026-04-24T05:37:06.943Z","avatar_url":"https://github.com/glynnbird.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cloudant-timeseries\n\nThe *cloudant-timeseries* library is an abstraction that allows multiple monthly databases to be treated as a single database. \n\nImagine you are storing web traffic events in a single, ever-growing Cloudant/CouchDB database with documents like this:\n\n```js\n{\n  \"_id\": \"someid\",\n  \"year\": 2018,\n  \"month\", 1,\n  \"day\": 24,\n  \"hour\": 20,\n  \"minute\": 59,\n  \"second\": 0,\n  \"event_type\": \"click\",\n  \"browser\": \"chrome\",\n  \"page_speed\": 1.042\n}\n```\n\nand using Cloudant's MapReduce views to aggregate the data, grouping by year, month, day etc. This works great until you come to archive your old data. How do you archive the data from 2015 if all the data is in the same ever-growing database? It turns out you can't. You can delete individual documents but that's impractical when you want to remove a year's worth of data. Deletion also leaves a stub document hanging around so you don't get all of your disk space back.\n\nThe only way to cleanly delete data is to delete a whole database. This is where *cloudant-timeseries* comes in.\n\nYou interact with library as if you are talking to a single ever-growing database, but under-the-hood it is a collection of monthly databases where data is added to the current month's database:\n\n![schematic](images/timeseries1.png)\n\nQueries are directed at all databases and the results aggregated:\n\n![schematic](images/timeseries2.png)\n\nOld data can be deleted cleanly:\n\n![schematic](images/timeseries3.png)\n\nIt is built on top of the the [cloudant-quickstart](https://www.npmjs.com/package/cloudant-quickstart) library which handles the aggregation API.\n\n## Pre-requisites\n\n- [Node.js \u0026 npm](https://nodejs.org/en/)\n- an Apache CouchDB or IBM Cloudant instance\n\n## Installation\n\nBuild `cloudant-timeseries` into your own Node.js project with:\n\n    npm install --save cloudant-timeseries\n\n## Creating a database\n\nGiven a URL which contains your Cloudant or CouchDB service's admin credentials\n\n    var url = \"https://user:pass@myaccount.cloudant.com\"\n\nYou can import the library into your code, passing the url and your database name:\n\n    var db = require('cloudant-timeseries')(url, 'analytics')\n\nor you can combine the two:\n\n    var url = \"https://user:pass@myaccount.cloudant.com/analytics\"\n    var db = require('cloudant-timeseries')(url)\n\nWe can then go ahead and do the one-off setup of our database:\n\n    db.setup.create().then( () =\u003e {\n        return db.setup.count('browser',['year','month','day']) \n      }).then( () =\u003e { \n        return db.setup.count('year')\n      }).then( () =\u003e {\n        return db.setup.stats('page_speed',['year','month','day']) \n      }).then( () =\u003e {\n        return db.setup.done() \n      }).catch(console.error);\n\nor in a slightly more compact form:\n\n    db.setup.create()\n      .then( () =\u003e db.setup.count('browser',['year','month','day']) )\n      .then( () =\u003e db.setup.count('year')\n      .then( () =\u003e db.setup.stats('page_speed',['year','month','day']) )\n      .then( () =\u003e db.setup.done() )\n      .catch(console.error);\n\nThis instructs *cloudant-timeseries* to\n\n- create the database\n- setup a design document to count instances of the 'browser' field by year, month and day\n- setup a design document to count documents by year\n- setup a design document to get stats on the 'page_speed' field\n- commits this change (the call to setup.done()), which actually does the writing to Cloudant\n\nThis takes a little while because 2 years of monthly databases are created with the correct design documents.\n\nThe \"setup a design document\" steps are optional. The shortest path to setup is:\n\n    db.setup.create()\n      .then( () =\u003e db.setup.done() )\n      .catch(console.error);\n\nwhich only creates the empty databases.\n\n## Just add data\n\nNow we're set up, we can start adding data, a document at a time:\n\n    var obj = { \n      event_type: 'click',\n      browser: 'chrome',\n      page_speed: 1.5\n    }\n    db.insert(obj).then(console.log)\n\nor in bulk:\n\n    var obj1 = { \n      event_type: 'click',\n      browser: 'chrome',\n      page_speed: 1.5\n    }\n    var obj2 = { \n      event_type: 'click',\n      browser: 'safari',\n      page_speed: 1.2\n    }\n    db.insert([obj1, obj2]).then(console.log)\n\nYou don't need to provide the `year`, `month`, `day`, `hour`, `minute`, `second` fields - they are added automatically. Data is added with a timestamp of *now* unless you supply a second parameter containing a custom timestamp:\n\n    ts = '2017-06-24 10:22:44'\n    db.insert([obj1, obj2], ts).then(console.log)\n\nThe *cloudant-timeseries* library uses the timestamp to decide which *under-the-hood* database the data will be stored in.\n\n## Querying the data set\n\nOnce we have data in a one or more monthly collection we can query the data \n\n    // get the counts of browsers grouped by year, month and day\n    db.count('browser',['year','month','day'])).then(console.log)\n\n    // get document counts by year\n    db.count('year').then(console.log)\n\n    // get stats on the page_speed, grouped by year, month and day\n    db.stats('page_speed',['year','month','day']).then(console.log)\n\nThese were the three queries we set up at the beginning. We can add new ones if we like:\n\n    // get stats on the page_speed, grouped by year\n    db.stats('page_speed','year').then(console.log)\n\nand the library will create any design documents required.\n\n## Deleting data\n\nOne of the main advantages of having monthly databases is that you can delete old data very easily. This library makes it easier still. Simply call `deleteOlderThan` passing in a timestamp and data older than that date (to the nearest month) will be removed:\n\n    // remove all data before 2017\n    db.deleteOlderThan('2017-01-01').then(console.log)\n\nNote: deletion is a permenant and irreversible operation. Be careful.\n\n## How does this all work?\n\nIf you ask to setup a database called \"mydb\", under the hood there will by a \"mydb\" database that holds the design documents you define at startup and a monthly database for 2 years in the future e.g. \"mydb_2018_01\", \"mydb_2018_02\" etc.\n\nWhen you query the database, *all* the databases are queried in turn and the results are aggregated for you. This is less efficient than the \"one database contains everything\" approach, but it does allow older unwanted data to be removed easily.\n\n## Running setup again\n\nIf you are running out of databases, simply run `db.setup.done()` again an fresh batch of two years of monthly databases will be created.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglynnbird%2Fcloudant-timeseries","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fglynnbird%2Fcloudant-timeseries","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglynnbird%2Fcloudant-timeseries/lists"}