{"id":18457224,"url":"https://github.com/accurat/data-juggler","last_synced_at":"2025-04-08T05:33:28.437Z","repository":{"id":34724207,"uuid":"175245662","full_name":"accurat/data-juggler","owner":"accurat","description":"A data wrapper and enricher 🤹‍♂️","archived":false,"fork":false,"pushed_at":"2022-12-03T01:48:37.000Z","size":13203,"stargazers_count":7,"open_issues_count":20,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-23T06:41:31.799Z","etag":null,"topics":["data-structures","mobx-state-tree"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/accurat.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-12T15:44:12.000Z","updated_at":"2023-11-11T17:22:19.000Z","dependencies_parsed_at":"2023-01-15T09:00:37.533Z","dependency_job_id":null,"html_url":"https://github.com/accurat/data-juggler","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/accurat%2Fdata-juggler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/accurat%2Fdata-juggler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/accurat%2Fdata-juggler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/accurat%2Fdata-juggler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/accurat","download_url":"https://codeload.github.com/accurat/data-juggler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247785864,"owners_count":20995641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-structures","mobx-state-tree"],"created_at":"2024-11-06T08:13:46.702Z","updated_at":"2025-04-08T05:33:23.805Z","avatar_url":"https://github.com/accurat.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# data-juggler 🤹‍♀️\n\n_all life is fermentation - Feynman_\n\nThis library serves little purpose, like all of us and everything we do, but is a bit covered and tested, as you can see.\n\nThis libary was `__init__.py`iated with [typescript starter](https://github.com/bitjson/typescript-starter) and uses the devil 👹 tslint.\n\nIn the context, the idea is to generalize and abstract the data propagation within an application, that we magically create in all of our projects.\nThis library abstracts that, normalizes, counts, calculates and does many inefficient things that feed into our laziness.\n\n## Usage\n\n### Installations\n\n```bash\nyarn add data-juggler\n```\n\n### Basic usage\n\nThe usage can be easily seen from the test, assume you are fetching some (csv like for now) data from an API and the columns have the following \"type\":\n\n```javascript\ndata = [\n  { height: 190, gender: 'male', timeOfMeasure: 1552397833139 },\n  { height: 170, gender: 'female', age: 22, timeOfMeasure: 1552397832139 },\n  { height: 164, gender: 'female', age: 20, timeOfMeasure: 15523912333139 },\n  { height: 176, gender: 'female', age: 12 }\n];\n\ntypes = {\n  height: 'continuous',\n  gender: 'categorical',\n  age: 'continuous',\n  timeOfMeasure: 'date'\n};\n\nconst { data, moments } = dataJuggler(data, { types });\n```\n\nLaunch the `dataJuggler` function with the sample data and instance types, and enjoy a beutiful dataset full of getters and stuff with everything that you need in it (this is, at least for now, a lie).\n\n### Properties\n\nYou'll get your data back (don't worry) with added properties!\n\n```javascript\n\nconst instance = data[0]\n\nt.deepEqual(instance, {\n  height: {\n    raw: 190,\n    scaled: 1\n  },\n  gender: {\n    raw: 'male',\n  },\n  timeOfMeasure: {\n    dateTime: // the day js instance of the dataset,\n    isValid: true,\n    iso: '2019-03-12',\n    raw: 1552397832139,\n    scaled: 1\n  }\n})\n// true\n\n```\n\nOn top of that you also get some getters for each variable that return the whole column, this could be cut eventually in a censorship attempt by my boss.\nEdit: the censorship did happen, this is not there anymore.\n\n## The rescaling functions\n\nIt could be annoying having to rescale all your functions because some time having them fitted from min-max to 0-1 is not what you need! We got your back. Another entry in the magical return object of the juggler is `scalers`! Imaging the classical situation below:\n\n```javascript\nconst d = [\n  {x: 0},\n  {x: 1},\n  {x: 2}\n]\n```\n\nYou can simply apply `dataJuggler`:\n\n```javascript\nconst { data, scalers } = dataJuggler(d, { types: infer})\n```\n\nNow if you need to rescale the data you can simply do this:\n\n```javascript\nconst newExtent = [0, 4]\nconst rescaleV = scalers.x(...newExtent)\nconst rescaledX = data.map(datum =\u003e rescaleX(datum.v.scaled)) // [0, 0.25, 0.5]\n```\n\n## The configuration object\n\nThe library accepts as a second parameter a config object of the form\n\n```javascript\nconst config = {\n  types,\n  formatter: [{ 'height': [/*stuff*/]}],\n  parser: { 'height': (cm: number) =\u003e /*more stuff*/ }\n}\n\nconst { data, moments, types } = dataJuggler(data, {...config});\n\n```\n\nAs in the example above the `types` object is rather explanatory! Let's look at the other ones, but first...\n\n### Annoyed by having to pass the types yourself? Look no further!\n\nIf you feel adventurous you can only pass the data without any `types` in the `config` object and our advanced (read naive) detecting system will try and determine, mostly leveraging the awesome [dayjs](https://github.com/iamkun/dayjs) ⏰ library, the type for you and then pass it as key `types` in the object returned by the function.\n\n### Custom formatter\n\nYou can also pass a custom formatter for each column type as follow.\n\n```javascript\n\nconst formatter = {\n  height: [\n    {\n      property: 'feet',\n      compute: (datum) =\u003e datum * 0.0328084\n    },\n    {\n      property: 'rescaled',\n      compute: (datum, min, max) =\u003e datum / max\n    }\n  ],\n  timeOfMeasure: [\n    {\n      property: 'year',\n      compute: (day) =\u003e day.format('YYYY')\n    }\n  ]\n}\n\n//  if we look for the same instance as before\n\nconst dataStore = dataJuggler(data, { types, formatter });\n\nt.deepEqual(istance, {\n  height: {\n    raw: 190,\n    scaled: 1,\n    // newly added\n    rescale: 1,\n    feet: 6.233596,\n  },\n  timeOfMeasure: {\n    dateTime: // the day js instance of the dataset,\n    isValid: true,\n    raw: 1552397832139,\n    scaled: 1,\n    // newly added\n    year: '2019',\n  }\n})\n\n// true\n\n```\n\n### Custom parser\n\nWhat if the data you are passing is not parsed correctly by the library. Once again, no worries! Just pass the parser yourself, the typechecker should prevent you from breaking everything, but hey, what do I know?\n\n```javascript\n  const { data: correctlyParsedData } = dataJuggler(datasetWithDates, {\n    parser: {\n      d: (day: string) =\u003e dayjs(day, 'YYYY-MM-DD').unix()\n    }\n  })\n  const { data: wronglyParsedData } = dataJuggler(datasetWithDates, {\n    parser: {\n      d: (day: string) =\u003e dayjs(day, 'MM-DD-YYYY').unix()\n    }\n  })\n```\n\n## Using other libary functions\n\nYou can of course use other functions from within the library. For example, let's say you want to use the autoinference function and do some operations on it you can simply import it and use it\n\n```javascript\nimport { autoInferenceType } from 'data-juggler/build/main/lib/utils/dataInference';\n\nconst inferredType = autoInferenceType(yourData, yourTypeObject || {}, yourParser || {})\n\nconst { data, momemnts } = dataJuggler(yourData, { types: inferredType })\n```\n\nThe types you passed in the config will not be overwritten by the library! The same can be done with other functions like `computeMoments` and `parseDates`. Not sure why you would do that but knock yourself out.\n\n## A more structured example, without having the data structure\n\nSo far we have been talking about cases where you know your data at \"compile time\" and neither does Typescript. This hinders the ability of the library to help you structure stuff correctly but all the core functionalities are intact and shining bright. Let's assume you need to fetch some data, of which you know the structure but you don't \"have\" it, you need not to worry! The function `dataJuggler` accepts one generic which is the type of the datum so you can do something like this:\n\n```javascript\ninterface ExpectedDatum {\n  height: number,\n  name: string\n}\n\nconst raw = ky.get('this.is.a.fancy.end.point.with.many.points.com/get').json()\nconst juggled = dataJuggler\u003cExpectedDatum\u003e(raw)\n\n```\n\n## The Mobx-state-tree in the room\n\nIf, for example, you use [Mobx state tree](https://github.com/mobxjs/mobx-state-tree) you could do the following:\n\n```javascript\nimport {} from ''\n\nexport const YourFancyMSTModel = t\n  .model('YourFancyMSTModel', {\n    juggleType: t.frozen\u003cInferObject\u003cYourDataStructure || Record\u003cstring, unknown\u003e(),\n    juggledData: t.frozen\u003cJuggledData\u003cYourDataStructure || Record\u003cstring, unknown\u003e(),\n    moments: t.frozen\u003cMomentsObject\u003cYourDataStructure || Record\u003cstring, unknown\u003e(),\n  })\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccurat%2Fdata-juggler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faccurat%2Fdata-juggler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccurat%2Fdata-juggler/lists"}