{"id":16992380,"url":"https://github.com/singerla/automizer-data","last_synced_at":"2025-04-12T04:20:30.804Z","repository":{"id":50487530,"uuid":"383052827","full_name":"singerla/automizer-data","owner":"singerla","description":"Stores and browses aggregated data using Prisma database tools","archived":false,"fork":false,"pushed_at":"2025-03-05T13:49:53.000Z","size":648,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T23:51:19.293Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/singerla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-05T07:29:49.000Z","updated_at":"2025-03-05T13:49:56.000Z","dependencies_parsed_at":"2023-10-02T20:43:48.964Z","dependency_job_id":"4d717b08-a7f9-4974-9e16-6c997fa47db4","html_url":"https://github.com/singerla/automizer-data","commit_stats":{"total_commits":179,"total_committers":5,"mean_commits":35.8,"dds":"0.16759776536312854","last_synced_commit":"0887defd05fcc5b953aaa0de280be3a6c3adaef1"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singerla%2Fautomizer-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singerla%2Fautomizer-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singerla%2Fautomizer-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singerla%2Fautomizer-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/singerla","download_url":"https://codeload.github.com/singerla/automizer-data/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514388,"owners_count":21116951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T03:29:00.604Z","updated_at":"2025-04-12T04:20:30.766Z","avatar_url":"https://github.com/singerla.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parse, tag and query xlsx-data\nThis package is used to parse structured tabular data from CSV or XLSX.\nStored data can be browsed and transformed into a desired two-dimensional result table.\n\nIts primary purpose is to deliver data for [pptx-automizer](https://github.com/singerla/pptx-automizer).\nThis project is *Work in progress*.\nNevertheless, you might already use `automizer-data` to handle table files coming from statistical analytics software.\nThe [example-xlsx](https://github.com/singerla/automizer-data/blob/main/__test__/data/test-data.xlsx) in `__test__/data`-folder is based on [GESStabs](https://gessgroup.de/software/gesstabs/).\n\nStorage and querying is done with [Prisma](https://github.com/prisma/prisma) ORM tools.\n\n## Installation\n### As a package\nIf you are working on an existing project, you can add *automizer-data* to it using npm or yarn. Run\n```\n$ yarn add automizer-data\n```\nor\n```\n$ npm install automizer-data\n```\nin the root folder of your project. This will download and install the most recent version into your existing project.\n\n### As a cloned repository\nIf you want to see how it works and you like to run own tests, you should clone this repository and install the dependencies:\n```\n$ git clone git@github.com:singerla/automizer-data.git my-project\n$ cd my-project\n$ yarn install\n$ yarn prisma generate\n```\n\n## Prisma studio\nYou can open prisma studio and take a look at the data:\n```\n$ yarn prisma studio\n```\nA lot of good stuff can be found at [prisma.io](https://www.prisma.io/).\n\n# Import Data\nAccording to parser's configuration, parsed data will sliced, tagged and separated into two-dimensional tables.\n\nThe Database contains:\n- __Categories:__ Generic nouns to describe the basic structure of your project \n- __Tags:__ Values of a certain category\n- __Sheets:__ Two-dimensional tables and their additional info\n\nEach __Sheet__ will contain:\n- a collection of rows\n- a collection of columns\n- the two-dimensional table body\n- a collection of tags\n- a collection of metadata that came along with the sheet\n\n## Example usage\n```ts\nimport { PrismaClient } from '@prisma/client'\nimport { Parser, Store } from '../src/index';\nimport { ParserOptions, Tagger, RawResultInfo } from \"../src/types\";\n\nconst store = new Store(\n  new PrismaClient()\n)\n\nconst config = \u003cParserOptions\u003e {\n  // This string separates tables if found in Column A\n  separator: 'Table Separator',\n  // A row that fits to any of the strings below will be\n  // separated into \"meta\"-field if found in Col A\n  metaMap: {\n    base: ['BASE'],\n    topBox: ['Top-2-Box (1-2)'],\n    bottomBox: ['Bottom-2-Box (4-5)'],\n    mean: ['Mean Value']\n  },\n  // Rows that equal to one of the labels below will be skipped.\n  skipRows: [\n    'company',\n    '* Annotation',\n    '(Sum of answers)'\n  ],\n  // A callback function to be applied to every body row\n  renderRow: (row: string[]): (number|null)[] =\u003e {\n    return row.map(cell =\u003e {\n      if(cell === ' ') return null\n      else return Number(cell)\n    })\n  },\n  // The info array of each sub-table will be passed to this\n  // callback. Tagging can be fine tuned here.\n  renderTags: (info: RawResultInfo[], tags: Tagger): void =\u003e {\n    info.forEach((info, level) =\u003e {\n      let cat\n      // info.key contains the info string's original section\n      // could be 'body' or 'info'\n      if(info.key === 'body') {\n        cat = 'vartitle'\n      } else if(info.value.indexOf('- ') === 0) {\n        cat = 'measure'\n      } else if(level === 0) {\n        // We strip the table separator and pass CountryName\n        info.value = info.value.replace('Table Separator – ', '')\n        cat = 'country'\n      } else if(level === 1) {\n        cat = 'variable'\n      } else if(level === 2) {\n        cat = 'questionText'\n      }\n      if(cat) {\n        tags.push(cat, info.value)\n      }\n    })\n  },\n}\n\nconst parse = new Gesstabs(config)\nconst file = `${__dirname}/data/test-data.xlsx`\nconst datasheets = await parse.fromXlsx(filename)\nconst summary = await store.run(datasheets)\n```\n\n# Intermediate JSON\nXlsx-Parser will tranform tabular data into an intermediate JSON object. The closer your input data comes to this format, the easier it will be to implement a new parser type. \n```json\n{\n  \"tags\": [\n    {\n      \"category\": \"country\",\n      \"value\": \"Norway\"\n    },\n    {\n      \"category\": \"variable\",\n      \"value\": \"Q12\"\n    },\n    {\n      \"category\": \"category\",\n      \"value\": \"Bar soap\"\n    },\n    {\n      \"category\": \"subgroup\",\n      \"value\": \"Age\"\n    },\n    {\n      \"category\": \"measure\",\n      \"value\": \"nominal\"\n    }\n  ],\n  \"columns\": [\"Total\", \"19-29\", \"30-39\", \"40-69\"],\n  \"rows\": [\"answer 1\", \"answer 2\", \"answer 3\"],\n  \"data\": [\n    [29,18,36,12],\n    [39,19,24,11],\n    [19,28,46,10]\n  ],\n  \"meta\": {\n    \"significance\": [\n      [null,null,\"h\",null],\n      [null,\"h\",\"l\",null],\n      [null,null,\"h\",\"l\"]\n    ]\n  }\n}\n```\n\n# Query Data\nAs all the Sheets are tagged, our queries will use tags to find the desired datasets.\n\n```ts\nimport { getData, Store } from '../src';\nimport { all } from '../src/filter';\nimport { value } from '../src/cell';\n\n// A selector is an array of tags.\nconst selector = [\n    {\n      category: 'country',\n      value: 'Norway'\n    },\n    {\n      category: \"variable\",\n      value: \"Q12\"\n    }\n]\n\n// The grid will define rows, cols and a callback\n// to run inside a target cell.\nconst grid = {\n    rows: all('row'),\n    columns: all('column'),\n    cell: value\n}\n\nconst result = await getData(selector, grid)\n\n// automizer-data will convert the result directly into\n// a pptx-automizer-object. \nconst chartData = result.toSeriesCategories()\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsingerla%2Fautomizer-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsingerla%2Fautomizer-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsingerla%2Fautomizer-data/lists"}