{"id":15513434,"url":"https://github.com/githubocto/flat-postprocessing","last_synced_at":"2025-04-10T16:50:36.619Z","repository":{"id":43334309,"uuid":"354141779","full_name":"githubocto/flat-postprocessing","owner":"githubocto","description":"A collection of postprocessing utilities for flat","archived":false,"fork":false,"pushed_at":"2022-09-05T02:54:56.000Z","size":1213,"stargazers_count":57,"open_issues_count":1,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T14:40:49.617Z","etag":null,"topics":["flat-data"],"latest_commit_sha":null,"homepage":"https://octo.github.com/projects/flat-data","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/githubocto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-02T21:55:03.000Z","updated_at":"2025-03-04T12:51:04.000Z","dependencies_parsed_at":"2022-09-03T04:50:59.776Z","dependency_job_id":null,"html_url":"https://github.com/githubocto/flat-postprocessing","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubocto%2Fflat-postprocessing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubocto%2Fflat-postprocessing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubocto%2Fflat-postprocessing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubocto%2Fflat-postprocessing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/githubocto","download_url":"https://codeload.github.com/githubocto/flat-postprocessing/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248255747,"owners_count":21073374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flat-data"],"created_at":"2024-10-02T09:54:18.124Z","updated_at":"2025-04-10T16:50:36.595Z","avatar_url":"https://github.com/githubocto.png","language":"TypeScript","readme":"# Flat Data: Postprocessing Library and Examples\n\nA collection of postprocessing [helper functions](https://deno.land/x/flat/mod.ts) and examples for [Flat Data](https://octo.github.com/projects/flat-data).\n\nThese examples and functions are written in [Deno](https://deno.land/), a new language created by the same founders of Node.js and meant to improve on many aspects of Node. \n\n\u003e Note: If you're noticing your scripts failing recently, try updating to `0.0.15`. [More info here](https://github.com/githubocto/flat/issues/67#issuecomment-1025911772)\n\n## Usage\n\nWhen writing a [Flat Data]() Action, you can specify a path to a postprocessing Deno script that can manipulate the data downloaded by Flat even further. \n\n```yaml\n- name: Fetch data \n        uses: githubocto/flat@v2\n        with:\n          http_url: http://api.coindesk.com/v2/bpi/currentprice.json # The endpoint to fetch\n          downloaded_filename: btc-price.json # The http_url gets saved and renamed in our repository as btc-price.json\n          postprocess: postprocess.ts # A postprocessing javascript or typescript file written in Deno\n```\n\nThis is an example of a postprocessing script. Notice the use of `Deno.args[0]` to pass in the path of the `downloaded_filename`.\n\n```ts\n// The Flat Data postprocessing libraries can be found at https://deno.land/x/flat/mod.ts\n// Replace 'x' with latest library version\nimport { readJSON, writeJSON } from 'https://deno.land/x/flat@0.0.x/mod.ts'\n\nconst filename = Deno.args[0] // equivalent to writing `const filename = 'btc-price.json'`\nconst data = await readJSON(filename)\n\n// pluck a specific key off and write it out to a new file\nconst newfile = `postprocessed_${filename}`\nawait writeJSON(newfile, data.path.to.something)\n```\n\n## Examples\n\nCan be found in the examples folder. Once you [install Deno](https://deno.land/) you can run these examples with:\n\n* `deno run -A examples/csv/csv-example.ts`\n* `deno run -A examples/csv/arquero-example.ts`\n* `deno run -A --unstable examples/image/image-example.ts`\n* `deno run -A examples/json/json-example.ts`\n* `deno run -A examples/sheets/sheets-example.ts`\n* `deno run -A examples/xlsx/xlsx-example.ts`\n* `deno run -A --unstable examples/zip/zip-example.ts`\n\nDeno can run javascript or typescript files, so you can easily convert any of these examples to javascript and run them in the same way:\n\n`deno run -A examples/csv/csv-example.js`\n\n## Using Python\n\nWhile our examples use a Deno file to run postprocessing tasks, you can also use Python as specified in this example: [https://github.com/pierrotsmnrd/flat_data_py_example](https://github.com/pierrotsmnrd/flat_data_py_example). Thank you [@pierrotsmnrd](https://github.com/pierrotsmnrd)!\n\n## Using bash\n\nYou can also use bash as specified in this example: [https://github.com/aborruso/flat_data_bash_example](https://github.com/aborruso/flat_data_bash_example). By [@aborruso](https://twitter.com/aborruso)\n\n## Postprocessing Library\n\nThe Flat Data postprocessing library can be found at: [https://deno.land/x/flat/mod.ts](https://deno.land/x/flat/mod.ts)\n\nYou can import and use these helper functions directly, or treat them as a starting point for writing your own postprocessing scripts.\n\n### CSV\n\n#### readCSV\n\n```ts\nreadCSV(path: string, options?: ParseOptions): Promise\u003cRecord\u003cstring, unknown\u003e[]\u003e\n```\n\nArgs:\n\n* **path:** path to a local CSV file\n* **options:** [options](https://deno.land/std@0.92.0/encoding#codeparseoptionscode) for parsing the CSV file\n\nUsage:\n\n`const csv = await readCSV('./path/to/file.csv')`\n\n#### writeCSV\n\n```ts\nwriteCSV(path: string, data: Record\u003cstring, unknown\u003e[] | string, options?: Deno.WriteFileOptions)\n```\n\nArgs:\n\n* **path**: path to a local CSV file\n* **data**: string or object array to store\n* **options**: [options](https://doc.deno.land/builtin/stable#Deno.WriteFileOptions) for writing the CSV file\n\nUsage:\n\n```ts\nconst data = [ \n\t{ age: 70, name: 'Rick' },\n\t{ age: 14, name: 'Smith' }\n]\nawait writeCSV('./path/to/file.csv', data)\n```\n\n### TXT\n\n#### readTXT\n\n```ts\nreadTXT(path: string): string\n```\n\nArgs:\n\n* **path**: path to a local TXT file\n\n\nUsage: \n\n```ts\nconst text = await readTXT('./path/to/file.txt')\n```\n\n#### writeTXT\n\n```ts\nwriteTXT(path: string, text: string, options?: Deno.WriteFileOptions): void\n```\n\nArgs:\n\n* **path**: path to a local TXT file\n* **text**: text to write to file\n* **options**: [options](https://doc.deno.land/builtin/stable#Deno.WriteFileOptions) for writing the TXT file\n\nUsage:\n\n```ts\nawait writeTXT('./path/to/file.txt', 'Content for the file')\n```\n\n### JSON\n\n\n#### readJSON\n\n```ts\nreadJSON(path: string): JSON\n```\n\nArgs:\n\n* **path**: path to a local JSON file\n\n\nUsage: \n\n```ts\nconst json = await readJSON('./path/to/file.json')\n```\n\n#### readJSONFromURL\n\n```ts\nreadJSONFromURL(url: string): JSON\n```\n\nArgs:\n\n* **url**: URL to a json file\n\n\nUsage: \n\n```ts\nconst json = await readJSON('www.url.com/file.json')\n```\n\n#### writeJSON\n\n```ts\nwriteJSON(path: string, data: any, replacer?: any, space?: string | number): void\n\n\n```\n\nArgs:\n\n* **path**: path to a local JSON file\n* **data**: data to store as JSON\n* **replacer**: [replacer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#parameters) function that transforms the results or an array of strings and numbers that acts as an approved list for selecting the object properties that will be stringified\n* **space**: adds indentation, white space, and line break characters to the to the JSON text to make it easier to read.\n\n\nUsage: \n\n```ts\nconst data = { age: 40 }\nawait writeJSON('./path/to/file.json', data)\n\nawait writeJSON('./path/to/file-with-indentation', data, null, 2)\n```\n\n### XLSX\n\nOur library relies on [SheetJS](https://github.com/SheetJS/sheetjs), a library for parsing various spreadsheet formats. In addition to a simple `readXLSX` function you can access the core `xlsx` module by importing it directly.\n\n```ts\nimport { xlsx, readXLSX } from 'https://deno.land/x/flat/mod.ts'\n```\n\n`xlsx` provides many more [utility functions](https://github.com/SheetJS/sheetjs). \n\n\n#### readXLSX\n\n```ts\nreadXLSX(path: string): XLSX.WorkBook\n```\n\nArgs:\n\n* **path**: path to a local XLSX file\n\n\nUsage: \n\n```ts\nconst workbook = await readXLSX('./path/to/file.xlsx')\nconst sheetData = workbook.Sheets[workbook.SheetNames[0]]\nconst csvString = await xlsx.utils.sheet_to_csv(sheetData)\n```\n\n\n### Image\n\nWe recommend using a library like [imagescript](https://deno.land/x/imagescript) for more advanced image manipulation. See an example [here](https://github.com/githubocto/flat-postprocessing/blob/main/examples/image/image-example.ts).\n\n#### readImageFromFile\n\n```ts\nreadImageFromFile(path: string): Promise\u003cUint8Array\u003e\n```\n\nArgs:\n\n* **path**: path to a local image file\n\n\nUsage: \n\n```ts\nconst bytes = await readImageFromFile('./path/to/image.jpeg')\n```\n\n\n#### readImageFromURL\n\n```ts\nreadImageFromURL(url: string): Promise\u003c{ bytes: Uint8Array; name: string; }\u003e\n```\n\nArgs:\n\n* **url**: url string to an image\n\nUsage: \n\n```ts\nconst image = await readImageFromURL('www.url.com/image.jpg')\nconst bytes = image.bytes\nconst name = image.name\n```\n\n#### writeImage\n\n```ts\nwriteImage(imageBytes: Uint8Array, path: string): void\n```\n\nArgs:\n\n* **imageBytes**: a byte array\n* **path**: path and name to write the image file\n\n\nUsage: \n\n```ts\nawait writeImage(bytes, './path/to/image.jpeg')\n```\n\n### Zip\n\n#### unZipFromFile\n\n```ts\nunZipFromFile(\n    filePath: string,\n    destinationPath: string | null = \"./\",\n    options: any = {},\n): Promise\u003cstring | false\u003e\n```\n\nArgs:\n\n* **filePath**: a path to a local zip file\n* **destinationPath**: a folder path to unzip the files\n* **options**: option.includeFileName can be true or false\n\n\nUsage: \n\n```ts\nconst result = await unZipFromFile('./path/to/folder.zip', './unzip/path')\nconst output = result ? 'File unzipped successfully' : 'Error unzipping'\n```\n\n#### unZipFromURL\n```ts\nunZipFromURL(\n    fileURL: string,\n    destinationPath: string | null = \"./\",\n    options: any = {},\n): Promise\u003cstring | false\u003e\n```\n\nArgs:\n\n* **filePath**: a path to a local zip file\n* **destinationPath**: a folder path to unzip the files\n* **options**: option.includeFileName can be true or false\n\nUsage: \n\n```ts\nconst result = await unZipFromURL('www.url.com/file.zip', './unzip/path')\nconst output = result ? 'File unzipped successfully' : 'Error unzipping'\n```\n\n\n### Remove\n\n#### removeFile\n\n```ts\nremoveFile(path: string): void\n```\n\nArgs:\n\n* **path**: path to a local file to delete\n\n\nUsage: \n\n```ts\nawait removeFile('/path/to/file.x')\n```\n\n## Testing\n\nRun all the tests:\n\n`deno test -A --unstable tests/*`\n\nRun separate tests\n\n`deno test -A --unstable tests/csv-test.ts`\n\n\n## License\n\n[MIT](LICENSE)\n\n","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithubocto%2Fflat-postprocessing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgithubocto%2Fflat-postprocessing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithubocto%2Fflat-postprocessing/lists"}