{"id":20616731,"url":"https://github.com/andyduke/data_processor_cli","last_synced_at":"2025-07-23T07:04:52.310Z","repository":{"id":150348940,"uuid":"458639398","full_name":"andyduke/data_processor_cli","owner":"andyduke","description":"Flexible command-line data processor.","archived":false,"fork":false,"pushed_at":"2022-03-22T20:07:17.000Z","size":3095,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-06T19:22:23.870Z","etag":null,"topics":["command-line","command-line-tool","converter","csv","data","data-structures","json","template","toml","xml","yaml"],"latest_commit_sha":null,"homepage":"","language":"Dart","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andyduke.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-12T21:05:47.000Z","updated_at":"2025-02-23T23:07:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa25c68a-f668-4342-90b5-a7cb406501f1","html_url":"https://github.com/andyduke/data_processor_cli","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/andyduke/data_processor_cli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyduke%2Fdata_processor_cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyduke%2Fdata_processor_cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyduke%2Fdata_processor_cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyduke%2Fdata_processor_cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andyduke","download_url":"https://codeload.github.com/andyduke/data_processor_cli/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyduke%2Fdata_processor_cli/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266633226,"owners_count":23959516,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line","command-line-tool","converter","csv","data","data-structures","json","template","toml","xml","yaml"],"created_at":"2024-11-16T11:20:06.046Z","updated_at":"2025-07-23T07:04:52.286Z","avatar_url":"https://github.com/andyduke.png","language":"Dart","readme":"# Data Processor\n\nThe Data Processor is a tool for converting structured data from one format to another, with the ability to select a portion of a piece of data to convert, or use a template to compose the result.\n\nSupported input and output formats: JSON, Yaml, XML, CSV, TOML.\n\nThe output can be generated using a template language similar to [django/liquid template language](https://docs.djangoproject.com/en/4.0/ref/templates/language/).\n\nIt is possible to select only a portion of the input data using the [JMESPath](https://jmespath.org/) query language.\n\n## Usage\n\n```\ndp \"\u003cquery\u003e\" [\u003cfilename\u003e] [options] \u003e output_file\n```\nor\n```\ntype \u003cfilename\u003e | dp \"\u003cquery\u003e\" [options] \u003e output_file\n```\n\nThe *query* is a JMESPath.\n\nYou can find out all the options using the **--help** parameter.\n\n### JMESPath\n\nIn the Data Processor, the JMESPath must be passed as the first parameter to extract a chunk from the input data.\nIf you want to use the entire input data (for example, to convert from one format to another), you must pass a dot instead of JMESPath.\n\n```shell\ndp \".\" users.json -o yaml \u003e users.yaml\n```\n\n### Input data\n\nThe Data Processor tries to determine the input data type from the input file name extension, but you can force the data type (using the **-i** option) if the file name has no extension or the extension is non-standard.\nYou can also pass input using a pipe, but you must specify the type of the input.\n\n## Getting a piece of data as JSON from XML\n\nFor example, you need to select certain users with the \"user\" role from the XML data and display their names in the JSON list format.\n\nThere is a `sample.xml` file with data:\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cusers\u003e\n  \u003cuser id=\"1\"\u003e\n    \u003cname\u003eJohn Doe\u003c/name\u003e\n    \u003crole\u003euser\u003c/role\u003e\n  \u003c/user\u003e\n  \u003cuser id=\"2\"\u003e\n    \u003cname\u003eRichard Roe\u003c/name\u003e\n    \u003crole\u003eadministrator\u003c/role\u003e\n  \u003c/user\u003e\n  \u003cuser id=\"3\"\u003e\n    \u003cname\u003eJane Doe\u003c/name\u003e\n    \u003crole\u003euser\u003c/role\u003e\n  \u003c/user\u003e\n\u003c/users\u003e\n```\n\n...command\n```\ndp \"users.user[?role=='user'].name\" sample.xml -o json\n```\n\n...will output the following:\n```json\n[\n  \"John Doe\",\n  \"Jane Doe\"\n]\n```\n\n## JSON to XML conversion with mapping\n\nFor example, you need to convert a JSON structure to XML, but transform some JSON properties (id) into XML attributes, and change the names of some properties (users to persons and user to person) - this can be done using a query with multiselect hash and a pipe expression.\n\nThere is a `sample.json` file with data:\n```json\n{\n  \"users\": {\n    \"user\": [\n      {\n        \"id\": 1,\n        \"name\": \"John Doe\",\n        \"role\": \"user\"\n      },\n      {\n        \"id\": 2,\n        \"name\": \"Richard Roe\",\n        \"role\": \"admin\"\n      }\n    ]\n  }\n}\n```\n\n...command\n```\ndp \"users.user[*].{\\\"@id\\\": id, name: name} | {persons: {person: @}}\" sample.json -o xml\n```\n\n...will output the following:\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cpersons\u003e\n  \u003cperson id=\"1\"\u003e\n    \u003cname\u003eJohn Doe\u003c/name\u003e\n  \u003c/person\u003e\n  \u003cperson id=\"2\"\u003e\n    \u003cname\u003eRichard Roe\u003c/name\u003e\n  \u003c/person\u003e\n\u003c/persons\u003e\n```\n\n## Creating a text file of any format based on structured data\n\nIt is possible to create files in formats not supported by the Data Processor using the template language.\n\nFor example, from the data of the `orders.yaml` file:\n```yaml\naccount:\n  name: Classic Cars\n  customerName: John Doe\n  orders:\n  - orderId: 1\n    productId: C6\n    productName: Corvette C6\n    price: 45849\n  - orderId: 2\n    productId: M4\n    productName: Morgan Plus 4\n    price: 69995\n```\n\n...using the template file `orders.template`:\n```\nAccount \"{{ account.name }}\"\nCustomer \"{{ account.customerName }}\"\n\nOrders\n======\n{% for order in account.orders -%}\nOrder ID #{{ order.orderId }}\nProduct #{{ order.productId }}, {{ order.productName }}\n  Price {{ order.price }}\n\n{% endfor -%}\n\nTotal {% by_path \"sum(account.orders[*].price)\" %}\n```\n\n...with the command:\n```\ndp \".\" orders.yaml -o template -t orders.template\n```\n\n...generate the following file: \n```\nAccount \"Classic Cars\"\nCustomer \"John Doe\"\n\nOrders\n======\nOrder ID #1\nProduct #C6, Corvette C6\n  Price 45849\n\nOrder ID #2\nProduct #M4, Morgan Plus 4\n  Price 69995\n\nTotal 115844.0\n```\n\nYou can learn more about the template language using the **--template-guide** option.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyduke%2Fdata_processor_cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandyduke%2Fdata_processor_cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyduke%2Fdata_processor_cli/lists"}