{"id":22931318,"url":"https://github.com/ntnn/dataparse","last_synced_at":"2025-04-01T17:27:11.153Z","repository":{"id":188347376,"uuid":"678072114","full_name":"ntnn/dataparse","owner":"ntnn","description":"Parsing, transforming and unmarshalling data.","archived":false,"fork":false,"pushed_at":"2024-07-23T09:03:23.000Z","size":289,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-07T11:22:12.685Z","etag":null,"topics":["data","data-parser","data-parsing","data-transformation","golang","golang-lib"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ntnn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-13T15:41:02.000Z","updated_at":"2024-07-23T09:03:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"6ebf5329-3dd8-400c-ae48-9e7dacb024d0","html_url":"https://github.com/ntnn/dataparse","commit_stats":null,"previous_names":["ntnn/dataparse"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ntnn%2Fdataparse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ntnn%2Fdataparse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ntnn%2Fdataparse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ntnn%2Fdataparse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ntnn","download_url":"https://codeload.github.com/ntnn/dataparse/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246679552,"owners_count":20816508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-parser","data-parsing","data-transformation","golang","golang-lib"],"created_at":"2024-12-14T10:38:40.041Z","updated_at":"2025-04-01T17:27:11.126Z","avatar_url":"https://github.com/ntnn.png","language":"Go","readme":"# dataparse\n\nToo often I have to work with these annyances:\n\n1. CSV files that require to rewrite parsing for varying types from\n   string\n1. Excel files that report erroneous types\n1. Unreliable APIs reporting e.g. integer as floats\n1. Unreliable, (almost) undocumented APIs that return an object or\n   a list depending on the number of results\n\netc.pp.\n\nTo solve these annoyances `dataparse` was born.\n\nA onestop shop that makes it easy to retrieve information from varying\nsources and handles the transformation between types.\n\n## General use\n\n### APIs\n\nIf an API does not offer an OpenAPI spec it is left to the consumer to\nimplement a client. Usually it is enough to have a look at the results\nwith `curl`, define structs with tags accordingly and then\n`json.Unmarshal` into those.\n\nSometimes these APIs (especially SGML-to-JSON-wrapped and to a lesser\nextend Java-backed APIs) report values in wild inconsistency, e.g.\nreporting integers as floats or numbers as strings.\n\nIn those cases `dataparse.Map` can help:\n\n```go\n// Execute the request to the API\nresp, err := http.Get(\"https://outdated-but-important.api/path/to/endpoint\")\nif err != nil {\n    return err\n}\n\n// Read the returned JSON data into a dataparse.Map\nm, err := dataparse.FromJsonSingle(resp.Body)\nif err != nil {\n    return err\n}\n\ni, err := m.Int(\"integer_value\")\nif err != nil {\n    return err\n}\n\nlog.Printf(\"integer value: %d\")\n```\n\nIn this case the API can return the integer as integer, string or float\nand dataparse will transform it into the desired integer.\n\n### Unmarshalling into structs\n\nAnother useful utility is unmarshalling data into structs, e.g. when\nreading CSVs:\n\nAssuming a CSV file with the headers `hostname,ip,logsize`:\n\n```go\ntype myData struct {\n    Hostname string  `dataparse:\"hostname\"`\n    IPAddress net.IP `dataparse:\"ip\"`\n    Logsize int      `dataparse:\"logsize\"`\n}\n\n// If the CSV file has no headers they can also be passed like this:\n// dataparse.From(\"...\", dataparse.WithHeaders(\"hostname\", \"ip\", \"logsize\"))\nmapCh, errCh, err := dataparse.From(\"/path/to/data.csv\")\nif err != nil {\n    return err\n}\n\nfor mapCh != nil || errCh != nil {\n    select {\n    case m, ok := \u003c- mapCh:\n        if !ok {\n            mapCh = nil\n            continue\n        }\n        // Read the CSV data into a struct to utilize the discrete types.\n        d := myData{}\n        if err := m.To(\u0026d); err != nil {\n            log.Errorf(\"error reading data: %v | %#v\", err, m)\n            continue\n        }\n        // handle d further\n    case err, ok := \u003c- errCh:\n        if !ok {\n            errCh = nil\n            continue\n        }\n        log.Errorf(\"error from dataparse: %v\", err)\n    }\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fntnn%2Fdataparse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fntnn%2Fdataparse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fntnn%2Fdataparse/lists"}