{"id":45493659,"url":"https://github.com/tbal999/flowtocsv","last_synced_at":"2026-02-22T17:33:12.603Z","repository":{"id":57546544,"uuid":"300539792","full_name":"tbal999/flowtocsv","owner":"tbal999","description":"small ETL library to convert MRASCO \u0026 SPAA UK energy industry dataflows into csv","archived":false,"fork":false,"pushed_at":"2023-04-02T14:35:34.000Z","size":184,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-06-20T12:36:17.541Z","etag":null,"topics":["csv","dataflows","etl-library","parse"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tbal999.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-02T07:45:03.000Z","updated_at":"2021-11-02T20:36:41.000Z","dependencies_parsed_at":"2024-06-20T11:51:23.231Z","dependency_job_id":"e21a815d-e453-45d5-8ed1-f9dfc3f761ea","html_url":"https://github.com/tbal999/flowtocsv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tbal999/flowtocsv","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbal999%2Fflowtocsv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbal999%2Fflowtocsv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbal999%2Fflowtocsv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbal999%2Fflowtocsv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tbal999","download_url":"https://codeload.github.com/tbal999/flowtocsv/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbal999%2Fflowtocsv/sbom","scorecard":{"id":870030,"data":{"date":"2025-08-11","repo":{"name":"github.com/tbal999/flowtocsv","commit":"bf8144fbf784af9ff2f96b768e244f1eaaec87d1"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/go.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/tbal999/flowtocsv/go.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/go.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/tbal999/flowtocsv/go.yml/main?enable=pin","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/go.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-24T03:52:41.257Z","repository_id":57546544,"created_at":"2025-08-24T03:52:41.257Z","updated_at":"2025-08-24T03:52:41.257Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29720567,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-22T15:10:41.462Z","status":"ssl_error","status_checked_at":"2026-02-22T15:10:04.636Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","dataflows","etl-library","parse"],"created_at":"2026-02-22T17:33:07.887Z","updated_at":"2026-02-22T17:33:12.586Z","avatar_url":"https://github.com/tbal999.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# flowtocsv\n\nThis is an extremely lightweight MRASCO ETL library that can convert MRASCO flows into CSV files.\nThen you can load them into SQL to do automation / analysis. UNDER MIT LICENSE.... 2021\nfeel free to use it !\n\n\u003cb\u003eDOCUMENTATION:\u003c/b\u003e\nhttps://pkg.go.dev/github.com/tbal999/flowtocsv\n\n\u003cb\u003eFOR:\u003c/b\u003e Data analysts \u0026 BI Specialists in small energy industry retail firms who want to develop BI around energy industry data i.e gas and electricity data but do not have the capability/functionality to automate the parsing of MRASCO dataflows into their databases.\n\n\u003cb\u003eEXAMPLE USE CASE:\u003c/b\u003e You want to monitor read disputes actively, tracking when they are raised, and when they are rejected etc, but you don't want to hire an expensive contractor to build you a workflow. \n1) Ensure you have SFTP server (or API) up and running which gives you access to daily incoming D0300 dataflows\n2) Set up pipeline to download all D0300 dataflows daily to server to a flowcrunch_inputfiles folder\n3) Set up flowtocsv so that it can parse D0300 dataflows into CSV format\n4) Upload CSV file into SQL using staging/production tables \u0026 merge methodology\n5) Use Tableau / Power BI / Qlik to load table into view and join necessray tables (will require energy industry knowledge!)\n\nYou've got an up and running BI pipeline that keeps track of read disputes and whether they are successful/rejected/ignored etc etc.\n\nThe number of use cases for this tool is AT LEAST the same as the number of UK energy industry dataflows that exist for suppliers/retailers (128). Each of varying significance and impact.\n\nThis particular library can also handle other flat file types. My stance would be to try it out. If it works, great!\nThrough doing this I discovered new ways how to wrangle data using recursion in go. This library can be inserted into ETL packaged solutions.\n\nAt the moment there are a few workflows for this library.\n\n1) SFTP server -\u003e download data flows to a folder -\u003e parse data flows into csv -\u003e load into SQL.\n2) API -\u003e load clobs into Go as strings -\u003e parse clobs into csv -\u003e load into SQL.\n\n\u003cb\u003eExample code:\u003c/b\u003e\n```\npackage main\n\nimport (\n\tflow \"github.com/tbal999/flowtocsv\"\n)\n\nfunc main() {\n\tf := flow.Instructions{}\n\tf.Init()\n\tf.LearnFile(\"|\") //f.Learn(\",\") - for Gas SPAA Dataflows.\n\tf.StartFiles()\n}\n```\nThis will create four folders - flowcrunch_inputfiles, flowcrunch_instructions, flowcrunch_learn and flowcrunch_outputfiles.\nSimply save an example dataflow i.e a D0150001 in the 'learn' folder and it will generate the necessary instructions to parse that dataflow into a CSV format:\n```\na fake dataflow clob\nZHV|99999|D0150001|M|LLLL|X|AAAA|12345||||TR01|\n288|12345|20179568|D|x|\n289|0158|20179568|Z||\n290|XXXTEST011|||40|F|A2Z 5000|LLLL|||||||||||K|20179568|20179568|20200101|||H|20179568|\n293|L|C|AI|9.00||9||DE|\n293|N|C|AI|9.00||7|||\n08A|KXXA 12387|20179568|LLLL|\nZPT|99999|6||1|12345|\n```\nThen it will learn the structure of the dataflow and create a set of instructions in the 'instructions' folder:\n\n```\n{\n\t\"Dataflow\": \"150\",\n\t\"Delimiter\": \"|\",\n\t\"DataItems\": [\n\t\t\"288\",\n\t\t\"289\",\n\t\t\"290\",\n\t\t\"293\",\n\t\t\"08A\"\n\t],\n\t\"Spaces\": [\n\t\t4,\n\t\t4,\n\t\t25,\n\t\t8,\n\t\t3\n\t],\n\t\"Headers\": [\n\t\t\"150_COLUMN_1\",\n\t\t\"150_COLUMN_2\",\n\t\t\"150_COLUMN_3\",\n\t\t\"150_COLUMN_4\",\n\t\t...\n\t\t...\n\t],\n\t\"Outputname\": \"150_Converted\"\n}\n```\nThen you can place as many D0150 dataflows as you want in the 'inputfiles' folder and it will create an aggregated CSV file of all those dataflows.\nInstructions can sometimes add too many columns, so just remove some from the JSON if necessary.\n\nThis methodology works for nearly all UK energy industry data flows.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftbal999%2Fflowtocsv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftbal999%2Fflowtocsv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftbal999%2Fflowtocsv/lists"}