{"id":20250338,"url":"https://github.com/nilsmagnus/grib","last_synced_at":"2025-07-08T05:42:15.975Z","repository":{"id":57499583,"uuid":"86376486","full_name":"nilsmagnus/grib","owner":"nilsmagnus","description":"Golang GRIB2 parser","archived":false,"fork":false,"pushed_at":"2024-06-21T15:00:10.000Z","size":70235,"stargazers_count":57,"open_issues_count":6,"forks_count":18,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-12T15:07:41.414Z","etag":null,"topics":["forecasts","go","golang","grib","grib2","library","meteorological-data","meteorology","weather"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nilsmagnus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-03-27T19:37:44.000Z","updated_at":"2024-11-05T07:00:29.000Z","dependencies_parsed_at":"2022-08-28T15:22:30.785Z","dependency_job_id":null,"html_url":"https://github.com/nilsmagnus/grib","commit_stats":{"total_commits":123,"total_committers":8,"mean_commits":15.375,"dds":"0.12195121951219512","last_synced_commit":"0658b67d579bf64c8dea767aa9b018db9d3d04ab"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilsmagnus%2Fgrib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilsmagnus%2Fgrib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilsmagnus%2Fgrib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilsmagnus%2Fgrib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nilsmagnus","download_url":"https://codeload.github.com/nilsmagnus/grib/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248586236,"owners_count":21128997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["forecasts","go","golang","grib","grib2","library","meteorological-data","meteorology","weather"],"created_at":"2024-11-14T09:57:58.708Z","updated_at":"2025-04-12T15:07:50.106Z","avatar_url":"https://github.com/nilsmagnus.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Build](https://github.com/nilsmagnus/grib/actions/workflows/go.yml/badge.svg)\n\nGRIB2 Golang parser application and library\n================================\n\nParser and library for the grib2 file format for meteorological data representations. \n\nFor a full presentation of the library, see https://github.com/nilsmagnus/what-is-griblib\n\n## Usage\n\nInstall by typing\n\n    go get -u github.com/nilsmagnus/grib\n\n## Library Usage:\n\nHave a look at 'main.go' for main usage:\n\n    gribfile, err := os.Open(\"somegrib2file.grib2\")\n\tif err != nil {\n\t\tlog.Fatalf(\"Could not open test-file %v\", err)\n\t}\n    messages, err := griblib.ReadMessages(gribfile)\n    \n    for _, message := range messages {\n    \t// do your thing\n    }\n\nRead first n messages in a grib-file:\n\n    gribfile, err := os.Open(\"somegrib2file.grib2\")\n    if err != nil { log.Fatalf(\"Could not open test-file %v\", err) }\n    messages, err := griblib.ReadNMessages(gribfile, 2)\n\n    for _, message := range messages {\n        // do your thing with the n first messages\n    }\n\n### Application Usage:\n\n    $ grib -h \n    \n    Usage of grib:\n     -category int\n       \tFilters on Category within discipline. -1 means all categories (default -1)\n     -dataExport\n       \tExport data values. (default true)\n     -discipline int\n       \tFilters on Discipline. -1 means all disciplines (default -1)\n     -export int\n       \tExport format. Valid types are 0 (none) 1(print discipline names) 2(print categories) 3(json) \n     -file string\n       \tGrib filepath\n     -latMax int\n       \tMaximum latitude multiplied with 100000. (default 36000000)\n     -latMin int\n       \tMinimum latitude multiplied with 100000.\n     -longMax int\n       \tMaximum longitude multiplied with 100000. (default 9000000)\n     -longMin int\n       \tMinimum longitude multiplied with 100000. (default -9000000)\n     -maxmsg int\n       \tMaximum number of messages to parse. Does not work in combination with filters. (default 2147483647)\n     -operation string\n       \tOperation. Valid values: 'parse', 'reduce'. (default \"parse\")\n     -reducefile string\n       \tDestination for reduced file. (default \"reduced.grib2\")\n\n#### Examples:\n\nReduce input file to default output-file with discipline 0 (Meteorology):\n\n    grib -operation reduce -file testdata/reduced.grib2 -discipline 0\n\n\nFilter on area on size of norway+sweden, output to json:\n      \n    grib -file testdata/gfs.t00z.pgrb2.2p50.f003  -latMin 57000000 -latMax 71000000 -longMin 4400000 -longMax 32000000 -export 3\n\nFilter on temperature only:\n\n    grib -file testdata/gfs.t00z.pgrb2.2p50.f003 -discipline 0 -category 0 \n\n\n\t\n\n## What works?\n\n- basic binary parsing of GRIB2 GFS files from NOAA\n- implemented only \"Grid point data - complex packing and spatial differencing\"\n- Parsing Data3 type\n- Parsing Data0 type (Thanks to Cyrille Meichel)\n\n## Development\n\n### Dependencies\n\ngrib itself has _no dependencies_ and wants to stay that way to keep it simple. Therefore, there is no dep/glide and no dependency-configuration-files.\n\n### Build\n\nGrib uses make to build. You probably need go-lint installed in order to build.\n\nTo build, simply type\n\n    $ make\n    \nto test:\n\n    $ make test\n\n\n## TODOs\n\n- Support different types of grids, not only grid0\n- Support different types of products, not only product0\n- Tests for reduction\n- Tests for reading all sections\n\n\n# Contributors\n\n - \"analogic\", https://github.com/analogic\n - Cyrille Meichel, https://github.com/landru29\n - Nils Larsgård, https://github.com/nilsmagnus\n - measuredweighed, https://github.com/measuredweighed\n \n## Help appreciated\n\nFeel free to fork and submit pull requests or simply create issues for improvements :)\n\n### Performance\n\n\nto run a performance-test go to griblib/gribtest and run `make benchmark`. \n\n#### CPU\nSample output:\n\n```\ngo version go1.11.2 linux/amd64\ngo test -bench=. -benchmem -memprofile memprofile.out -cpuprofile profile.out\ncategory number 0,parameter number 0,surface type 1, surface value 0 max: 329.500000 min: 197.900000\ngoos: linux\ngoarch: amd64\npkg: github.com/nilsmagnus/grib/griblib/gribtest\nBenchmarkReadMessages-4   \t     100\t  11681028 ns/op\t 6587281 B/op\t    9344 allocs/op\nPASS\nok  \tgithub.com/nilsmagnus/grib/griblib/gribtest\t5.456s\ngo tool pprof -top profile.out\nFile: gribtest.test\nType: cpu\nTime: Feb 9, 2019 at 6:44pm (CET)\nDuration: 5.41s, Total samples = 5.34s (98.77%)\nShowing nodes accounting for 5.12s, 95.88% of 5.34s total\nDropped 69 nodes (cum \u003c= 0.03s)\n      flat  flat%   sum%        cum   cum%\n     1.58s 29.59% 29.59%      1.78s 33.33%  github.com/nilsmagnus/grib/griblib.(*BitReader).readBit\n     1.07s 20.04% 49.63%      2.85s 53.37%  github.com/nilsmagnus/grib/griblib.(*BitReader).readUint\n     0.53s  9.93% 59.55%      0.53s  9.93%  runtime.memclrNoHeapPointers\n     0.36s  6.74% 66.29%      0.85s 15.92%  encoding/binary.(*decoder).value\n     0.15s  2.81% 69.10%      2.78s 52.06%  github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock\n     0.15s  2.81% 71.91%      0.15s  2.81%  github.com/nilsmagnus/grib/griblib.(*Data3).applySpacialDifferencing\n     0.14s  2.62% 74.53%      0.26s  4.87%  reflect.Value.Index\n     0.13s  2.43% 76.97%      0.20s  3.75%  bytes.(*Buffer).ReadByte\n     0.11s  2.06% 79.03%      0.28s  5.24%  github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues\n     0.11s  2.06% 81.09%      0.19s  3.56%  reflect.Value.SetUint\n     0.08s  1.50% 82.58%      0.08s  1.50%  github.com/nilsmagnus/grib/griblib.Data0.scaleFunc.func1\n     0.08s  1.50% 84.08%      0.08s  1.50%  reflect.(*rtype).Kind (inline)\n     0.07s  1.31% 85.39%      0.07s  1.31%  bytes.(*Buffer).empty (inline)\n     0.06s  1.12% 86.52%      0.68s 12.73%  runtime.mallocgc\n     0.06s  1.12% 87.64%      0.06s  1.12%  runtime.memmove\n     0.05s  0.94% 88.58%      0.05s  0.94%  reflect.flag.mustBeAssignable\n     0.05s  0.94% 89.51%      0.05s  0.94%  runtime.nextFreeFast (inline)\n     0.04s  0.75% 90.26%      0.04s  0.75%  encoding/binary.(*decoder).uint8 (inline)\n   \n ```\n\n* As you can see from the sample output, it is the readBit and readUint that takes most of the time. If anyone know how to optimize these functions further, please let me know :)\n* Performance can be better if you only want to read certain kinds of messages. The filter that is implemented will filter only after reading all the messages. (help appreciated :) )\n\n#### Memory\n\n\n```\ngo tool pprof -top memprofile.out\nFile: gribtest.test\nType: alloc_space\nTime: Feb 12, 2019 at 8:19pm (CET)\nShowing nodes accounting for 3.39GB, 99.31% of 3.41GB total\nDropped 43 nodes (cum \u003c= 0.02GB)\n      flat  flat%   sum%        cum   cum%\n    1.52GB 44.71% 44.71%     2.05GB 60.09%  github.com/nilsmagnus/grib/griblib.(*Data2).extractData\n    0.76GB 22.19% 66.90%     0.76GB 22.19%  github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues\n    0.48GB 14.09% 80.99%     0.48GB 14.09%  github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock\n    0.26GB  7.68% 88.67%     0.34GB 10.01%  github.com/nilsmagnus/grib/griblib.(*Data2).extractBitGroupParameters\n    0.08GB  2.38% 91.05%     0.08GB  2.38%  github.com/nilsmagnus/grib/griblib.(*BitReader).readUintsBlock\n    0.06GB  1.90% 92.95%     3.34GB 97.93%  github.com/nilsmagnus/grib/griblib.readMessage\n    0.06GB  1.76% 94.70%     0.06GB  1.77%  encoding/binary.Read\n    0.06GB  1.70% 96.40%     3.40GB 99.64%  github.com/nilsmagnus/grib/griblib.ReadMessage\n    0.06GB  1.62% 98.02%     0.06GB  1.62%  github.com/nilsmagnus/grib/griblib.makeBitReader\n    0.04GB  1.29% 99.31%     0.04GB  1.29%  github.com/nilsmagnus/grib/griblib.(*bitGroupParameter).zeroGroup (inline)\n```\n\n* extractData is the most memory-hungry function. It could probably be more efficient, but is now optimized 80%.  \n* scaleValues could possibly be optimized, but is much better now than previous versions.\n\n\n# Grib Documentation\n\nGrib specification:\n\nhttps://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/\n\nDocumentation from noaa.gov :\n\nhttp://www.nco.ncep.noaa.gov/pmb/docs/on388/\n\n\nDaily grib2 files from NOAA can be found at\n\nhttp://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnilsmagnus%2Fgrib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnilsmagnus%2Fgrib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnilsmagnus%2Fgrib/lists"}