{"id":20294180,"url":"https://github.com/grab/gosm","last_synced_at":"2025-04-11T11:43:08.067Z","repository":{"id":45708001,"uuid":"513728869","full_name":"grab/gosm","owner":"grab","description":"Gosm is a golang library which implements writing OSM pbf files. ","archived":false,"fork":false,"pushed_at":"2023-05-24T13:47:44.000Z","size":61,"stargazers_count":11,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-25T08:03:30.725Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-14T02:06:01.000Z","updated_at":"2023-11-11T06:07:30.000Z","dependencies_parsed_at":"2024-11-14T15:35:35.610Z","dependency_job_id":"a36c9bf9-60b2-40bc-ae33-19cdaafcbe71","html_url":"https://github.com/grab/gosm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fgosm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fgosm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fgosm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grab%2Fgosm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grab","download_url":"https://codeload.github.com/grab/gosm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248386155,"owners_count":21095021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T15:28:08.960Z","updated_at":"2025-04-11T11:43:08.039Z","avatar_url":"https://github.com/grab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gosm\nGosm is a golang library which implements writing [OSM pbf](https://wiki.openstreetmap.org/wiki/PBF_Format) files. The initial idea is that when we use osmosis tool to dump data from database to pbf file, it causes high load of the database during that time. We need to maintain a large size of database just for the dump. For osmosis, we do not have control on how to query the database, so we implemented this library in golang. With this library, we can do the database query by ourselves during the data dump phase and we managed to reduce half size of the original database to finish the data dump.\n\nSo if you are looking for a full control during the data dump in your golang code, gosm is your choice. \nThis is running in our production usage and under a stable version right now.\n\n# Installation\n`go get github/grab/gosm`\n# Usage\n## Quick start\n```\n\t// initialize an encoder\n\twc := \u0026myWriter{}\n\tencoder := NewEncoder(\u0026NewEncoderRequiredInput{\n\t\tRequiredFeatures: []string{\"OsmSchema-V0.6\"},\n\t\tWriter:           wc,\n\t},\n\t\tWithWritingProgram(\"example\"),\n\t\tWithZlipEnabled(false),\n\t)\n\terrChan, err := encoder.Start()\n\n\t// write some nodes\n\tnodes := []*Node{\n\t\t{\n\t\t\tID: 7278995748,\n\t\t\tLatitude:  -7.2380901,\n\t\t\tLongitude: 112.6773289,\n\t\t\tTags: map[string]string{\n\t\t\t\t\"node\": \"node1\",\n\t\t\t\t\"erp\":  \"no\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: 6978510772,\n\t\t\tLatitude:  -7.2381273,\n\t\t\tLongitude: 112.6775354,\n\t\t},\n\t}\n\tencoder.AppendNodes(nodes)\n\tencoder.Close()\n```\n\n\n\n## End to end demo\nThis is not a fully runnable code, would like to demo how to use the library.\n\nYou can replace oriNodes, oriWays, oriRelations with your own data.\n```\n\tencoder := gosm.NewEncoder(\u0026gosm.NewEncoderRequiredInput{\n\t\tRequiredFeatures: []string{\"OsmSchema-V0.6\", \"DenseNodes\"},\n\t\tWriter:           f,\n\t},\n\t\tgosm.WithWritingProgram(\"wp1\"),\n\t\tgosm.WithZlipEnabled(true),\n\t)\n\tdefer func() {\n\t\t_ = encoder.Close()\n\t}()\n\n\terrChan, err := encoder.Start()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tvar errs []error\n\tgo func() {\n\t\tfor e := range errChan {\n\t\t\terrs = append(errs, e)\n\t\t}\n\t}()\n\n    sorts := make([]*Node, len(oriNodes))\n\ti := 0\n\tfor _, node := range oriNodes {\n\t\tsorts[i] = node\n\t\ti++\n\t}\n\t\n\t// write nodes order by id \n\tsort.Slice(sorts, func(i, j int) bool {\n\t\treturn sorts[i].ID \u003c sorts[j].ID\n\t})\n\n    // at most write 8000 nodes in one batch\n\tnodes := make([]*gosm.Node, 0, gosmWriteElementsMax)\n\tcount := 0\n\tfor _, n := range sorts {\n\t\tnodes = append(nodes, n.ToGosmNode())\n\t\tcount++\n\t\tif count == gosmWriteElementsMax {\n\t\t\tencoder.AppendNodes(nodes)\n\t\t\tnodes = make([]*gosm.Node, 0, gosmWriteElementsMax)\n\t\t\tcount = 0\n\t\t}\n\t}\n\tif len(nodes) \u003e 0 {\n\t\tencoder.AppendNodes(nodes)\n\t}\n\t// remember to flush\n\tencoder.Flush(gosm.NodeType)\n\n    // write ways order by way id\n\tsorts := make([]*Way, len(oriWays))\n\ti := 0\n\tfor _, way := range oriWays {\n\t\tsorts[i] = way\n\t\ti++\n\t}\n\tsort.Slice(sorts, func(i, j int) bool {\n\t\treturn sorts[i].ID \u003c sorts[j].ID\n\t})\n\n\tways := make([]*gosm.Way, 0, gosmWriteElementsMax)\n\tcount := 0\n\tfor _, w := range sorts {\n\t\tways = append(ways, w.ToGosmWay())\n\t\tcount++\n\t\tif count == gosmWriteElementsMax {\n\t\t\tencoder.AppendWays(ways)\n\t\t\tways = make([]*gosm.Way, 0, gosmWriteElementsMax)\n\t\t\tcount = 0\n\t\t}\n\t}\n\tif len(ways) \u003e 0 {\n\t\tencoder.AppendWays(ways)\n\t}\n\tencoder.Flush(gosm.WayType)\n\t\n\t// write relations by relation id\n\tsorts := make([]*Relation, len(oriRelations))\n\ti := 0\n\tfor _, rel := range oriRelations {\n\t\tsorts[i] = rel\n\t\ti++\n\t}\n\tsort.Slice(sorts, func(i, j int) bool {\n\t\treturn sorts[i].ID \u003c sorts[j].ID\n\t})\n\n\trelations := make([]*gosm.Relation, 0, gosmWriteElementsMax)\n\tcount := 0\n\tfor _, r := range sorts {\n\t\trelations = append(relations, r.ToGosmRelation())\n\t\tcount++\n\t\tif count == gosmWriteElementsMax {\n\t\t\tencoder.AppendRelations(relations)\n\t\t\trelations = make([]*gosm.Relation, 0, gosmWriteElementsMax)\n\t\t\tcount = 0\n\t\t}\n\t}\n\tif len(relations) \u003e 0 {\n\t\tencoder.AppendRelations(relations)\n\t}\n\n\tif len(errs) \u003e 0 {\n\t\t...\n\t}\n```\n\n# Notes\n1. When you use `AppendNodes`, `AppendWays` or `AppendRelations`, at most 8000 items are allowed in one append operation\n2. Use `encoder.Flush(memberType MemberType)` when you finished writing of one member type.\n\n# Contributing\nPlease raise an issue with your problem or new ideas. MRs are not accepted so far.\n\n# License\nWe are using MIT license. Please check [LICENSE.md](./LICENSE.md)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrab%2Fgosm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrab%2Fgosm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrab%2Fgosm/lists"}