{"id":20705491,"url":"https://github.com/whosonfirst/go-whosonfirst-iterate-github","last_synced_at":"2025-09-29T07:12:06.080Z","repository":{"id":97594548,"uuid":"343920521","full_name":"whosonfirst/go-whosonfirst-iterate-github","owner":"whosonfirst","description":"Go package for iterating through a set of Who's On First documents stored in a GitHub repository, using the GitHub API.","archived":false,"fork":false,"pushed_at":"2025-06-23T15:41:05.000Z","size":1279,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-23T16:55:50.925Z","etag":null,"topics":["github","golang","whosonfirst","whosonfirst-iterate"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/whosonfirst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-02T21:41:46.000Z","updated_at":"2025-04-22T19:14:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"7c176fac-2db6-439b-8acd-1e493197eba6","html_url":"https://github.com/whosonfirst/go-whosonfirst-iterate-github","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/whosonfirst/go-whosonfirst-iterate-github","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-iterate-github","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-iterate-github/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-iterate-github/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-iterate-github/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/whosonfirst","download_url":"https://codeload.github.com/whosonfirst/go-whosonfirst-iterate-github/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-iterate-github/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277478805,"owners_count":25824872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-29T02:00:09.175Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github","golang","whosonfirst","whosonfirst-iterate"],"created_at":"2024-11-17T01:18:20.192Z","updated_at":"2025-09-29T07:12:06.073Z","avatar_url":"https://github.com/whosonfirst.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-whosonfirst-iterate-github\n\nGo package for iterating through a set of Who's On First documents stored in a GitHub repository, using the GitHub API.\n\n## Important\n\nThis package has been deprecated and is no longer maintained. All of the functionality in this package has been migrated in to [whosonfirst/go-whosonfirst-git/v3](https://github.com/whosonfirst/go-whosonfirst-iterate-git).\n\n## Example\n\n```\npackage main\n\nimport (\n       \"context\"\n       \"flag\"\n       \"io\"\n       \"log\"\n\n       _ \"github.com/whosonfirst/go-whosonfirst-iterate-github/v2\"\n       \n       \"github.com/whosonfirst/go-whosonfirst-iterate/emitter/v2\"       \n       \"github.com/whosonfirst/go-whosonfirst-iterate/indexer/v2\"\n)\n\nfunc main() {\n\n\temitter_uri := flag.String(\"emitter-uri\", \"githubapi://\", \"A valid whosonfirst/go-whosonfirst-iterate/emitter URI\")\n\t\n     \tflag.Parse()\n\n\tctx := context.Background()\n\n\temitter_cb := func(ctx context.Context, path string, fh io.ReadSeeker, args ...interface{}) error {\n\t\tlog.Printf(\"Indexing %s\\n\", path)\n\t\treturn nil\n\t}\n\n\titer, _ := iterator.NewIterator(ctx, *emitter_uri, cb)\n\n\turis := flag.Args()\n\titer.IterateURIs(ctx, uris...)\n}\n```\n\n_Error handling removed for the sake of brevity._\n\n\n## URIs and Schemes \n\n### githubapi://\n\n```\ngithubapi://{GITHUB_ORGANIZATION}/{GITHUB_REPO}\n```\n\n## Query parameters\n\nIn addition to the [default go-whosonfirst-iterate query parameters](https://github.com/whosonfirst/go-whosonfirst-iterate#query-parameters) the following query parameters are supported:\n\n| Name | Value | Required | Notes\n| --- | --- | --- | --- |\n| access_token | String | Yes | A valid [GitHub API access token](https://docs.github.com/en/rest/overview/other-authentication-methods) |\n| branch | String | No | The branch to use when iterating the repository contents |\n| concurrent | Bool | No | If true iterate through documents concurrently. There is still a throttle on the number of API requests per second but this can speed things up significantly with the risk that you will still trigger GitHub API limits. |\n\n## Filters\n\n### QueryFilters\n\nYou can also specify inline queries by appending one or more `include` or `exclude` parameters to a `emitter.Emitter` URI, where the value is a string in the format of:\n\n```\n{PATH}={REGULAR EXPRESSION}\n```\n\nPaths follow the dot notation syntax used by the [tidwall/gjson](https://github.com/tidwall/gjson) package and regular expressions are any valid [Go language regular expression](https://golang.org/pkg/regexp/). Successful path lookups will be treated as a list of candidates and each candidate's string value will be tested against the regular expression's [MatchString](https://golang.org/pkg/regexp/#Regexp.MatchString) method.\n\nFor example:\n\n```\nrepo://?include=properties.wof:placetype=region\n```\n\nYou can pass multiple query parameters. For example:\n\n```\nrepo://?include=properties.wof:placetype=region\u0026include=properties.wof:name=(?i)new.*\n```\n\nThe default query mode is to ensure that all queries match but you can also specify that only one or more queries need to match by appending a `include_mode` or `exclude_mode` parameter where the value is either \"ANY\" or \"ALL\".\n\n## Tools\n\n```\n$\u003e make cli\ngo build -mod vendor -o bin/count cmd/count/main.go\ngo build -mod vendor -o bin/emit cmd/emit/main.go\n```\n\n### count\n\nCount files in one or more whosonfirst/go-whosonfirst-iterate/emitter sources.\n\n```\n\u003e ./bin/count -h\nCount files in one or more whosonfirst/go-whosonfirst-iterate/emitter sources.\nUsage:\n\t ./bin/count [options] uri(N) uri(N)\nValid options are:\n\n  -emitter-uri string\n        A valid whosonfirst/go-whosonfirst-iterate/emitter URI. Supported emitter URI schemes are: directory://,featurecollection://,file://,filelist://,geojsonl://,githubapi://,repo://\n\n```\n\nFor example:\n\n```\n$\u003e ./bin/count \\\n\t-emitter-uri 'githubapi://sfomuseum-data/sfomuseum-data-architecture?concurrent=1\u0026access_token={TOKEN}' \\\n\tdata\n\n2021/03/02 13:06:08 time to index paths (1) 1m11.522679037s\n2021/03/02 13:06:08 Counted 1077 records (1077) in 1m11.522714392s\n```\n\nOr:\n\n```\n$\u003e ./bin/count \\\n\t-emitter-uri 'githubapi://sfomuseum-data/sfomuseum-data-architecture?concurrent=1\u0026access_token={TOKEN}\u0026include=properties.sfomuseum:placetype=museum' \\\n\tdata\n\n2021/03/02 13:35:25 time to index paths (1) 1m10.897179298s\n2021/03/02 13:35:25 Counted 7 records (7) in 1m10.897222091s\n```\n\n### emit\n\nPublish features from one or more whosonfirst/go-whosonfirst-index/v2/emitter sources.\n\n```\n\u003e ./bin/emit -h\nPublish features from one or more whosonfirst/go-whosonfirst-iterate/emitter sources.\nUsage:\n\t ./bin/emit [options] uri(N) uri(N)\nValid options are:\n\n  -emitter-uri string\n        A valid whosonfirst/go-whosonfirst-iterate/emitter URI. Supported emitter URI schemes are: directory://,featurecollection://,file://,filelist://,geojsonl://,githubapi://,repo://\n  -geojson\n    \tEmit features as a well-formed GeoJSON FeatureCollection record.\n  -json\n    \tEmit features as a well-formed JSON array.\n  -null\n    \tPublish features to /dev/null\n  -stdout\n    \tPublish features to STDOUT. (default true)\n```\n\nFor example:\n\n```\n$\u003e ./bin/emit \\\n\t-emitter-uri 'githubapi://sfomuseum-data/sfomuseum-data-architecture?concurrent=1\u0026access_token={TOKEN}\u0026include=properties.sfomuseum:placetype=museum' \\\n\t-geojson \\\t\n\tdata\n\n| jq '.features[][\"properties\"][\"wof:id\"]'\n\n1729813675\n1477855937\n1360521563\n1360521569\n1360521565\n1360521571\n1159157863\n```\n\n## See also\n\n* https://github.com/whosonfirst/go-whosonfirst-iterate\n* https://github.com/google/go-github/github\n* https://docs.github.com/en/rest/reference/repos#contents","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhosonfirst%2Fgo-whosonfirst-iterate-github","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwhosonfirst%2Fgo-whosonfirst-iterate-github","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhosonfirst%2Fgo-whosonfirst-iterate-github/lists"}