{"id":18309401,"url":"https://github.com/masterjoyhunan/region","last_synced_at":"2026-01-20T08:07:53.193Z","repository":{"id":81016626,"uuid":"257470916","full_name":"MasterJoyHunan/region","owner":"MasterJoyHunan","description":"国家统计局省市区编码 使用 GO 语言 colly 爬虫框架","archived":false,"fork":false,"pushed_at":"2020-04-23T03:52:15.000Z","size":414,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-09T11:49:10.545Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MasterJoyHunan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-21T03:34:44.000Z","updated_at":"2020-04-23T03:52:17.000Z","dependencies_parsed_at":"2023-03-23T02:31:19.722Z","dependency_job_id":null,"html_url":"https://github.com/MasterJoyHunan/region","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MasterJoyHunan/region","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MasterJoyHunan%2Fregion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MasterJoyHunan%2Fregion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MasterJoyHunan%2Fregion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MasterJoyHunan%2Fregion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MasterJoyHunan","download_url":"https://codeload.github.com/MasterJoyHunan/region/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MasterJoyHunan%2Fregion/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28598875,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T02:08:49.799Z","status":"ssl_error","status_checked_at":"2026-01-20T02:08:44.148Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T16:11:19.337Z","updated_at":"2026-01-20T08:07:53.175Z","avatar_url":"https://github.com/MasterJoyHunan.png","language":"TSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"### 由GO语言编写的爬虫爬取国家统计局省市区编码,并存储MYSQL\n\n* [国家统计局省市区统计地址](http://www.stats.gov.cn/tjsj/tjbz/tjyqhdmhcxhfdm)\n\n注意：国家统计局官网有反爬虫系统，爬取速度不能过快，否则会出现请求重定向，导致爬取失败\n所以添加如下参数：\n```go\n// 限制速度\nif err := c.Limit(\u0026colly.LimitRule{\n    DomainGlob:  \"*\", \n    Parallelism: 2,\n    Delay:       200 * time.Millisecond, \n    RandomDelay: 5 * time.Second,\n}); err != nil {\n    fastmysql.Logger.Panic(\"set colly limit error :\", err)\n}\n```\n这样导致的后果也很明显，没有错误，但是爬取速度也变的非常慢。好在这是一劳永逸的工作，爬取之后就完事了。\n项目是爬取2019年的数据，如需修改，修改爬取地址（页面结构都是一样的，无需修改 OnHtml 回调）\n```go\nc.Visit(\"http://www.stats.gov.cn/tjsj/tjbz/tjyqhdmhcxhfdm/2019/index.html\")\n```\n爬取结果不包含港澳台\n\ntips: 在实战项目中，省市区数据应该存储在 redis 中最好","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmasterjoyhunan%2Fregion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmasterjoyhunan%2Fregion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmasterjoyhunan%2Fregion/lists"}