{"id":25144607,"url":"https://github.com/fzdwx/go-pachong","last_synced_at":"2025-04-28T11:20:59.088Z","repository":{"id":104120329,"uuid":"397915046","full_name":"fzdwx/go-pachong","owner":"fzdwx","description":"go 爬虫，能根据一个入口url不断爬取。go web crawler, able to continuously crawl data according to an entry url","archived":false,"fork":false,"pushed_at":"2022-06-18T02:49:37.000Z","size":23,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-30T09:05:16.032Z","etag":null,"topics":["crawler","go","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"afl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fzdwx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-19T11:11:00.000Z","updated_at":"2023-03-31T07:51:39.000Z","dependencies_parsed_at":"2023-03-15T01:18:42.897Z","dependency_job_id":null,"html_url":"https://github.com/fzdwx/go-pachong","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fzdwx%2Fgo-pachong","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fzdwx%2Fgo-pachong/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fzdwx%2Fgo-pachong/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fzdwx%2Fgo-pachong/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fzdwx","download_url":"https://codeload.github.com/fzdwx/go-pachong/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251302769,"owners_count":21567601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","go","golang"],"created_at":"2025-02-08T19:46:54.689Z","updated_at":"2025-04-28T11:20:59.066Z","avatar_url":"https://github.com/fzdwx.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 流量粉碎机\n\n* [流量粉碎机](#流量粉碎机)\n    * [usage](#usage)\n    * [todo List](#todo-List)\n    * [last](#last)\n\n\n_巨耗流量_\n\n能根据入口的url爬取页面中的解析出来的url路径，然后再次对解析出来的url继续爬取...\n\ncan crawl the parsed URL path in the page according to the URL of the entrance, and then continue to crawl the parsed\nURL again...\n\n## usage\n\n调用`pa.NewPa(string)`函数，第一个入参是入口url，然后调用`AddCallback(func(string,string))`传入的函数是每次爬取到的页面返回的页面数据，可以根据需要实现。最后调用`Go()`开始爬取\n\ncall `pa.NewPa(string)`, the first input parameter is the entry URL, and the call `AddCallback(func(string,string))`\nfunction passed in is the page data returned by each crawled page, which can be implemented as needed,last call `Go()`\n\n```go\nvar wg sync.WaitGroup\n\nfunc TestPa(t *testing.T) {\n\n\turl := \"https://github.com/\"\n\n\t// 一直阻塞，没有调用wg.Done()  keeps blocking\n\twg.Add(1)\n\n\t_ = pa.NewPa(url).AddCallback(func(url, body string) {\n\t\t// nothing\n\t}).Go()\n\twg.Wait()\n}\n```\n## todo List\n\n- [ ] todo...\n\n## last\n\n欢迎任何有助于项目的issue。\n\nAny issues that help the project are welcome。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffzdwx%2Fgo-pachong","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffzdwx%2Fgo-pachong","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffzdwx%2Fgo-pachong/lists"}