{"id":13393476,"url":"https://github.com/geziyor/geziyor","last_synced_at":"2025-05-10T05:18:44.775Z","repository":{"id":38307923,"uuid":"190592538","full_name":"geziyor/geziyor","owner":"geziyor","description":"Geziyor, blazing fast web crawling \u0026 scraping framework for Go. Supports JS rendering.","archived":false,"fork":false,"pushed_at":"2025-04-16T22:28:38.000Z","size":296,"stargazers_count":2708,"open_issues_count":28,"forks_count":153,"subscribers_count":46,"default_branch":"master","last_synced_at":"2025-05-08T20:57:56.327Z","etag":null,"topics":["crawler","go","scraper","scraping","spider"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geziyor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-06-06T14:08:48.000Z","updated_at":"2025-05-08T05:32:35.000Z","dependencies_parsed_at":"2024-01-13T22:55:17.604Z","dependency_job_id":"2647f067-272f-4c63-97ce-15f6ba333b08","html_url":"https://github.com/geziyor/geziyor","commit_stats":{"total_commits":142,"total_committers":12,"mean_commits":"11.833333333333334","dds":"0.10563380281690138","last_synced_commit":"a242b58aaa654d9e521a37bc3468d047eaf5bace"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geziyor%2Fgeziyor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geziyor%2Fgeziyor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geziyor%2Fgeziyor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geziyor%2Fgeziyor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geziyor","download_url":"https://codeload.github.com/geziyor/geziyor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253157505,"owners_count":21863129,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","go","scraper","scraping","spider"],"created_at":"2024-07-30T17:00:53.833Z","updated_at":"2025-05-08T22:30:31.701Z","avatar_url":"https://github.com/geziyor.png","language":"Go","funding_links":[],"categories":["开源类库","Awesome","Go","Open source library","Repositories"],"sub_categories":["爬虫","Crawlers"],"readme":"# Geziyor\nGeziyor is a blazing fast web crawling and web scraping framework. It can be used to crawl websites and extract structured data from them. Geziyor is useful for a wide range of purposes such as data mining, monitoring and automated testing. \n\n[![GoDoc](https://godoc.org/github.com/geziyor/geziyor?status.svg)](https://godoc.org/github.com/geziyor/geziyor)\n[![report card](https://goreportcard.com/badge/github.com/geziyor/geziyor)](http://goreportcard.com/report/geziyor/geziyor)\n[![Code Coverage](https://img.shields.io/codecov/c/github/geziyor/geziyor/master.svg)](https://codecov.io/github/geziyor/geziyor?branch=master)\n\n## Features\n- **JS Rendering**\n- 5.000+ Requests/Sec\n- Caching (Memory/Disk/LevelDB)\n- Automatic Data Exporting (JSON, CSV, or custom)\n- Metrics (Prometheus, Expvar, or custom)\n- Limit Concurrency (Global/Per Domain)\n- Request Delays (Constant/Randomized)\n- Cookies, Middlewares, robots.txt\n- Automatic response decoding to UTF-8\n- Proxy management (Single, Round-Robin, Custom)\n\nSee scraper [Options](https://godoc.org/github.com/geziyor/geziyor#Options) for all custom settings. \n\n## Status\nWe highly recommend you to use Geziyor with go modules.\n\n## Usage\n\nThis example extracts all quotes from *quotes.toscrape.com* and exports to JSON file.\n\n```go\nfunc main() {\n    geziyor.NewGeziyor(\u0026geziyor.Options{\n        StartURLs: []string{\"http://quotes.toscrape.com/\"},\n        ParseFunc: quotesParse,\n        Exporters: []export.Exporter{\u0026export.JSON{}},\n    }).Start()\n}\n\nfunc quotesParse(g *geziyor.Geziyor, r *client.Response) {\n    r.HTMLDoc.Find(\"div.quote\").Each(func(i int, s *goquery.Selection) {\n        g.Exports \u003c- map[string]interface{}{\n            \"text\":   s.Find(\"span.text\").Text(),\n            \"author\": s.Find(\"small.author\").Text(),\n        }\n    })\n    if href, ok := r.HTMLDoc.Find(\"li.next \u003e a\").Attr(\"href\"); ok {\n        g.Get(r.JoinURL(href), quotesParse)\n    }\n}\n```\n\nSee [tests](https://github.com/geziyor/geziyor/blob/master/geziyor_test.go) for more usage examples.\n\n## Documentation\n\n### Installation\n\n    go get -u github.com/geziyor/geziyor\n\nIf you want to make JS rendered requests, make sure you have Chrome installed.\n\n**NOTE**: macOS limits the maximum number of open file descriptors.\nIf you want to make concurrent requests over 256, you need to increase limits.\nRead [this](https://wilsonmar.github.io/maximum-limits/) for more.\n\n### Making Normal Requests\n\nInitial requests start with ```StartURLs []string``` field in ```Options```. \nGeziyor makes concurrent requests to those URLs.\nAfter reading response, ```ParseFunc func(g *Geziyor, r *Response)``` called.\n\n```go\ngeziyor.NewGeziyor(\u0026geziyor.Options{\n    StartURLs: []string{\"http://api.ipify.org\"},\n    ParseFunc: func(g *geziyor.Geziyor, r *client.Response) {\n        fmt.Println(string(r.Body))\n    },\n}).Start()\n```\n\nIf you want to manually create first requests, set ```StartRequestsFunc```.\n```StartURLs``` won't be used if you create requests manually.  \nYou can make requests using ```Geziyor``` [methods](https://godoc.org/github.com/geziyor/geziyor#Geziyor):\n\n```go\ngeziyor.NewGeziyor(\u0026geziyor.Options{\n    StartRequestsFunc: func(g *geziyor.Geziyor) {\n    \tg.Get(\"https://httpbin.org/anything\", g.Opt.ParseFunc)\n        g.Head(\"https://httpbin.org/anything\", g.Opt.ParseFunc)\n    },\n    ParseFunc: func(g *geziyor.Geziyor, r *client.Response) {\n        fmt.Println(string(r.Body))\n    },\n}).Start()\n``` \n\n### Making JS Rendered Requests\n\nJS Rendered requests can be made using ```GetRendered``` method. \nBy default, geziyor uses local Chrome application CLI to start Chrome browser. Set ```BrowserEndpoint``` option to use different chrome instance. Such as, \"ws://localhost:3000\"\n\n```go\ngeziyor.NewGeziyor(\u0026geziyor.Options{\n    StartRequestsFunc: func(g *geziyor.Geziyor) {\n        g.GetRendered(\"https://httpbin.org/anything\", g.Opt.ParseFunc)\n    },\n    ParseFunc: func(g *geziyor.Geziyor, r *client.Response) {\n        fmt.Println(string(r.Body))\n    },\n    //BrowserEndpoint: \"ws://localhost:3000\",\n}).Start()\n```\n\n### Extracting Data\n\nWe can extract HTML elements using ```response.HTMLDoc```. HTMLDoc is Goquery's [Document](https://godoc.org/github.com/PuerkitoBio/goquery#Document).\n\nHTMLDoc can be accessible on Response if response is HTML and can be parsed using Go's built-in HTML [parser](https://godoc.org/golang.org/x/net/html#Parse)\nIf response isn't HTML, ```response.HTMLDoc``` would be ```nil```.  \n\n```go\ngeziyor.NewGeziyor(\u0026geziyor.Options{\n    StartURLs: []string{\"http://quotes.toscrape.com/\"},\n    ParseFunc: func(g *geziyor.Geziyor, r *client.Response) {\n        r.HTMLDoc.Find(\"div.quote\").Each(func(_ int, s *goquery.Selection) {\n            log.Println(s.Find(\"span.text\").Text(), s.Find(\"small.author\").Text())\n        })\n    },\n}).Start()\n```\n\n### Exporting Data\n\nYou can export data automatically using exporters. Just send data to ```Geziyor.Exports``` chan.\n[Available exporters](https://godoc.org/github.com/geziyor/geziyor/export)\n\n```go\ngeziyor.NewGeziyor(\u0026geziyor.Options{\n    StartURLs: []string{\"http://quotes.toscrape.com/\"},\n    ParseFunc: func(g *geziyor.Geziyor, r *client.Response) {\n        r.HTMLDoc.Find(\"div.quote\").Each(func(_ int, s *goquery.Selection) {\n            g.Exports \u003c- map[string]interface{}{\n                \"text\":   s.Find(\"span.text\").Text(),\n                \"author\": s.Find(\"small.author\").Text(),\n            }\n        })\n    },\n    Exporters: []export.Exporter{\u0026export.JSON{}},\n}).Start()\n```\n\n\n### Custom Requests - Passing Metadata To Callbacks\n\nYou can create custom requests with ```client.NewRequest```\n\nUse that request on ```geziyor.Do(request, callback)``` \n\n```go\ngeziyor.NewGeziyor(\u0026geziyor.Options{\n    StartRequestsFunc: func(g *geziyor.Geziyor) {\n        req, _ := client.NewRequest(\"GET\", \"https://httpbin.org/anything\", nil)\n        req.Meta[\"key\"] = \"value\"\n        g.Do(req, g.Opt.ParseFunc)\n    },\n    ParseFunc: func(g *geziyor.Geziyor, r *client.Response) {\n        fmt.Println(\"This is our data from request: \", r.Request.Meta[\"key\"])\n    },\n}).Start()\n```\n\n### Proxy - Use proxy per request\nIf you want to use proxy for your requests, and you have 1 proxy, you can just set these env values:\n`HTTP_PROXY`\n`HTTPS_PROXY`\nAnd geziyor will use those proxies.\n\nAlso, you can use in-order proxy per request by setting `ProxyFunc` option to `client.RoundRobinProxy` \nOr any custom proxy selection function that you want. See `client/proxy.go` on how to implement that kind of custom proxy selection function.\n\nProxies can be HTTP, HTTPS and SOCKS5.\n\nNote: If you use `http` scheme for proxy, It'll be used for http requests and not for https requests.   \n\n```go\ngeziyor.NewGeziyor(\u0026geziyor.Options{\n    StartURLs:         []string{\"http://httpbin.org/anything\"},\n    ParseFunc:         parseFunc,\n    ProxyFunc:         client.RoundRobinProxy(\"http://some-http-proxy.com\", \"https://some-https-proxy.com\", \"socks5://some-socks5-proxy.com\"),\n}).Start()\n```\n\n## Benchmark\n\n**8748 request per seconds** on *Macbook Pro 15\" 2016*\n\nSee [tests](https://github.com/geziyor/geziyor/blob/master/geziyor_test.go) for this benchmark function:\n\n```bash\n\u003e\u003e go test -run none -bench Requests -benchtime 10s\ngoos: darwin\ngoarch: amd64\npkg: github.com/geziyor/geziyor\nBenchmarkRequests-8   \t  200000\t    108710 ns/op\nPASS\nok  \tgithub.com/geziyor/geziyor\t22.861s\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeziyor%2Fgeziyor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeziyor%2Fgeziyor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeziyor%2Fgeziyor/lists"}