{"id":19084344,"url":"https://github.com/coghost/wee","last_synced_at":"2025-10-14T18:35:14.453Z","repository":{"id":214976602,"uuid":"713383532","full_name":"coghost/wee","owner":"coghost","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-23T08:35:56.000Z","size":251,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-14T18:35:09.473Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coghost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-02T12:12:05.000Z","updated_at":"2025-09-23T08:34:37.000Z","dependencies_parsed_at":"2024-01-01T17:23:26.295Z","dependency_job_id":"48c13882-bffc-464b-a76a-fff6c75b6569","html_url":"https://github.com/coghost/wee","commit_stats":null,"previous_names":["coghost/wee"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/coghost/wee","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coghost%2Fwee","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coghost%2Fwee/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coghost%2Fwee/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coghost%2Fwee/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coghost","download_url":"https://codeload.github.com/coghost/wee/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coghost%2Fwee/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279020353,"owners_count":26086866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T02:50:59.307Z","updated_at":"2025-10-14T18:35:14.438Z","avatar_url":"https://github.com/coghost.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wee\n\nWee is a powerful and flexible web crawling library based on go-rod, providing a clean interface for writing web crawlers in Go. It offers advanced browser control and customization options.\n\n## Features\n\n- Easy-to-use API for web crawling tasks\n- Support for headless and headed browser modes\n- User mode for connecting to system Chrome browser\n- Cookie management and persistence\n- Customizable timeouts and error handling\n- Humanized and stealth mode options\n- Popover handling\n- Advanced browser configuration options\n- Support for Chrome extensions\n- Proxy configuration\n\n## Installation\n\n```\ngo get github.com/coghost/wee\n```\n\n## Usage\n\nHere's a basic example of using wee to crawl search results from Baidu:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/coghost/wee\"\n)\n\nfunc main() {\n\tbot := wee.NewBotDefault()\n\tdefer bot.BlockInCleanUp()\n\n\tconst (\n\t\turl     = `https://www.baidu.com`\n\t\tinput   = `input[id=\"kw\"]`\n\t\tkeyword = `golang`\n\t\titems   = `div.result[id] h3\u003ea`\n\t\tnoItems = `div.this_should_not_existed`\n\t)\n\n\tbot.MustOpen(url)\n\tbot.MustInput(input, keyword, wee.WithSubmit(true))\n\n\tgot := bot.MustAnyElem([]string{items, noItems})\n\tif got == noItems {\n\t\tfmt.Printf(\"%s has no results\\n\", keyword)\n\t\treturn\n\t}\n\n\telems, err := bot.Elems(items)\n\tif err != nil {\n\t\tfmt.Printf(\"get items failed: %v\\n\", err)\n\t\treturn\n\t}\n\n\tfor _, elem := range elems {\n\t\tfmt.Printf(\"%s - %s\\n\", elem.MustText(), *elem.MustAttribute(\"href\"))\n\t}\n}\n```\n\n## Advanced Usage\n\n### Creating a Bot\n\nWee provides several ways to create a bot:\n\n```go\n// Default bot\nbot := wee.NewBotDefault()\n\n// Headless bot\nbot := wee.NewBotHeadless()\n\n// Bot for debugging\nbot := wee.NewBotForDebug()\n\n// User mode bot (connects to system Chrome)\nbot := wee.NewBotUserMode()\n```\n\n### Customizing Bot Behavior\n\nYou can customize bot behavior using options:\n\n```go\nbot := wee.NewBot(\n    wee.UserAgent(\"Custom User Agent\"),\n    wee.AcceptLanguage(\"en-US\"),\n    wee.WithCookies(true),\n    wee.Humanized(true),\n    wee.StealthMode(true),\n    wee.WithPopovers(\"popover-selector\"),\n)\n```\n\n### Cookie Management\n\nWee supports various ways to manage cookies:\n\n```go\n// Use cookies from default location\nwee.WithCookies(true)\n\n// Specify cookie folder\nwee.WithCookieFolder(\"/path/to/cookies\")\n\n// Use specific cookie file\nwee.WithCookieFile(\"/path/to/cookiefile.json\")\n\n// Use cookies from clipboard (Copy as cURL)\nwee.CopyAsCURLCookies(clipboardContent)\n```\n\n### Error Handling\n\nYou can customize error handling behavior:\n\n```go\nbot.SetPanicWith(wee.PanicByLogError)\n```\n\n## Advanced Browser Configuration\n\nWee provides powerful options to configure the browser:\n\n### Creating a Browser\n\n```go\nlauncher, browser := wee.NewBrowser(\n    wee.BrowserHeadless(false),\n    wee.BrowserSlowMotion(1000),\n    wee.BrowserUserDataDir(\"/path/to/user/data\"),\n    wee.BrowserPaintRects(true),\n    wee.BrowserProxy(\"http://proxy.example.com:8080\"),\n)\n```\n\n### User Mode Browser\n\n```go\nlauncher, browser := wee.NewUserMode(\n    wee.BrowserUserDataDir(\"/path/to/user/data\"),\n    wee.LaunchLeakless(true),\n)\n```\n\n### Customizing Launcher\n\n```go\nlauncher := wee.NewLauncher(\n    wee.BrowserHeadless(true),\n    wee.BrowserExtensions(\"/path/to/extension1\", \"/path/to/extension2\"),\n    wee.BrowserFlags(\"--disable-gpu\", \"--no-sandbox\"),\n)\n```\n\n## Browser Options\n\nWee supports various browser options:\n\n- `BrowserHeadless(bool)`: Run browser in headless mode\n- `BrowserSlowMotion(int)`: Set slow motion delay in milliseconds\n- `BrowserUserDataDir(string)`: Set user data directory\n- `BrowserPaintRects(bool)`: Show paint rectangles (useful for debugging)\n- `BrowserProxy(string)`: Set proxy server\n- `BrowserExtensions(...string)`: Load Chrome extensions\n- `BrowserFlags(...string)`: Set additional Chrome flags\n- `LaunchLeakless(bool)`: Use leakless mode when launching browser\n- `BrowserIncognito(bool)`: Launch browser in incognito mode\n- `BrowserIgnoreCertErrors(bool)`: Ignore certificate errors\n\n## More Examples\n\nCheck out the [examples folder](https://github.com/coghost/wee/tree/main/examples) for more detailed examples.\n\n## Common Issues\n\n### panic: context canceled\n\nThe \"context canceled\" issue usually occurs when an element is bound with a timeout on `bot.Elem(xxx)`. To resolve this, you can use:\n\n```go\nelem.CancelTimeout().Timeout(xxx)\n```\n\nThis will rewrite the previous timeout value.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\n[MIT License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoghost%2Fwee","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoghost%2Fwee","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoghost%2Fwee/lists"}