{"id":26649028,"url":"https://github.com/rahulsdevloper/gosearch-search-engine-scraper","last_synced_at":"2025-03-25T00:47:45.469Z","repository":{"id":283868685,"uuid":"953143820","full_name":"RahulSDevloper/GoSearch-Search-Engine-Scraper","owner":"RahulSDevloper","description":"Get search results from google, bing, duckduckgo, etc easily using GoSearch","archived":false,"fork":false,"pushed_at":"2025-03-23T06:01:33.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-23T06:27:40.375Z","etag":null,"topics":["golang","golang-library","golang-module","golang-package","golang-scraper","golang-search","golang-searcher","scraper","scraper-engine","search","search-engine","search-engine-scraper"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RahulSDevloper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-22T17:07:01.000Z","updated_at":"2025-03-23T06:01:36.000Z","dependencies_parsed_at":"2025-03-23T06:27:43.315Z","dependency_job_id":null,"html_url":"https://github.com/RahulSDevloper/GoSearch-Search-Engine-Scraper","commit_stats":null,"previous_names":["rahulsdevloper/searchenginescraper-golang","rahulsdevloper/gosearch-search-engine-scraper"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RahulSDevloper%2FGoSearch-Search-Engine-Scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RahulSDevloper%2FGoSearch-Search-Engine-Scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RahulSDevloper%2FGoSearch-Search-Engine-Scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RahulSDevloper%2FGoSearch-Search-Engine-Scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RahulSDevloper","download_url":"https://codeload.github.com/RahulSDevloper/GoSearch-Search-Engine-Scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245377954,"owners_count":20605375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","golang-library","golang-module","golang-package","golang-scraper","golang-search","golang-searcher","scraper","scraper-engine","search","search-engine","search-engine-scraper"],"created_at":"2025-03-25T00:47:44.341Z","updated_at":"2025-03-25T00:47:45.448Z","avatar_url":"https://github.com/RahulSDevloper.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🔍 Search Engine Scraper - GoSearch\n\n![GoSearch](https://github.com/user-attachments/assets/6a5cd17b-1494-4804-942e-2d104d1d533d)\n\n\u003cdiv align=\"center\"\u003e\n\n```\n   _____                     _     _____             _            \n  / ____|                   | |   |  ___|           (_)           \n | (___   ___  __ _ _ __ ___| |__ | |__ _ __   __ _  _ _ __   ___ \n  \\___ \\ / _ \\/ _` | '__/ __| '_ \\|  __| '_ \\ / _` || | '_ \\ / _ \\\n  ____) |  __/ (_| | | | (__| | | | |__| | | | (_| || | | | |  __/\n |_____/ \\___|\\__,_|_|  \\___|_| |_\\____/_| |_|\\__, ||_|_| |_|\\___|\n   _____                                        __/ |              \n  / ____|                                      |___/               \n | (___   ___ _ __ __ _ _ __   ___ _ __                            \n  \\___ \\ / __| '__/ _` | '_ \\ / _ \\ '__|                           \n  ____) | (__| | | (_| | |_) |  __/ |                              \n |_____/ \\___|_|  \\__,_| .__/ \\___|_|                              \n                       | |                                         \n                       |_|                                         \n```\n\n\u003cimg src=\"https://img.shields.io/badge/Go-1.19%2B-00ADD8?style=for-the-badge\u0026logo=go\u0026logoColor=white\" alt=\"Go Version\"/\u003e\n\u003cimg src=\"https://img.shields.io/badge/Powered%20by-Chromedp-4285F4?style=for-the-badge\u0026logo=google-chrome\u0026logoColor=white\" alt=\"Powered by Chromedp\"/\u003e\n\u003cimg src=\"https://img.shields.io/badge/Version-1.0.0-success?style=for-the-badge\" alt=\"Version\"/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/RahulSDevloper/Search-Engine-Scraper---Golang/main/assets/demo.gif\" alt=\"Search Engine Scraper Demo\" width=\"700\"/\u003e\n\n\u003cp\u003e\n\u003cb\u003eHigh-performance, anti-detection search engine scraper - Built with advanced Go concurrency patterns\u003c/b\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n\u003ch2 align=\"center\"\u003e\n  \u003ca href=\"#%EF%B8%8F-key-features\"\u003e✨ Features\u003c/a\u003e •\n  \u003ca href=\"#-installation\"\u003e🚀 Install\u003c/a\u003e •\n  \u003ca href=\"#-usage\"\u003e🔧 Usage\u003c/a\u003e •\n  \u003ca href=\"#%EF%B8%8F-examples\"\u003e🌟 Examples\u003c/a\u003e •\n  \u003ca href=\"#-advanced-techniques\"\u003e🧠 Advanced\u003c/a\u003e •\n  \u003ca href=\"#-debugging\"\u003e🐞 Debug\u003c/a\u003e\n\u003c/h2\u003e\n\n---\n\n## ✨ Key Features\n\n\u003cdiv align=\"center\"\u003e\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\" width=\"33%\"\u003e\n\u003cimg width=\"120\" src=\"https://img.shields.io/badge/Multi--Engine-Support-blue?style=for-the-badge\u0026logo=google\u0026logoColor=white\" /\u003e\n\u003cbr\u003e\u003csub\u003eGoogle, Bing \u0026 DuckDuckGo\u003c/sub\u003e\n\u003c/td\u003e\n\u003ctd align=\"center\" width=\"33%\"\u003e\n\u003cimg width=\"120\" src=\"https://img.shields.io/badge/Anti--Detection-Technology-red?style=for-the-badge\u0026logo=shield\u0026logoColor=white\" /\u003e\n\u003cbr\u003e\u003csub\u003eBypass CAPTCHAs \u0026 Blocks\u003c/sub\u003e\n\u003c/td\u003e\n\u003ctd align=\"center\" width=\"33%\"\u003e\n\u003cimg width=\"120\" src=\"https://img.shields.io/badge/Headless-Browser-green?style=for-the-badge\u0026logo=googlechrome\u0026logoColor=white\" /\u003e\n\u003cbr\u003e\u003csub\u003eChrome-Based Scraping\u003c/sub\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\" width=\"33%\"\u003e\n\u003cimg width=\"120\" src=\"https://img.shields.io/badge/Advanced-Filtering-orange?style=for-the-badge\u0026logo=filter\u0026logoColor=white\" /\u003e\n\u003cbr\u003e\u003csub\u003eDomain, Keyword \u0026 More\u003c/sub\u003e\n\u003c/td\u003e\n\u003ctd align=\"center\" width=\"33%\"\u003e\n\u003cimg width=\"120\" src=\"https://img.shields.io/badge/Smart-Processing-purple?style=for-the-badge\u0026logo=brain\u0026logoColor=white\" /\u003e\n\u003cbr\u003e\u003csub\u003eKeyword Extraction \u0026 Ad Detection\u003c/sub\u003e\n\u003c/td\u003e\n\u003ctd align=\"center\" width=\"33%\"\u003e\n\u003cimg width=\"120\" src=\"https://img.shields.io/badge/Proxy-Support-teal?style=for-the-badge\u0026logo=privatenetwork\u0026logoColor=white\" /\u003e\n\u003cbr\u003e\u003csub\u003eAvoid Rate Limiting\u003c/sub\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n## 🚀 Installation\n\n\u003cdiv align=\"center\"\u003e\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003eMethod\u003c/th\u003e\u003cth\u003eCommands\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src=\"https://img.shields.io/badge/From-Binary-brightgreen?style=flat-square\" alt=\"From Binary\"/\u003e\u003c/td\u003e\n\u003ctd\u003e\n\n```bash\n# Download the latest release\ncurl -sSL https://github.com/RahulSDevloper/Search-Engine-Scraper---Golang/releases/download/v1.0.0/gosearch-linux-amd64 -o gosearch\nchmod +x gosearch\n./gosearch --query \"golang programming\"\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src=\"https://img.shields.io/badge/From-Source-blue?style=flat-square\" alt=\"From Source\"/\u003e\u003c/td\u003e\n\u003ctd\u003e\n\n```bash\ngit clone https://github.com/RahulSDevloper/Search-Engine-Scraper---Golang.git\ncd Search-Engine-Scraper---Golang\ngo build -ldflags=\"-s -w\" -o gosearch\n./gosearch --query \"golang programming\"\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src=\"https://img.shields.io/badge/Using-Docker-blue?style=flat-square\u0026logo=docker\" alt=\"Using Docker\"/\u003e\u003c/td\u003e\n\u003ctd\u003e\n\n```bash\ndocker pull rahulsdevloper/gosearch:latest\ndocker run rahulsdevloper/gosearch --query \"golang programming\"\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n## 🔧 Usage\n\n```\nUsage: gosearch [OPTIONS] [QUERY]\n\nOptions:\n  --query string         Search query\n  --engine string        Search engine (google, bing, duckduckgo, all) (default \"google\")\n  --max int              Maximum results to fetch (default 10)\n  --ads                  Include advertisements in results\n  --timeout duration     Search timeout (default 30s)\n  --proxy string         Proxy URL (e.g., http://user:pass@host:port)\n  --headless             Use headless browser (recommended for avoiding detection)\n  --lang string          Language code (default \"en\")\n  --region string        Region code (default \"us\")\n  --format string        Output format (json, csv, table) (default \"json\")\n  --output string        Output file (default: stdout)\n  --page int             Result page number (default 1)\n  --min-words int        Minimum word count in description\n  --max-words int        Maximum word count in description\n  --domain string        Filter results by domain (include)\n  --exclude-domain string Filter results by domain (exclude)\n  --keyword string       Filter results by keyword\n  --type string          Filter by result type (organic, special, etc.)\n  --site string          Limit results to specific site\n  --filetype string      Limit results to specific file type\n  --verbose              Enable verbose logging\n  --debug                Enable debug mode (saves HTML responses)\n  --log string           Log file path\n  --stats string         Statistics output file\n  --help                 Show help\n```\n\n## 🌟 Examples\n\n\u003cdetails open\u003e\n\u003csummary\u003e\u003cb\u003eBasic Search with Google\u003c/b\u003e 🔍\u003c/summary\u003e\n\n```bash\n./gosearch --query \"golang programming\"\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/RahulSDevloper/Search-Engine-Scraper---Golang/main/assets/example1.gif\" alt=\"Basic Search Example\" width=\"600\"/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSearch with Advanced Filters\u003c/b\u003e 🧰\u003c/summary\u003e\n\n```bash\n./gosearch --query \"machine learning\" --engine bing --domain edu --format table\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/RahulSDevloper/Search-Engine-Scraper---Golang/main/assets/example2.gif\" alt=\"Advanced Search Example\" width=\"600\"/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eMulti-Engine Search with Headless Browser\u003c/b\u003e 🌐\u003c/summary\u003e\n\n```bash\n./gosearch --query \"climate science\" --engine all --headless --output results.json\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/RahulSDevloper/Search-Engine-Scraper---Golang/main/assets/example3.gif\" alt=\"Multi-Engine Example\" width=\"600\"/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eFiletype Specific Search\u003c/b\u003e 📄\u003c/summary\u003e\n\n```bash\n./gosearch --query \"research papers\" --filetype pdf --site edu --max 20\n```\n\u003c/details\u003e\n\n## 🧠 Advanced Techniques\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/RahulSDevloper/Search-Engine-Scraper---Golang/main/assets/advanced.svg\" alt=\"Advanced Features\" width=\"700\"/\u003e\n\u003c/div\u003e\n\n### Using as a Library\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"time\"\n    \n    \"github.com/RahulSDevloper/Search-Engine-Scraper---Golang/pkg/engines\"\n    \"github.com/RahulSDevloper/Search-Engine-Scraper---Golang/pkg/models\"\n)\n\nfunc main() {\n    // Create a new Google search engine\n    engine := engines.NewGoogleSearchEngine()\n    \n    // Configure search request with optimization strategy\n    request := models.SearchRequest{\n        Query:       \"golang concurrency patterns\",\n        MaxResults:  10,\n        Timeout:     30 * time.Second,\n        UseHeadless: true,\n        Debug:       true,\n    }\n    \n    // Execute search with context for cancellation\n    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)\n    defer cancel()\n    \n    results, err := engine.Search(ctx, request)\n    if err != nil {\n        fmt.Printf(\"Error: %v\\n\", err)\n        return\n    }\n    \n    // Process and analyze results\n    for i, result := range results {\n        fmt.Printf(\"%d. %s\\n%s\\n\\n\", i+1, result.Title, result.URL)\n    }\n}\n```\n\n### Custom Rate Limiting\n\n```yaml\n# ~/.config/gosearch/config.yaml\nrate_limits:\n  google: 10   # requests per minute\n  bing: 15\n  duckduckgo: 20\n\nproxy_rotation:\n  enabled: true\n  proxies:\n    - http://proxy1:8080\n    - http://proxy2:8080\n  rotation_strategy: round-robin  # or random\n```\n\n## 🐞 Debugging\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/RahulSDevloper/Search-Engine-Scraper---Golang/main/assets/debugging.svg\" alt=\"Debugging Techniques\" width=\"700\"/\u003e\n\u003c/div\u003e\n\n### No Results Found?\n\nIf you're not getting any results, try these solutions:\n\n1. **Use Headless Mode** to avoid detection\n   ```bash\n   ./gosearch --query \"your search\" --headless\n   ```\n\n2. **Use a Proxy** to route through a clean IP address\n   ```bash\n   ./gosearch --query \"your search\" --proxy http://your-proxy-server:port\n   ```\n\n3. **Enable Debug Mode** to examine the HTML response\n   ```bash\n   ./gosearch --query \"your search\" --debug\n   ```\n\n### Debugging Process Flow\n\n```mermaid\ngraph TD\n    A[Run Search] --\u003e B{Results Found?}\n    B --\u003e|Yes| C[Process Results]\n    B --\u003e|No| D[Enable Debug Mode]\n    D --\u003e E[Check HTML Responses]\n    E --\u003e F{Captcha Present?}\n    F --\u003e|Yes| G[Use Headless + Proxy]\n    F --\u003e|No| H[Check Selectors]\n    H --\u003e I[Update Selectors]\n    I --\u003e A\n    G --\u003e A\n```\n\n## 📊 Performance Benchmarks\n\n\u003cdiv align=\"center\"\u003e\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003eEngine\u003c/th\u003e\u003cth\u003eResults/Second\u003c/th\u003e\u003cth\u003eMemory Usage\u003c/th\u003e\u003cth\u003eDetection Avoidance\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eGoogle\u003c/td\u003e\u003ctd\u003e6.5\u003c/td\u003e\u003ctd\u003eLow\u003c/td\u003e\u003ctd\u003eHigh\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eBing\u003c/td\u003e\u003ctd\u003e8.2\u003c/td\u003e\u003ctd\u003eLow\u003c/td\u003e\u003ctd\u003eMedium\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eDuckDuckGo\u003c/td\u003e\u003ctd\u003e7.3\u003c/td\u003e\u003ctd\u003eLow\u003c/td\u003e\u003ctd\u003eVery High\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003eAll (Concurrent)\u003c/td\u003e\u003ctd\u003e4.8\u003c/td\u003e\u003ctd\u003eMedium\u003c/td\u003e\u003ctd\u003eMedium\u003c/td\u003e\u003c/tr\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n## 📚 Design Philosophy\n\nThe Search Engine Scraper follows these core principles:\n\n1. **Resilience First**: Designed to handle the constantly changing DOM structures of search engines\n2. **Performance Focused**: Optimized for speed while maintaining low resource usage\n3. **Privacy Conscious**: Minimal footprint to avoid detection\n4. **Developer Friendly**: Clean API for integration into other Go applications\n\n## 📝 License\n\n\u003cdiv align=\"center\"\u003e\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n\u003c/div\u003e\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://img.shields.io/badge/Made%20with%20%E2%9D%A4%EF%B8%8F%20by-RahulSDevloper-red.svg?style=for-the-badge\" alt=\"Made with love by RahulSDevloper\"/\u003e\n\n⭐ Star this project if you find it useful! ⭐\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahulsdevloper%2Fgosearch-search-engine-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frahulsdevloper%2Fgosearch-search-engine-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahulsdevloper%2Fgosearch-search-engine-scraper/lists"}