{"id":28713642,"url":"https://github.com/asimpleidea/website-poller","last_synced_at":"2025-06-15T00:11:36.387Z","repository":{"id":54688981,"uuid":"321399428","full_name":"asimpleidea/website-poller","owner":"asimpleidea","description":"A simple package for polling websites automatically","archived":false,"fork":false,"pushed_at":"2021-02-03T20:02:05.000Z","size":59,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-07-11T06:12:43.614Z","etag":null,"topics":["automatic","golang","http","poll","scrape"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asimpleidea.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-14T15:59:58.000Z","updated_at":"2021-02-03T20:01:40.000Z","dependencies_parsed_at":"2022-08-14T00:01:12.709Z","dependency_job_id":null,"html_url":"https://github.com/asimpleidea/website-poller","commit_stats":null,"previous_names":["sunsince90/auto-poller","sunsince90/website-poller"],"tags_count":12,"template":null,"template_full_name":null,"purl":"pkg:github/asimpleidea/website-poller","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimpleidea%2Fwebsite-poller","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimpleidea%2Fwebsite-poller/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimpleidea%2Fwebsite-poller/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimpleidea%2Fwebsite-poller/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asimpleidea","download_url":"https://codeload.github.com/asimpleidea/website-poller/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimpleidea%2Fwebsite-poller/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259901390,"owners_count":22929227,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automatic","golang","http","poll","scrape"],"created_at":"2025-06-15T00:11:35.718Z","updated_at":"2025-06-15T00:11:36.344Z","avatar_url":"https://github.com/asimpleidea.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Website Poller\n\n![GitHub Workflow Status](https://img.shields.io/github/workflow/status/SunSince90/website-poller/Go)\n![GitHub top language](https://img.shields.io/github/languages/top/sunsince90/website-poller)\n![GitHub](https://img.shields.io/github/license/sunsince90/website-poller)\n![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/sunsince90/website-poller)\n![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/sunsince90/website-poller)\n\nA simple and lightweight *Go* package that helps you make recurrent requests\nto a website of your page.\n\n## Overview\n\nThe package will poll a website every *X* seconds or at a random time at each\ncall with a provided list of *Headers* and a User Agent of your choice.\nAlternatively, you can provide a list of user agents that can be rotated or\nchosen randomly each time.\nA *Handler Function* of your choice will be executed whenever the requests\ncompletes - whether it failed or not.\n\n### Features\n\n* Load options from file or define them on your file\n* Poll at a fixed time\n* Poll at a random time based on a range of seconds to mimick user behavior,\ni.e. between `[30 - 50]`\nseconds\n  * Example: first poll after 32 seconds\n  * second poll after 45 seconds\n  * third poll after 30 seconds\n  * fourth poll after 37 seconds\n  * and so on...\n* Provide custom *Headers\n* Provide a *User Agents* list with the ability to either:\n  * Rotate them at each request\n  * Pick a random one each time\n* Provide no user agents list and let the package choose a random one each time\n\n### Limitations and warnings\n\nRemember that if you poll too aggressively you could be probably banned by the\nwebsite, put behind captchas or exceed quotas to the *API* service.\nThis package **will not** prevent you from being banned nor will solve captchas\nfor you. Remember to be polite and respect the rules defined by the website you\nintend to poll.\n\nThe package does not support sending a body with each request yet.\n\n### Features that will be introduced on future\n\n* Headers generator to generate headers for every request\n* Body generator to generate a different body for every request\n* Custom http client\n* Custom http request\n\n## Install\n\n```bash\ngo get github.com/SunSince90/website-poller\n```\n\n## How to use\n\nFirst of all, import it in your go file:\n\n```go\nimport (\n    poller \"github.com/SunSince90/website-poller\"\n)\n```\n\nThen, define a *Handler Function* that will be called when each request\ncompletes.\n\n```go\nfunc handleResponse(id string, resp *http.Response, err error) {\n    if err != nil {\n        // handle the error here\n    }\n\n    // Do your stuff here...\n}\n```\n\nDefine the website to poll:\n\n```go\n// Poll a website every 30 seconds\npage := \u0026poller.Page {\n    ID: \"github-sunsince90\",\n    URL: \"https://api.github.com/users/sunsince90\",\n}\n```\n\nFinally, start polling:\n\n```go\np := poller.New(page)\np.SetHandlerFunc(handleResponse)\n\nctx, canc := context.WithCancel(context.Background())\np.Start(ctx)\n```\n\n## Examples\n\nThe above program will block the main thread, follow the examples contained\nin the `examples` folder to learn more:\n\n* [Log](./examples/log/log.go): a simple logger\n* [File](./examples/file/file.go): load the pages to poll from a file\n* [Custom](./examples/custom/custom.go): a more advanced poller with\npolling options\n* [Concurrent](./examples/concurrent/concurrent.go): how to load multiple\npollers and correctly wait for them to finish\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasimpleidea%2Fwebsite-poller","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasimpleidea%2Fwebsite-poller","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasimpleidea%2Fwebsite-poller/lists"}