{"id":13564148,"url":"https://github.com/foomo/walker","last_synced_at":"2025-11-25T22:03:33.606Z","repository":{"id":57519532,"uuid":"171853025","full_name":"foomo/walker","owner":"foomo","description":"Crawls website and collect SEO relevant data","archived":false,"fork":false,"pushed_at":"2022-09-27T17:14:50.000Z","size":192,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-01-12T15:46:04.351Z","etag":null,"topics":["apache-benchmark","benchmarking","foomo","foomo-walker","siege","spider","website-crawler"],"latest_commit_sha":null,"homepage":"https://www.foomo.org","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/foomo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-21T10:41:27.000Z","updated_at":"2024-07-07T13:14:40.000Z","dependencies_parsed_at":"2022-08-30T04:22:08.730Z","dependency_job_id":null,"html_url":"https://github.com/foomo/walker","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foomo%2Fwalker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foomo%2Fwalker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foomo%2Fwalker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foomo%2Fwalker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/foomo","download_url":"https://codeload.github.com/foomo/walker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241451677,"owners_count":19964901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-benchmark","benchmarking","foomo","foomo-walker","siege","spider","website-crawler"],"created_at":"2024-08-01T13:01:27.179Z","updated_at":"2025-11-25T22:03:33.507Z","avatar_url":"https://github.com/foomo.png","language":"Go","readme":"# Walker\n\nWalker walkes aka as crawls through websites and collects performance and SEO relevant data. The results can be browsed through a very simple web interface. Apart from that they are exposed as prometheus metrics (not implemented yet).\n\n**Be careful when crawling your website with walker with aggressive settings, it might take your site down**\n\n## Configuration\n\n```yaml\n---\n# target of your scrape\ntarget: http://www.bestbytes.de\n# number of concurrent go routines\nconcurrency: 2\n# where to run the webinterface\naddr: \":3001\"\n# if you want to ignore \u003cmeta name=\"robots\" content=\"noindex,nofollow\"/\u003e\nignorerobots: true\n# in some cases using cookies is friendlier to the server\nusecookies: true\n\n# ignoring urls\n## based on query parameters in this example all links, that contain a queryparameter foo\nignorequerieswith:\n  - foo\n## skip everything that has a query\nignoreallqueries: true\n# what paths (that would be a prefixes)\nignore:\n  - /foomo\n...\n```\n\n## error detection\n\n- everything greater than 400 will be tracked as an error\n\n## external link validation (not implemented yet)\n\n- check external links\n- forbidden sites like a stage system \n\n## seo validation\n\n- missing title, description, h1\n- duplication title, description, h1\n\n### seo validation schemata\n\nWIP\n\n## metrics\n\nWork in progress exposed on /metrics\n\n- vector of status codes\n- performance buckets","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoomo%2Fwalker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffoomo%2Fwalker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoomo%2Fwalker/lists"}