{"id":20906025,"url":"https://github.com/telanflow/scrago","last_synced_at":"2025-06-25T22:41:14.895Z","repository":{"id":57658704,"uuid":"124474917","full_name":"telanflow/scrago","owner":"telanflow","description":"A micro crawler framework. achieved by GOLANG.","archived":false,"fork":false,"pushed_at":"2020-11-04T15:09:02.000Z","size":196,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-12T21:34:58.604Z","etag":null,"topics":["crawler","go","micro-framework","spider"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/telanflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-09T02:17:09.000Z","updated_at":"2020-11-04T15:09:04.000Z","dependencies_parsed_at":"2022-09-16T04:41:50.976Z","dependency_job_id":null,"html_url":"https://github.com/telanflow/scrago","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/telanflow/scrago","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telanflow%2Fscrago","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telanflow%2Fscrago/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telanflow%2Fscrago/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telanflow%2Fscrago/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/telanflow","download_url":"https://codeload.github.com/telanflow/scrago/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telanflow%2Fscrago/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261966466,"owners_count":23237622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","go","micro-framework","spider"],"created_at":"2024-11-18T13:28:40.639Z","updated_at":"2025-06-25T22:41:14.868Z","avatar_url":"https://github.com/telanflow.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Teler   \n\n  A micro crawler framework. achieved by GOLANG.\n\n[![Build Status](https://travis-ci.org/telanflow/scrago.svg?branch=master)](https://travis-ci.org/telanflow/scrago) [![GitHub stars](https://img.shields.io/github/stars/telanflow/scrago.svg)](https://github.com/telanflow/scrago/stargazers) [![Go version](https://img.shields.io/badge/Go-%3E1.7-brightgreen.svg)](https://github.com/telanflow/scrago)\n[![996.icu](https://img.shields.io/badge/link-996.icu-red.svg)](https://996.icu)\n[![LICENSE](https://img.shields.io/badge/license-NPL%20(The%20996%20Prohibited%20License)-blue.svg)](https://github.com/996icu/996.ICU/blob/master/LICENSE)\n\n## Quick Start\n\n#### Download and install\n\n    go get github.com/telanflow/scrago\n    \n#### Create file `my_spider.go`\n```go\npackage main\n\nimport (\n\t\"net/http\"\n\t\"net/http/cookiejar\"\n\n\t\"github.com/telanflow/scrago\"\n\t\"github.com/telanflow/scrago/pages\"\n\t\"github.com/telanflow/scrago/downloader\"\n)\n\n\ntype MySpider struct{\n\tjar http.CookieJar\n}\n\n// Init\nfunc (m *MySpider) Init(ctx *teler.Context) {\n\t// Set the persistent cookie.\n\tm.jar, _ = cookiejar.New(nil)\n\tctx.GetDownloader().UseOptions(downloader.WithCookieJar(m.jar))\n\n\t// Add Target Url\n\t//ctx.AddUrl(\"https://www.baidu.com\")\n}\n\n// Page Process\nfunc (m *MySpider) Process(ctx *teler.Context, page *pages.Page) {\n\n}\n\n// Pipeline Output\nfunc (m *MySpider) Output(items *pages.PageItem) {\n\n}\n\n\nfunc main() {\n\t// Start Spider\n\tscrago.New(\u0026MySpider{}).AddUrl(\"https://www.baidu.com\").Run()\n}\n\n```\n\n#### Build and run\n\n    go build my_spider.go\n    ./my_spider\n    \n## Documentation\n[中文文档](https://github.com/telanflow/scrago/wiki/%E6%A1%86%E6%9E%B6%E7%AE%80%E4%BB%8B)\n    \n## License\n\nteler licensed under the Apache Licence, Version 2.0\n(http://www.apache.org/licenses/LICENSE-2.0.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelanflow%2Fscrago","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftelanflow%2Fscrago","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelanflow%2Fscrago/lists"}