{"id":20996315,"url":"https://github.com/hktalent/scrapysite","last_synced_at":"2025-05-14T21:32:05.603Z","repository":{"id":64308096,"uuid":"479915557","full_name":"hktalent/scrapysite","owner":"hktalent","description":"ScrapySite,go Web Crawler（spider）, scraping，intelligence gathering","archived":false,"fork":false,"pushed_at":"2023-10-11T21:54:42.000Z","size":121,"stargazers_count":4,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-02T23:51:10.507Z","etag":null,"topics":["crawler","elasticsearch","go","scraping","site","spider","web"],"latest_commit_sha":null,"homepage":"https://crawler.51pwn.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hktalent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-10T04:48:39.000Z","updated_at":"2024-10-03T05:16:14.000Z","dependencies_parsed_at":"2024-06-20T01:44:52.951Z","dependency_job_id":null,"html_url":"https://github.com/hktalent/scrapysite","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fscrapysite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fscrapysite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fscrapysite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fscrapysite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hktalent","download_url":"https://codeload.github.com/hktalent/scrapysite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254231100,"owners_count":22036299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","elasticsearch","go","scraping","site","spider","web"],"created_at":"2024-11-19T07:29:10.783Z","updated_at":"2025-05-14T21:32:05.229Z","avatar_url":"https://github.com/hktalent.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScrapySite\nScrapySite\n\u003cimg width=\"1082\" alt=\"image\" src=\"https://user-images.githubusercontent.com/18223385/162614808-b58ce2a8-41cc-498c-ab68-08c5ed75fdef.png\"\u003e\n\u003cimg width=\"874\" alt=\"image\" src=\"https://user-images.githubusercontent.com/18223385/162614856-72d7406f-d988-4043-b25b-9941354cb6f3.png\"\u003e\n\n\n\n# how build\n```bash\ngit clone git@github.com:hktalent/scrapysite.git\ncd scrapysite\ngo build main.go\n#or build for all palteform\nmake all -f Makefile.cross-compiles\nls -lah release/\n# or build \nmake all\nls -lah bin/\nls -lah main\n```\n\n# how use Elasticsearch\nhttp://127.0.0.1:9200/_cat/indices?v\n1、create index\n```bash\n./tools/CreateEs.sh scrapy\n```\n\nhttp://127.0.0.1:9200/scrapy_index/_doc/\n\n# how use\n```bash\n./main -url=\"http://www.xxx1.cn;http://www.xx2.cn\" -resUrl=\"http://127.0.0.1:9200/st_index/_doc/\"\n```\nhttp://127.0.0.1:9200/st_index/_search?q=edu\u0026pretty=true\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhktalent%2Fscrapysite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhktalent%2Fscrapysite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhktalent%2Fscrapysite/lists"}