{"id":13464637,"url":"https://github.com/manning23/MSpider","last_synced_at":"2025-03-25T11:31:55.274Z","repository":{"id":23311333,"uuid":"26670991","full_name":"manning23/MSpider","owner":"manning23","description":"Spider","archived":false,"fork":false,"pushed_at":"2022-07-11T12:03:58.000Z","size":1254,"stargazers_count":348,"open_issues_count":6,"forks_count":192,"subscribers_count":55,"default_branch":"master","last_synced_at":"2024-10-29T17:51:15.315Z","etag":null,"topics":["mspider"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manning23.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-11-15T05:59:08.000Z","updated_at":"2024-09-21T12:47:42.000Z","dependencies_parsed_at":"2022-08-21T23:30:27.591Z","dependency_job_id":null,"html_url":"https://github.com/manning23/MSpider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manning23%2FMSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manning23%2FMSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manning23%2FMSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manning23%2FMSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manning23","download_url":"https://codeload.github.com/manning23/MSpider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245454066,"owners_count":20617970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mspider"],"created_at":"2024-07-31T14:00:47.796Z","updated_at":"2025-03-25T11:31:51.492Z","avatar_url":"https://github.com/manning23.png","language":"Python","funding_links":[],"categories":["All","Python"],"sub_categories":[],"readme":"# MSpider\n\n## Talk\n\nThe information security department of 360 company has been recruiting for a long time and is interested in contacting the mailbox zhangxin1[at]360.cn.\n\n## Installation\n\nIn Ubuntu, you need to install some libraries.\n\nYou can use pip or easy_install or apt-get to do this.\n\n- lxml\n- chardet\n- splinter\n- gevent\n- phantomjs\n\n## Example\n\n1. Use MSpider collect the vulnerability information on the wooyun.org.\n```\n\tpython mspider.py -u \"http://www.wooyun.org/bugs/\" --focus-domain \"wooyun.org\" --filter-keyword \"xxx\" --focus-keyword \"bugs\" -t 15 --random-agent true\n```\n\n\n2. Use MSpider collect the news information on the news.sina.com.cn.\n```\n\tpython mspider.py -u \"http://news.sina.com.cn/c/2015-12-20/doc-ifxmszek7395594.shtml\" --focus-domain \"news.sina.com.cn\"  -t 15 --random-agent true\n```\n\n## ToDo\n\n1. Crawl and storage of information.\n2. Distributed crawling.\n\n## MSpider's help\n\n```\nUsage:\n  __  __  _____       _     _\n |  \\/  |/ ____|     (_)   | |\n | \\  / | (___  _ __  _  __| | ___ _ __\n | |\\/| |\\___ \\| '_ \\| |/ _` |/ _ \\ '__|\n | |  | |____) | |_) | | (_| |  __/ |\n |_|  |_|_____/| .__/|_|\\__,_|\\___|_|\n               | |\n               |_|\n                        Author: Manning23\n\n\nOptions:\n  -h, --help            show this help message and exit\n  -u MSPIDER_URL, --url=MSPIDER_URL\n                        Target URL (e.g. \"http://www.site.com/\")\n  -t MSPIDER_THREADS_NUM, --threads=MSPIDER_THREADS_NUM\n                        Max number of concurrent HTTP(s) requests (default 10)\n  --depth=MSPIDER_DEPTH\n                        Crawling depth\n  --count=MSPIDER_COUNT\n                        Crawling number\n  --time=MSPIDER_TIME   Crawl time\n  --referer=MSPIDER_REFERER\n                        HTTP Referer header value\n  --cookies=MSPIDER_COOKIES\n                        HTTP Cookie header value\n  --spider-model=MSPIDER_MODEL\n                        Crawling mode: Static_Spider: 0  Dynamic_Spider: 1\n                        Mixed_Spider: 2\n  --spider-policy=MSPIDER_POLICY\n                        Crawling strategy: Breadth-first 0  Depth-first 1\n                        Random-first 2\n  --focus-keyword=MSPIDER_FOCUS_KEYWORD\n                        Focus keyword in URL\n  --filter-keyword=MSPIDER_FILTER_KEYWORD\n                        Filter keyword in URL\n  --filter-domain=MSPIDER_FILTER_DOMAIN\n                        Filter domain\n  --focus-domain=MSPIDER_FOCUS_DOMAIN\n                        Focus domain\n  --random-agent=MSPIDER_AGENT\n                        Use randomly selected HTTP User-Agent header value\n  --print-all=MSPIDER_PRINT_ALL\n                        Will show more information\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanning23%2FMSpider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanning23%2FMSpider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanning23%2FMSpider/lists"}