{"id":36513293,"url":"https://github.com/hunterhug/marmot","last_synced_at":"2026-01-21T21:17:53.296Z","repository":{"id":57487033,"uuid":"79702456","full_name":"hunterhug/marmot","owner":"hunterhug","description":"💐Marmot A Golang HTTP Download","archived":false,"fork":false,"pushed_at":"2023-03-07T02:36:02.000Z","size":10945,"stargazers_count":27,"open_issues_count":1,"forks_count":14,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-01-15T06:56:11.531Z","etag":null,"topics":["crawler","gohttp","gospider","marmot","spider"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hunterhug.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-22T07:43:29.000Z","updated_at":"2025-07-26T07:32:01.000Z","dependencies_parsed_at":"2024-06-18T19:58:42.347Z","dependency_job_id":"283b9b81-f34f-4c07-bca8-72b701aa9ce9","html_url":"https://github.com/hunterhug/marmot","commit_stats":null,"previous_names":["sillybobo/marmot","hunterhug/gospider"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/hunterhug/marmot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hunterhug%2Fmarmot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hunterhug%2Fmarmot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hunterhug%2Fmarmot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hunterhug%2Fmarmot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hunterhug","download_url":"https://codeload.github.com/hunterhug/marmot/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hunterhug%2Fmarmot/sbom","scorecard":{"id":473586,"data":{"date":"2025-08-11","repo":{"name":"github.com/hunterhug/marmot","commit":"cea47e15666542faca08210eea6a83e0f408791c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"11 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0288","Warn: Project is vulnerable to: GO-2022-0969 / GHSA-69cg-p879-7622","Warn: Project is vulnerable to: GO-2022-1144 / GHSA-xrjj-mj9h-534m","Warn: Project is vulnerable to: GO-2023-1571 / GHSA-vvpx-j8f3-3w6h","Warn: Project is vulnerable to: GO-2023-1988 / GHSA-2wrh-6pvc-2jm9","Warn: Project is vulnerable to: GO-2023-2102 / GHSA-4374-p667-p6c8","Warn: Project is vulnerable to: GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2687 / GHSA-4v7x-pqxf-cx7m","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-19T14:38:58.310Z","repository_id":57487033,"created_at":"2025-08-19T14:38:58.310Z","updated_at":"2025-08-19T14:38:58.310Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28643315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-21T18:04:35.752Z","status":"ssl_error","status_checked_at":"2026-01-21T18:03:55.054Z","response_time":86,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","gohttp","gospider","marmot","spider"],"created_at":"2026-01-12T02:38:35.833Z","updated_at":"2026-01-21T21:17:53.291Z","avatar_url":"https://github.com/hunterhug.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project: Marmot | HTTP Download\n\n[![GitHub forks](https://img.shields.io/github/forks/hunterhug/marmot.svg?style=social\u0026label=Forks)](https://github.com/hunterhug/marmot/network)\n[![GitHub stars](https://img.shields.io/github/stars/hunterhug/marmot.svg?style=social\u0026label=Stars)](https://github.com/hunterhug/marmot/stargazers)\n[![GitHub last commit](https://img.shields.io/github/last-commit/hunterhug/marmot.svg)](https://github.com/hunterhug/marmot)\n[![Go Report Card](https://goreportcard.com/badge/github.com/hunterhug/marmot)](https://goreportcard.com/report/github.com/hunterhug/marmot)\n[![GitHub issues](https://img.shields.io/github/issues/hunterhug/marmot.svg)](https://github.com/hunterhug/marmot/issues)\n[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)\n\n[中文介绍](/README_ZH.md)\n\nHTTP Download Helper, Supports Many Features such as Cookie Persistence, HTTP(S) and SOCKS5 Proxy....\n\n![Marmot](logo.png)\n\n## 1. Introduction\n\nWorld-Wide-Web robot, also known as spiders and crawlers. The principle is to falsify network data by constructing appointed HTTP protocol data packet, then request resource to the specified host, goal is to access the data returned. \nThere are a large number of web information, human's hand movement such as `copy-paste data` from web page is `time-consuming` and `laborious`, thus inspired the data acquisition industry.\n\nBatch access to public network data does not break the law, but because there is no difference, no control, very violent means will lead to other services is not stable, therefore, most of the resources provider will filtering some data packets(falsify), \nin this context,  batch small data acquisition has become a problem. Integrated with various requirements, such as various API development, automated software testing(all this have similar technical principle). So this project come into the world(very simple).\n\nThe `Marmot` is very easy to understand, just like Python's library `requests`(Not yet Smile~ --| ). By enhancing native Golang HTTP library, help you deal with some trivial logic (such as collecting information, checking parameters), and add some fault-tolerant mechanisms (such as add lock, close time flow, ensure the high concurrent run without accident).\nIt provides a human friendly API interface, you can reuse it often. Very convenient to support `Cookie Persistence`, `Crawler Proxy Settings`, as well as others general settings, such as  `HTTP request header settings, timeout/pause settings, data upload/post settings`.\nIt supports all the HTTP methods `POST/PUT/GET/DELETE/...` and has built-in spider pool and browser UA pool, easy to develop UA+Cookie persistence distributed spider.\n\nThe library is simple and practical, just a few lines of code to replace the previous `Spaghetti code`, has been applied in some large projects.\n\nThe main uses: `WeChat development`/ `API docking` / `Automated test` / `Rush Ticket Scripting` / `Vote Plug-in` / `Data Crawling`\n\nNow We support Default Worker, You can easy use:\n\n`lesson1.go`\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/hunterhug/marmot/miner\"\n)\n\nfunc main() {\n\tminer.SetLogLevel(miner.DEBUG)\n\n\t// Use Default Worker, You can Also New One:\n\t//worker, _ := miner.New(nil)\n\t//worker = miner.NewWorkerWithNoProxy()\n\t//worker = miner.NewAPI()\n\t//worker, _ = miner.NewWorkerWithProxy(\"socks5://127.0.0.1:1080\")\n\tworker := miner.Clone()\n\t_, err := worker.SetUrl(\"https://www.bing.com\").Go()\n\tif err != nil {\n\t\tfmt.Println(err.Error())\n\t} else {\n\t\tfmt.Println(worker.ToString())\n\t}\n}\n```\n\nSee the [example](example) dir. such as lesson or practice.\n\n## 2. How To Use\n\nYou can get it by:\n\n```\ngo get -v github.com/hunterhug/marmot/miner\n```\n\n### 2.1 The First Step\n\nThere are four kinds of worker:\n\n1. `worker, err := miner.NewWorker(\"http://xx:xxxx@103.118.111.26:808\") ` // proxy worker, format: `protocol://user(optional):password(optional)@ip:port`, alias to`New()`, support http(s), socks5\n2. `worker, err := miner.NewWorker(nil)`  // normal worker, default keep Cookie, alias to `New()`\n3. `worker := miner.NewAPI()` // API worker, will not keep Cookie\n4. `worker, err := miner.NewWorkerByClient(\u0026http.Client{})` // You can also pass a `http.Client` if you want\n\nif you want to use worker twice, you can call `Clone()` method to clone a new worker, it can isolate the request and response of http, otherwise, you should deal concurrent program carefully.\n\n### 2.2 The Second Step\n\nCamouflage our worker:\n\n1. `worker.SetUrl(\"https://www.bing.com\")`  // required: set url you want to\n2. `worker.SetMethod(miner.GET)`  // optional: set http method `POST/GET/PUT/POSTJSON` and so on\n3. `worker.SetWaitTime(2)`                         // optional: set timeout of http request\n4. `worker.SetUa(miner.RandomUa())`                 // optional: set http browser user agent, you can see miner/config/ua.txt\n5. `worker.SetRefer(\"https://www.bing.com\")`       // optional: set http request Refer\n6. `worker.SetHeaderParam(\"diyheader\", \"diy\")` // optional: set http diy header\n7. `worker.SetBData([]byte(\"file data\"))` // optional: set binary data for post or put\n8. `worker.SetFormParam(\"username\",\"hunterhug\")` // optional: set form data for post or put \n9. `worker.SetCookie(\"xx=dddd\")` // optional: you can set a init cookie, some website you can login and F12 copy the cookie\n10. `worker.SetCookieByFile(\"/root/cookie.txt\")` // optional: set cookie which store in a file\n\n### 2.3 The Third Step\n\nRun our worker:\n\n1. `body, err := worker.Go()` // if you use SetMethod(), auto use following ways, otherwise use Get()\n2. `body, err := worker.Get()` // default\n3. `body, err := worker.Post()` // post form request, data fill by SetFormParam()\n4. `body, err := worker.PostJSON()` // post JSON request, data fill by SetBData()\n5. `body, err := worker.PostXML()` // post XML request, data fill by SetBData()\n6. `body, err := worker.PostFILE()` // upload file, data fill by SetBData(), and should set SetFileInfo(fileName, fileFormName string)\n7. `body, err := worker.Delete()` // you know!\n8. `body, err := worker.Put()` // ones http method...\n9. `body, err := worker.PutJSON()` // put JSON request\n10. `body, err := worker.PutXML()`\n11. `body, err := worker.PutFILE()`\n12. `body, err := worker.OtherGo(\"OPTIONS\", \"application/x-www-form-urlencoded\")` // Other http method, Such as OPTIONS etc., can not sent binary.\n13. `body, err := worker.OtherGoBinary(\"OPTIONS\", \"application/x-www-form-urlencoded\")` // Other http method, Such as OPTIONS etc., just sent binary.\n14. `body, err := worker.GoByMethod(\"POST\")` // you can override SetMethod() By this, equal SetMethod() then Go()\n\n### 2.4 The Fourth Step\n\nDeal the return data, all data will be return as binary, You can immediately store it into a new variable:\n\n1. `fmt.Println(string(html))` // type change directly\n2. `fmt.Println(worker.ToString())` // use spider method, after http response, data will keep in the field `Raw`, just use ToString\n3. `fmt.Println(worker.JsonToString())` // some json data will include chinese and other multibyte character, such as `我爱你,我的小绵羊`,`사랑해`\n\nAttention: \n\nAfter every request for an url, the next request you should cover your http request header, otherwise http header you set still exist,\nif just want clear post data, use `Clear()`, and want clear HTTP header too please use `ClearAll()`, but I suggest use `Clone()` to avoid this.\n\n### 2.5 Other\n\nHook:\n\n1. `SetBeforeAction(fc func(context.Context, *Worker))`\n2. `SetAfterAction(fc func(context.Context, *Worker))`\n\n# License\n\n```\nCopyright [2016-2022] [github.com/hunterhug]\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    https://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhunterhug%2Fmarmot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhunterhug%2Fmarmot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhunterhug%2Fmarmot/lists"}