{"id":36575638,"url":"https://github.com/saeeddhqan/evine","last_synced_at":"2026-01-12T07:32:18.549Z","repository":{"id":48187534,"uuid":"283785697","full_name":"saeeddhqan/evine","owner":"saeeddhqan","description":"Interactive CLI Web Crawler","archived":false,"fork":false,"pushed_at":"2021-10-15T22:22:39.000Z","size":808,"stargazers_count":175,"open_issues_count":0,"forks_count":32,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-07-30T18:34:53.573Z","etag":null,"topics":["cli","crawler","data-mining","fuzzing","go","golang","osint","scraper","web-crawler"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saeeddhqan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-30T13:37:55.000Z","updated_at":"2024-07-26T03:10:43.000Z","dependencies_parsed_at":"2022-09-04T08:01:47.280Z","dependency_job_id":null,"html_url":"https://github.com/saeeddhqan/evine","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/saeeddhqan/evine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saeeddhqan%2Fevine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saeeddhqan%2Fevine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saeeddhqan%2Fevine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saeeddhqan%2Fevine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saeeddhqan","download_url":"https://codeload.github.com/saeeddhqan/evine/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saeeddhqan%2Fevine/sbom","scorecard":{"id":794712,"data":{"date":"2025-08-11","repo":{"name":"github.com/saeeddhqan/evine","commit":"46051ded7145e7daf429a847ca9249da55798ea6"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.7,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 1/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: GNU General Public License v3.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v.0.1.3 not signed: https://api.github.com/repos/saeeddhqan/evine/releases/41968425","Warn: release artifact v.0.1.2 not signed: https://api.github.com/repos/saeeddhqan/evine/releases/29983618","Warn: release artifact v.0.1.1 not signed: https://api.github.com/repos/saeeddhqan/evine/releases/29606179","Warn: release artifact v.0.1.0 not signed: https://api.github.com/repos/saeeddhqan/evine/releases/29150865","Warn: release artifact v.0.1.3 does not have provenance: https://api.github.com/repos/saeeddhqan/evine/releases/41968425","Warn: release artifact v.0.1.2 does not have provenance: https://api.github.com/repos/saeeddhqan/evine/releases/29983618","Warn: release artifact v.0.1.1 does not have provenance: https://api.github.com/repos/saeeddhqan/evine/releases/29606179","Warn: release artifact v.0.1.0 does not have provenance: https://api.github.com/repos/saeeddhqan/evine/releases/29150865"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-23T08:36:26.622Z","repository_id":48187534,"created_at":"2025-08-23T08:36:26.622Z","updated_at":"2025-08-23T08:36:26.622Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28336537,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","crawler","data-mining","fuzzing","go","golang","osint","scraper","web-crawler"],"created_at":"2026-01-12T07:32:17.876Z","updated_at":"2026-01-12T07:32:18.542Z","avatar_url":"https://github.com/saeeddhqan.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Go Report Card](https://goreportcard.com/badge/github.com/saeeddhqan/evine)](https://goreportcard.com/report/github.com/saeeddhqan/evine)\n[![License](https://img.shields.io/github/license/saeeddhqan/evine?color=%234ac41c)](https://opensource.org/licenses/GPL-3.0)\n[![Build Status](https://travis-ci.com/saeeddhqan/evine.svg?branch=master)](https://travis-ci.com/saeeddhqan/evine)\n# Evine\n\nInteractive CLI Web Crawler.\n\nEvine is a simple, fast, and interactive web crawler and web scraper written in Golang.\nEvine is useful for a wide range of purposes such as metadata and data extraction, data mining, reconnaissance and testing.\n\n[![asciicast](https://asciinema.org/a/351624.svg)](https://asciinema.org/a/351624)\n\nIf you like the project, give it a star. It forces me to develop the project!\n\n\n## Install\n\n### From Binary\nPre-build [binary releases](https://github.com/saeeddhqan/evine/releases) are also available(Suggested).\n### From source\n```\ngo get github.com/saeeddhqan/evine\n\"$GOPATH/bin/evine\" -h\n```\n### From GitHub\n```\ngit clone https://github.com/saeeddhqan/evine.git\ncd evine\ngo build .\nmv evine /usr/local/bin\nevine --help\n```\n\nNote: golang 1.13.x required.\n\n## Commands \u0026 Usage\n\nKeybinding                              | Description\n----------------------------------------|---------------------------------------\n\u003ckbd\u003eEnter\u003c/kbd\u003e                        | Run crawler (from URL view)\n\u003ckbd\u003eEnter\u003c/kbd\u003e                        | Display response (from Keys and Regex views)\n\u003ckbd\u003eTab\u003c/kbd\u003e       \t\t\t\t\t          | Next view\n\u003ckbd\u003eCtrl+Space\u003c/kbd\u003e                   | Run crawler\n\u003ckbd\u003eCtrl+S\u003c/kbd\u003e                       | Save response\n\u003ckbd\u003eCtrl+Z\u003c/kbd\u003e                       | Quit\n\u003ckbd\u003eCtrl+R\u003c/kbd\u003e                       | Restore to default values (from Options and Headers views)\n\u003ckbd\u003eCtrl+Q\u003c/kbd\u003e                       | Close response save view (from Save view)\n\n```bash\nevine -h\n```\nIt will display help for the tool:\n\n| flag | Description | Example |\n|------|-------------|---------|\n| -url | URL to crawl for | evine -url toscrape.com |\n| -url-exclude string | Exclude URLs maching with this regex (default \".*\")  | evine -url-exclude ?id= | \n| -domain-exclude string | Exclude in-scope domains to crawl. Separate with comma. default=root domain | evine -domain-exclude host1.tld,host2.tld | \n| -code-exclude string | Exclude HTTP status code with these codes. Separate whit '\\|' (default \".*\") | evine -code-exclude 200,201 | \n| -delay int  | Sleep between each request(Millisecond) | evine -delay 300 | \n| -depth | Scraper depth search level (default 1) | evine -depth 2 | \n| -thread int | The number of concurrent goroutines for resolving (default 5) | evine -thread 10 |\n| -header | HTTP Header for each request(It should to separated fields by \\n). | evine -header KEY: VALUE\\nKEY1: VALUE1 | \n| -proxy string | Proxy by scheme://ip:port | evine -proxy http://1.1.1.1:8080 | \n| -scheme string | Set the scheme for the requests (default \"https\") | evine -scheme http | \n| -timeout int | Seconds to wait before timing out (default 10) | evine -timeout 15 | \n| -query string | JQuery expression(It could be a file extension(pdf), a key query(url,script,css,..) or a jquery selector($(\"a[class='hdr']).attr('hdr')\"))) | evine -query url,pdf,txt |\n| -regex string | Search the Regular Expression on the page contents | evine -regex 'User.+' |\n| -logger string | Log errors in a file | evine -logger log.txt | \n| -max-regex int | Max result of regex search for regex field (default 1000) | evine -max-regex -1 | \n| -robots | Scrape robots.txt for URLs and using them as seeds | evine -robots |\n| -sitemap | Scrape sitemap.xml for URLs and using them as seeds | evine -sitemap |\n| -wayback | Scrape WayBackURLs(web.archive.org) for URLs and using them as seeds | evine -sitemap |\n\n### VIEWS\n- \u003cb\u003eURL,\u003c/b\u003e In this view, you should enter the URL string.\n- \u003cb\u003eOptions,\u003c/b\u003e This view is for setting options.\n- \u003cb\u003eHeaders,\u003c/b\u003e This view is for setting the HTTP Headers.\n- \u003cb\u003eQuery,\u003c/b\u003e This view is used after the crawling web. \n  It will be used to extract the data(docs, URLs, etc) from the web pages that have been crawled.\n- \u003cb\u003eRegex,\u003c/b\u003e This view is useful to search the Regexes in web pages that have been crawled. Write your Regex in this view and press \u003ckbd\u003eEnter\u003c/kbd\u003e.\n- \u003cb\u003eResponse,\u003c/b\u003e All of the results writes in this view.\n- \u003cb\u003eSearch,\u003c/b\u003e This view is used to search the Regexes in the Response view content.\n\n### Extract methods\n#### From Keys\nKeys are predefined keywords that can be used to specify data like in scope URLs, out scope URLs, emails, etc.\nList of all keys:\n- \u003ci\u003eurl,\u003c/i\u003e to extract IN SCOPE urls. the urls completly are sanitized.\n- \u003ci\u003eemail,\u003c/i\u003e to extract IN SCOPE and out scope emails.\n- \u003ci\u003equery_urls,\u003c/i\u003e to extract IN SCOPE urls that contains the get query: ?foo=bar.\n- \u003ci\u003eall_urls,\u003c/i\u003e to extract OUT SCOPE urls.\n- \u003ci\u003ephone,\u003c/i\u003e to extract a[href]s that contains a phone number.\n- \u003ci\u003emedia,\u003c/i\u003e to extract files that are not web executable file. like .exe,.bat,.tar.xz,.zip, etc addresses.\n- \u003ci\u003ecss,\u003c/i\u003e to extract CSS files.\n- \u003ci\u003escript,\u003c/i\u003e to extract JavaScript files.\n- \u003ci\u003ecdn,\u003c/i\u003e to extract Content Delivery Networks(CDNs) addresses. like //api.foo.bar/jquery.min.js\n- \u003ci\u003ecomment,\u003c/i\u003e to extract html comments, \u003c\\!-- .* !--\u003e\n- \u003ci\u003edns,\u003c/i\u003e to extract subdomains that belongs to the website.\n- \u003ci\u003enetwork,\u003c/i\u003e to extract social network IDs. like facebook, twitter, etc.\n- \u003ci\u003eall,\u003c/i\u003e to extract all list of keys.(url,query_url,..)\nkeys are case-sensitive. Also, it could be written to or three key with comma separation.\n#### From Extensions\nMaybe you wanna a file that is not defined in keys. What can you do? You can easily write the extension of the file on the Query view. like png,xml,txt,docx,xlsx,a,mp3, etc.\n#### From JQuery selector\nIf you have basic JQuery skills, you can easily use this feature, but if not, it is not very difficult. To have a quick view about the selectors [w3schools](https://www.w3schools.com/jquery/jquery_ref_selectors.asp) is a great source.\u003cbr\u003e\nexample(To find source[src]):\n```javascript\n$(\"source\").attr(\"src\") // To find all of source[src] urls\n$(\"h1\").text() // To find h1 values\n```\nTemplate:\n```javascript\n$(\"SELECTOR\").METHOD_NAME(\"arg\")\n```\nIt does not support queries like below:\n```javascript\n$('SELECTOR').METHOD(\"arg\")\n$('SELECTOR').METHOD('arg')\n$(\"SELECTOR\"  ).METHOD(\"arg\" )\n```\nMethods are described below:\n- \u003ci\u003etext(),\u003c/i\u003e to returns the content of the SELECTOR without html tag.\n- \u003ci\u003ehtml(),\u003c/i\u003e to returns the content of the SELECTOR with html tag.\n- \u003ci\u003eattr(\"ATTR\"),\u003c/i\u003e to get the attribute of the SELECTOR. e.g $(\"a\").attr(\"href\")\n\n\n\n\n## Bugs or Suggestions\n\nTo report bugs or suggestions, create an [issue](https://github.com/saeeddhqan/evine/issues).\n\nEvine is heavily inspired by [wuzz](https://github.com/asciimoo/wuzz).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaeeddhqan%2Fevine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaeeddhqan%2Fevine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaeeddhqan%2Fevine/lists"}