{"id":18078247,"url":"https://github.com/imthaghost/crawler","last_synced_at":"2025-09-15T18:50:31.924Z","repository":{"id":48459878,"uuid":"389204095","full_name":"imthaghost/crawler","owner":"imthaghost","description":"Practice","archived":false,"fork":false,"pushed_at":"2021-09-26T23:05:45.000Z","size":339,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-13T19:47:29.207Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imthaghost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-24T21:21:25.000Z","updated_at":"2022-04-01T10:59:33.000Z","dependencies_parsed_at":"2022-09-13T12:01:16.653Z","dependency_job_id":null,"html_url":"https://github.com/imthaghost/crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/imthaghost/crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imthaghost%2Fcrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imthaghost%2Fcrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imthaghost%2Fcrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imthaghost%2Fcrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imthaghost","download_url":"https://codeload.github.com/imthaghost/crawler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imthaghost%2Fcrawler/sbom","scorecard":{"id":486510,"data":{"date":"2025-08-11","repo":{"name":"github.com/imthaghost/crawler","commit":"e29dbefe223ebf39fe2a5d6ac2e4a4663230119c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/22 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: dev.dockerfile:2: pin your Docker image by updating golang:1.16.4 to golang:1.16.4@sha256:8a106c4b4005efb43c0ba4bb5763b84742c7e222bad5a8dff73cc9f7710c64ee","Warn: downloadThenRun not pinned by hash: dev.dockerfile:5","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   1 downloadThenRun dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 9 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-19T17:57:36.344Z","repository_id":48459878,"created_at":"2025-08-19T17:57:36.344Z","updated_at":"2025-08-19T17:57:36.344Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275304908,"owners_count":25441168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-15T02:00:09.272Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-31T12:13:16.935Z","updated_at":"2025-09-15T18:50:31.888Z","avatar_url":"https://github.com/imthaghost.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003eCrawler Architecture\u003c/p\u003e\n\n\u003cbr\u003e\n\n\u003cp align=\"center\"\u003e\u003cimg alt=\"topology\" src=\"docs/topology.png\"\u003e\u003c/p\u003e\n\n\u003ca name=\"Component Design\"\u003e\u003c/a\u003e\n\u003ch2 align=\"center\"\u003eComponent Design\u003c/h2\u003e\n\n- [Seed URL](https://github.com/imthaghost/crawler/tree/main/pkg/seed) A web crawler starts with a URL to visit, called the _seeds_.  Remove an absolute URL from the shared URL frontier for downloading.\n\n\n- [Crawl Frontier](https://github.com/imthaghost/crawler/tree/main/pkg/frontier) **URL Frontier** is a data structure that contains all the URLs that are to be downloaded to crawl. Calling the crawl frontier will return the next URL that we need to be crawled.\n\n\n- [URL Fetcher](https://github.com/imthaghost/crawler/tree/main/pkg/fetcher) **URL Fetcher** purpose is given a URL from URL Frontier identify the DNS and later fetch the ip and pass it to the URL Renderer. Here we can also validate against the robot.txt to make certain parts of the websites are not crawled as per the robots.txt.\n\n\n- [Custom DNS Server](https://github.com/imthaghost/crawler/tree/main/pkg/dns) DNS name resolution will be a big bottleneck of our crawlers given the amount of URLs we will be working with. To avoid repeated requests, we can start caching DNS results by building our local DNS server.\n\n\n- [URL Renderer](https://github.com/imthaghost/crawler/tree/main/pkg/renderer) **URL Renderer** download’s the web document corresponding to a given URL using the appropriate network protocol like HTTP (Here we can discuss on if we have to use any other protocol like FTP). Here instead of downloading a document multiple times, we Cache the document locally.\n\n\n- [Content Extractor](https://github.com/imthaghost/crawler/tree/main/pkg/extractor) We can use an input stream reader that reads the entire content of the document which was downloaded from the URL Renderer and also can provide a method to re-read the same document if we pass the same url. The Content Extractor extracts the content and passes the content (Rabbit MQ or Kafka) to Content Filter Detection and URL Filter Detection\n\n\n- [Content Filter Detection](https://github.com/imthaghost/crawler/pkg/fetcher) Content Filter detects duplicate content and also filters sensitive content which we do not want to store in our data store. Some potential algorithms are Jaccard index and cosine similarity. Here we are only filtering text content from the document.\n\n\n- [URL Filter Detection](https://github.com/imthaghost/crawler/pkg/fetcher) Similar to Content Filtering URL filter is filtering the URL’s from the given document. We need to be careful the web crawler doesn’t get stuck in an infinite loop, which happens when the graph contains a cycle. This can also be used to blacklist URL’s. We can pass these filtered URL to URL Crawler which intern passes the URL to URL Frontier for next run. \n\n\n- [Reverse Index Search](https://github.com/imthaghost/crawler/pkg/fetcher) Based on the document content. Reverse Index Search will remove markup breaks up the text into multiple terms. It also normalizes the content and converts the query to use boolean operators. Once we have the query it can be used to retrieve the page and snippet given a text.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimthaghost%2Fcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimthaghost%2Fcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimthaghost%2Fcrawler/lists"}