{"id":18649816,"url":"https://github.com/ashwanthkumar/monzo-crawler","last_synced_at":"2026-02-06T06:32:21.172Z","repository":{"id":140450852,"uuid":"96664130","full_name":"ashwanthkumar/monzo-crawler","owner":"ashwanthkumar","description":"Solution to problem statement for Monzo's take home assignment","archived":false,"fork":false,"pushed_at":"2017-07-09T06:04:05.000Z","size":200,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-27T21:46:23.934Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashwanthkumar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-09T05:49:08.000Z","updated_at":"2024-07-31T14:43:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"6c3a58d7-af55-4aaa-b667-186ca37bfe0c","html_url":"https://github.com/ashwanthkumar/monzo-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ashwanthkumar/monzo-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fmonzo-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fmonzo-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fmonzo-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fmonzo-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashwanthkumar","download_url":"https://codeload.github.com/ashwanthkumar/monzo-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fmonzo-crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29153863,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T02:39:25.012Z","status":"ssl_error","status_checked_at":"2026-02-06T02:37:22.784Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T06:40:45.241Z","updated_at":"2026-02-06T06:32:21.151Z","avatar_url":"https://github.com/ashwanthkumar.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/ashwanthkumar/monzo-crawler.svg?branch=master)](https://travis-ci.org/ashwanthkumar/monzo-crawler)\n\n# monzo-crawler\n\n## Usage\n```\n$ make setup # uses glide to install all dependencies to vendor/\n$ make build \u0026\u0026 ./monzo-crawler tomblomfield.com | tee sitemap.txt\n$ less sitemap.txt\n```\n\n## Testing\n```\n$ make test\n```\n\n## Known Issues\n- Doesn't have politeness delay implemented. We'll bombard the site with `runtime.NumCPU() * 4` concurrent HTTP requests\n- HTML page parsing is done using [`goquery`](https://github.com/PuerkitoBio/goquery) library which is really slow for very big HTML pages (like that of amazon.com)\n\n---\n\n## Problem Statement\nWe'd like you to write a simple web crawler in a programming language of your choice. Feel free to either choose one you're very familiar with or, if you'd like to learn some Go, you can also make this your first Go program! The crawler should be limited to one domain - so when crawling tomblomfield.com it would crawl all pages within the domain, but not follow external links, for example to the Facebook and Twitter accounts. Given a URL, it should output a site map, showing which static assets each page depends on, and the links between pages.\n\nIdeally, write it as you would a production piece of code. Bonus points for tests and making it as fast as possible!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwanthkumar%2Fmonzo-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashwanthkumar%2Fmonzo-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwanthkumar%2Fmonzo-crawler/lists"}