https://github.com/ashwanthkumar/monzo-crawler
Solution to problem statement for Monzo's take home assignment
https://github.com/ashwanthkumar/monzo-crawler
Last synced: 5 months ago
JSON representation
Solution to problem statement for Monzo's take home assignment
- Host: GitHub
- URL: https://github.com/ashwanthkumar/monzo-crawler
- Owner: ashwanthkumar
- License: apache-2.0
- Created: 2017-07-09T05:49:08.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2017-07-09T06:04:05.000Z (almost 9 years ago)
- Last Synced: 2025-07-27T21:46:23.934Z (11 months ago)
- Language: Go
- Size: 195 KB
- Stars: 2
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://travis-ci.org/ashwanthkumar/monzo-crawler)
# monzo-crawler
## Usage
```
$ make setup # uses glide to install all dependencies to vendor/
$ make build && ./monzo-crawler tomblomfield.com | tee sitemap.txt
$ less sitemap.txt
```
## Testing
```
$ make test
```
## Known Issues
- Doesn't have politeness delay implemented. We'll bombard the site with `runtime.NumCPU() * 4` concurrent HTTP requests
- HTML page parsing is done using [`goquery`](https://github.com/PuerkitoBio/goquery) library which is really slow for very big HTML pages (like that of amazon.com)
---
## Problem Statement
We'd like you to write a simple web crawler in a programming language of your choice. Feel free to either choose one you're very familiar with or, if you'd like to learn some Go, you can also make this your first Go program! The crawler should be limited to one domain - so when crawling tomblomfield.com it would crawl all pages within the domain, but not follow external links, for example to the Facebook and Twitter accounts. Given a URL, it should output a site map, showing which static assets each page depends on, and the links between pages.
Ideally, write it as you would a production piece of code. Bonus points for tests and making it as fast as possible!