{"id":21831050,"url":"https://github.com/make-school-labs/makescraper","last_synced_at":"2026-05-19T03:52:17.059Z","repository":{"id":57521701,"uuid":"238578970","full_name":"Make-School-Labs/makescraper","owner":"Make-School-Labs","description":"🕷Create your very own web scraper and crawler using Golang!","archived":false,"fork":false,"pushed_at":"2021-02-09T21:34:50.000Z","size":11,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-01-26T09:15:03.879Z","etag":null,"topics":["bew2-5","go","golang","makeschool","webcrawling","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Make-School-Labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-06T00:50:54.000Z","updated_at":"2022-05-28T01:46:23.000Z","dependencies_parsed_at":"2022-09-26T18:10:34.988Z","dependency_job_id":null,"html_url":"https://github.com/Make-School-Labs/makescraper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Make-School-Labs%2Fmakescraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Make-School-Labs%2Fmakescraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Make-School-Labs%2Fmakescraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Make-School-Labs%2Fmakescraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Make-School-Labs","download_url":"https://codeload.github.com/Make-School-Labs/makescraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244803364,"owners_count":20512897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bew2-5","go","golang","makeschool","webcrawling","webscraping"],"created_at":"2024-11-27T19:08:13.678Z","updated_at":"2026-05-19T03:52:17.020Z","avatar_url":"https://github.com/Make-School-Labs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🕷 makescraper\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/YOUR_GITHUB_USERNAME/makescraper)](https://goreportcard.com/report/github.com/YOUR_GITHUB_USERNAME/makescraper)\n\n_Create your very own web scraper and crawler using Go and [Colly](https://go-colly.org)!_\n\n### 📚 Table of Contents\n\n1. [Project Structure](#project-structure)\n2. [Getting Started](#getting-started)\n3. [Deliverables](#deliverables)\n4. [Resources](#resources)\n\n## Project Structure\n\n```bash\n📂 makescraper\n├── README.md\n└── scrape.go\n```\n\n## Getting Started\n\n1. Visit [github.com/new](https://github.com/new) and create a new repository named `makescraper`.\n2. Run each command line-by-line in your terminal to set up the project:\n\n    ```bash\n    $ git clone git@github.com:Make-School-Labs/makescraper.git\n    $ cd makescraper\n    $ git remote rm origin\n    $ git remote add origin git@github.com:YOUR_GITHUB_USERNAME/makescraper.git\n    $ go mod download\n    ```\n\n3. Open `README.md` in your editor and replace all instances of `YOUR_GITHUB_USERNAME` with your GitHub username to enable the Go Report Card badge.\n\n## Deliverables\n\n_Complete each task in the order they appear. Use [GitHub Task List](https://help.github.com/en/github/managing-your-work-on-github/about-task-lists) syntax to update the task list._\n\n### Requirements\n\n#### Scraping\n\n- [ ] **IMPORTANT**: Complete the Web Scraper Workflow worksheet distributed in class.\n- [ ] Create a `struct` to store your data.\n- [ ] Refactor the `c.OnHTML` callback on line `16` to use the selector(s) you tested while completing the worksheet.\n- [ ] Print the data you scraped to `stdout`.\n\n##### Stretch Challenges\n\n- [ ] Add more fields to your `struct`. Extract multiple data points from the website. Print them to `stdout` in a readable format.\n\n#### Serializing \u0026 Saving\n\n- [ ] Serialize the `struct` you created to JSON. Print the JSON to `stdout` to validate it.\n- [ ] Write scraped data to a file named `output.json`.\n- [ ] **Add, commit, and push to GitHub**.\n\n## Resources\n\n### Lesson Plans\n\n- [**BEW 2.5** - Scraping the Web](https://make-school-courses.github.io/BEW-2.5-Strongly-Typed-Languages/#/Lessons/WebScraping.md): Concepts and examples covered in class related to web scraping and crawling.\n\n### Example Code\n\n#### Scraping\n\n- [**Colly** - Docs](http://go-colly.org/docs/): Check out the sidebar for 20+ examples!\n- [**Ali Shalabi** - Syntax-Helper](https://github.com/alishalabi/syntax-helper): Command line interface to help generate proper code syntax, pulled from the Golang documentation.\n\n#### Serializing \u0026 Saving\n\n- [JSON to Struct](https://mholt.github.io/json-to-go/): Paste any JSON data and convert it into a Go structure that will support storing that data.\n- [GoByExample - JSON](https://gobyexample.com/json): Covers Go's built-in support for JSON encoding and decoding to and from built-in and custom data types (structs).\n- [GoByExample - Writing Files](https://gobyexample.com/writing-files): Covers creating new files and writing to them.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmake-school-labs%2Fmakescraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmake-school-labs%2Fmakescraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmake-school-labs%2Fmakescraper/lists"}