{"id":30607009,"url":"https://github.com/exca-dk/webscraper","last_synced_at":"2025-08-30T03:25:58.828Z","repository":{"id":199796331,"uuid":"702645354","full_name":"Exca-DK/webscraper","owner":"Exca-DK","description":"simple 0 dependency go-webscrapper example","archived":false,"fork":false,"pushed_at":"2023-10-12T08:42:22.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-25T07:38:56.274Z","etag":null,"topics":["example","golang","webscraper","webscrapper"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Exca-DK.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-09T17:59:26.000Z","updated_at":"2023-11-28T20:10:23.000Z","dependencies_parsed_at":"2023-10-12T13:44:57.648Z","dependency_job_id":null,"html_url":"https://github.com/Exca-DK/webscraper","commit_stats":null,"previous_names":["exca-dk/webscraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Exca-DK/webscraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exca-DK%2Fwebscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exca-DK%2Fwebscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exca-DK%2Fwebscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exca-DK%2Fwebscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Exca-DK","download_url":"https://codeload.github.com/Exca-DK/webscraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exca-DK%2Fwebscraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272799945,"owners_count":24995100,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["example","golang","webscraper","webscrapper"],"created_at":"2025-08-30T03:25:57.300Z","updated_at":"2025-08-30T03:25:58.820Z","avatar_url":"https://github.com/Exca-DK.png","language":"Go","readme":"# WEBSCRAPER\n\nThis is a simple example of a web scraper implemented using Golang, with zero external dependencies. This repository serves as a basic demonstration of how to create a web scraper to extract web content using the Golang standard library.\n\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Getting Started](#getting-started)\n- [Features](#features)\n\n## Overview\n\nA web scraper is a program that automatically extracts information from provided web pages. This example demonstrates how to create a basic web scraper using the Golang standard library.\n\nWeb scrapers are commonly used in tandem with web crawlers to feed the scraper with additional links, enabling the extraction of valuable data from a broader range of web sources.\n\n## Getting Started\n\nTo get started with this example, follow these steps:\n\n1. Clone this repository to your local machine:\n\n   ```bash\n   git clone https://github.com/Exca-DK/webscraper.git\n   ```\n\n2. Navigate to the project directory:\n\n    ```bash\n    cd webscraper\n    ```\n\n3. Explore and run:\n\n    To run the web scraper help, execute the following command:\n\n    ```bash\n    go run main.go --help\n    ```\n\n    example configurations:\n    ```bash\n    go run main.go --urls=URL1,URL2\n    go run main.go --urls=URL1,URL2 --threads=32\n    go run main.go --urls=URL1,URL2 --threads=32 --verbosity=INFO\n    ```\n\n## Features\n\nThis example provides the following features:\n\n- Efficient page content downloading.\n- An adaptable cache system, which, by default, restricts revisiting websites for a specified lifetime, but can be configured to evict outdated entries.\n- A built-in thread pool for managing and limiting concurrent tasks.\n- A modular and extensible design for in-depth analysis of page content.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexca-dk%2Fwebscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexca-dk%2Fwebscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexca-dk%2Fwebscraper/lists"}