{"id":17283710,"url":"https://github.com/prettymuchbryce/rugburn","last_synced_at":"2025-03-26T16:16:14.348Z","repository":{"id":57589473,"uuid":"104585953","full_name":"prettymuchbryce/rugburn","owner":"prettymuchbryce","description":"A fast configuration-based caching web scraper written in Go","archived":false,"fork":false,"pushed_at":"2020-03-06T15:16:38.000Z","size":44,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-17T16:55:46.912Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prettymuchbryce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-23T17:28:16.000Z","updated_at":"2020-03-06T15:16:41.000Z","dependencies_parsed_at":"2022-08-29T23:20:45.546Z","dependency_job_id":null,"html_url":"https://github.com/prettymuchbryce/rugburn","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Frugburn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Frugburn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Frugburn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prettymuchbryce%2Frugburn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prettymuchbryce","download_url":"https://codeload.github.com/prettymuchbryce/rugburn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245689509,"owners_count":20656417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T09:52:01.081Z","updated_at":"2025-03-26T16:16:14.325Z","avatar_url":"https://github.com/prettymuchbryce.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rugburn\n[![Go Report Card](https://goreportcard.com/badge/github.com/prettymuchbryce/rugburn)](https://goreportcard.com/report/github.com/prettymuchbryce/rugburn)\n[![Build Status](https://travis-ci.org/prettymuchbryce/rugburn.svg?branch=master)](https://travis-ci.org/prettymuchbryce/rugburn)\n[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/prettymuchbryce/rugburn/master/LICENSE)\n\n\u003e A performant configuration-based caching web scraper\n\n## What is rugburn?\n\nRugburn is a web scraping framework. Unlike other web scraping frameworks which require writing\ncode, rugburn attempts to be a completely configuration-based web scraper.\n\n`rugburn` is installed as a CLI tool and allows you to create and modify a \"scraping\" environment\nusing only configuration. To see an example of a scraper for the website `Hacker News`, simply run\n`rugburn init` in a new directory. To see all available options run `rugburn help`.\n\nRugburn configuration files specify which pages to download (spider) and which elements to extract\nvia XPath (scrapers). For cases where additional custom behavior is required, rugburn supports\ntransformations. Transformations are scripts written in LUA which allow \"transformation\" of your \ndata into a more desirable format. You can re-use transformations between scrapers.\n\nRugburn supports caching of requests and responses into a local on-disk database. This is\nrecommended in order to avoid IP bans, improve performance, and in order to preserve the backwards\ncompatability of selectors.\n\nCaching also means you can check your configuration alongside your database into version control.\nThis means your configuration and transforms will remain deterministic and the history of them\nwill be retained.\n\nCaching is optional in the case where it is not desired, or infeasible due to a larger\ndataset.\n\n## Status\n\nRugburn is still experimental and likely to contain breaking changes going forward.\n\n## Installation\n\nMac OS X release binaries are available on the releases page. Other platforms coming soon.\n\nWindows and Linux can still manually build and install:\n\n```\ngo get github.com/prettymuchbryce/rugburn\ncd $GOPATH/src/github.com/prettymuchbryce/rugburn\nmake deps\nmake install\n\n# Now make sure it installed successfully\nrugburn help\n```\n\n## CLI Commands\n\n`rugburn init` - Initialize a new rugburn project in the current directory.\n\n`rugburn run` - Run the rugburn project in this directory.\n\n`rugburn clean` - Clean the rugburn cache of the project in this directory.\n\n`rugburn help` - Print some help information.\n\n## Configuration options\n\nConfiguration Example:\n```json\n{\n\t\"name\": \"Hacker News Scraper\",\n\t\"options\": {\n\t\t\"store\": {\n\t\t\t\"strategy\": \"disk\"\n\t\t},\n\t\t\"spiders\": {\n\t\t\t\"concurrency\": 3,\n\t\t\t\"max\": 5\n\t\t}\n\t},\n\t\"spider\": {\n\t\t\"urls\": [\n\t\t\t\"https://news.ycombinator.com/news\"\n\t\t],\n\t\t\"links\": [\n\t\t\t\"//a[@class=\\\"morelink\\\"]/@href\"\n\t\t]\n\t},\n\t\"scrapers\": [\n\t\t{\n\t\t\t\"name\": \"Links\",\n\t\t\t\"output\": \"links.jsonl\",\n\t\t\t\"context\": \"//a[@class=\\\"storylink\\\"]\",\n\t\t\t\"fields\": {\n\t\t\t\t\"title\": \"/text()\"\n\t\t\t},\n\t\t\t\"transforms\": [\n\t\t\t  \"./transforms/UppercaseTitle.lua\"\n\t\t\t]\n\t\t}\n\t]\n}\n```\n\n## Transform Example\n\n```lua\nfunction transform (state)\n\t-- Uppercase the title field\n\tif state[\"title\"] ~= nil then\n\t\tstate[\"title\"] = string.upper(state[\"title\"])\n\tend\n\treturn state\nend\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprettymuchbryce%2Frugburn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprettymuchbryce%2Frugburn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprettymuchbryce%2Frugburn/lists"}