{"id":21532770,"url":"https://github.com/sfproductlabs/scrp","last_synced_at":"2025-04-10T00:40:49.676Z","repository":{"id":149208627,"uuid":"266217922","full_name":"sfproductlabs/scrp","owner":"sfproductlabs","description":"Horizontally scalable web scraper written in Go","archived":false,"fork":false,"pushed_at":"2020-08-07T02:11:13.000Z","size":242,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-19T00:34:29.839Z","etag":null,"topics":["altavista","baidu","bing","google","yahoo"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sfproductlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-22T22:17:01.000Z","updated_at":"2024-06-19T07:36:10.858Z","dependencies_parsed_at":null,"dependency_job_id":"cd5edd95-a213-4f1c-ae6b-7129d8ae46c1","html_url":"https://github.com/sfproductlabs/scrp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sfproductlabs%2Fscrp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sfproductlabs%2Fscrp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sfproductlabs%2Fscrp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sfproductlabs%2Fscrp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sfproductlabs","download_url":"https://codeload.github.com/sfproductlabs/scrp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137996,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["altavista","baidu","bing","google","yahoo"],"created_at":"2024-11-24T02:22:02.385Z","updated_at":"2025-04-10T00:40:49.661Z","avatar_url":"https://github.com/sfproductlabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scrp\nA fully resumable horizontally (infinitely) scalable webscraper in Go. Think 1000's of machines scraping sites in a distributed way. Based on Docker Swarm, Cassandra, colly, gRPC, and my other [boilerplate](https://github.com/dioptre/gtrpc).\n\n## Note\nWhy are you even here? Maybe you could probably just use colly... Especially if you don't care about scalability... or use a shell script, for example:\n```bash\n#!/bin/bash\nurl=\"http://www.cityfeet.com/cont/api/search/listings-spatial\"\ncookie=\"ASP.NET_SessionId=x335iekckm5tqxcq12psv1p2; __RequestVerificationToken_L2NvbnQ1=FTTyjLMPpvjTLNYvWo5a5yFqhos830-fpyjtxwr4vsVnG8P7_bf5zEEpH4JjY2KfIKgHMuuotd9IyW4iUmSeYRHnLzQ1\"\nDATE=`date +%Y-%m-%d`\n#for i in $(cat query.txt); do\nfor i in {1..9}; do\n    body=\"{'location':{'name':'San Francisco, CA','bb':[37.708131,-122.51777,37.863424,-122.3570311],'lat':37.7857775,'lng':-122.43740055,'state':'CA','city':'San Francisco','id':'3-19282','level':3},'lt':1,'pt':0,'sort':null,'partnerId':null,'lc':[],'mode':2,'portfolio':-1,'tt':0,'ignoreLocation':false,'KeyWord':null,'rent':{'type':1,'basis':0},'term':'San Francisco, CA','PageNum':$i,'PageSize':30,'state':{'\\$type':'Cityfeet.Core.Listing.MultiSearchState, Core','ProviderPosition':{'PDS':$((30 * ($i -1))),'CF':0}}}\"\n    content=\"$(curl -v -s \"$url\" --header \"Cookie: $cookie\" --header \"Content-Type: application/json\" --data \"$body\" --cookie \"$cookie\")\"\n    echo \"$content\" \u003e ./data/city-feet-com-listings-spatial-$DATE-$i.json\n    sleep 5\ndone\n```\n## Why\nI built this to distribute scraping across multiple servers, so as to go undetected. I could have used proxies, but wanted to reuse the code for other distributed apps.\n\n## Local Execution\n\n### Installing (local)\nRun:\n```\ndocker-compose up\n```\nThen (on linux - you can use brew on mac):\n```\n#apt install go\n#./prereq.sh\n#./build.sh\n```\n\n### Scrape Instructions (local)\nAdd backend.local to your /etc/hosts file:\n```\nbash -c 'echo \"127.0.0.1 backend.local\" \u003e\u003e /etc/hosts'\n```\n\nFirst run the server on all the nodes (use the testlocal.sh script for brevity):\n```\n#GOCQL_HOST_LOOKUP_PREFER_V4=true /usr/bin/nice -n 5 ./gsvc localhost false false ./.setup/keys/rootCa.crt ./.setup/keys/cassandra-client.crt ./.setup/keys/cassandra-client.key\n```\nNotice the parameters:\n```\n[0] - cassandra-databases (comma-separated, no spaces)\n[1] - cassandra-retry (should we retry execution on the cassandra cluster)\n[2] - cassandra-veify (should we verify the cassandra service)\n[3] - cassandra-rootca  (only use this if you need)\n[4] - cassandra-client-cert (only use this if you need)\n[5] - cassandra-client-key (only use this if you need)\n```\n\nThen send a request via the client:\n```\n./gcli backend.local:50551 https://en.wikipedia.org/wiki/List_of_HTTP_status_codes\n```\nOr something a little more complex (with domain filter \u0026 regex [note you can split regex into multiple filters using ```||```]):\n```\n./gcli backend.local:50551 https://en.wikipedia.org/wiki/List_of_HTTP_status_codes en.wikipedia.org,wikipedia.org \".*List.*status_codes$\"\n```\nOr without the domain filter and just the regex (use the _ operator to skip):\n```\n./gcli backend.local:50551 https://en.wikipedia.org/wiki/List_of_HTTP_status_codes _ \".*List.*status_codes$\"\n```\n\n## Running on Docker Swarm\n### TL;DR Be your own google bot cluster in 5 minutes.\nThis will set you up with a cluster running 30 machines on [Hetzner Cloud](https://hetzner.cloud/?ref=kVvYlAsUNbOF) (change the first 20 lines to suit your own cloud provider). I use this on my own production servers. Nothing else. This assumes you've setup a project in Hetzner and an API key. It should be a fresh environment. We may delete ALL the machines in Hetzner if things don't work, so start with a fresh project and use the RIGHT API key.\nOn your local/desktop/development machine (from the scrp github repository):\n```\nsudo apt install hcloud-cli\nhcloud ssh-key create --name andy --public-key-from-file ~/.ssh/id_rsa.pub  \nhcloud network create --ip-range=10.1.0.0/16 --name=aftnet\nhcloud network add-subnet --ip-range=10.1.0.0/16 --type=server --network-zone=eu-central aftnet\nfor n in {1..30}; do (hcloud server create --name scrp$RANDOM$RANDOM$RANDOM$RANDOM --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy 2\u003e\u00261 \u003e/dev/null \u0026) ; done\nwatch -n 5 \"echo \"Press Ctrl-c to exit when your server count meets the desired amount. You will need to copy and paste just the following instructions to proceed.\" \u0026\u0026 hcloud server list | grep 'running' | awk 'END {print NR}'\"\n```\nWait until all your servers have been created then continue:\n```\nrm *.txt\nhcloud server list -o columns=name -o noheader \u003e scrps-names.txt\nhcloud server list -o columns=ipv4 -o noheader \u003e scrps-ips.txt\ncat scrps-names.txt | xargs -I {} hcloud server describe -o json {} | jq -r '.private_net[0].ip' \u003e\u003e scrps-vips.txt\nhcloud server create --name cassandra1 --type cx41 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy\nhcloud server describe -o json cassandra1 | jq -r '.private_net[0].ip' \u003e cassandra-vip.txt\nhcloud server create --name manager1 --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy\nhcloud server describe -o json manager1 | jq -r '.private_net[0].ip' \u003e manager-vip.txt\nscp -o StrictHostKeyChecking=no *.txt root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\nscp -o StrictHostKeyChecking=no ansible/* root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\nscp -o StrictHostKeyChecking=no scrp-docker-compose.yml root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\nscp -o StrictHostKeyChecking=no .setup/schema* root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\n```\nIf it stuffs up run **DANGEROUS** it will delete all your servers for the project:\n```\nhcloud server list -o columns=name -o noheader | xargs -P 8 -I {} hcloud server delete {}\n```\nIf not get on the manager node ```ssh -l root -A $(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}')``` and run (copy and paste the whole thing):\n```\napt-get update \u0026\u0026 \\\napt-get upgrade -y \u0026\u0026 \\\napt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y \u0026\u0026 \\\ncurl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - \u0026\u0026 \\\napt-key fingerprint 0EBFCD88 \u0026\u0026 \\\nadd-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable\" \u0026\u0026 \\\napt-get update \u0026\u0026 \\\napt-get install docker-ce docker-ce-cli containerd.io ansible -y \u0026\u0026 \\\ndocker swarm init --advertise-addr=ens10 \u0026\u0026 \\\ndocker swarm join-token worker | xargs | sed -r 's/^.*(docker.*).*$/\\1/' \u003e join.sh \u0026\u0026 \\\nchmod +x join.sh \u0026\u0026 \\\nprintf \"\\n[defaults]\\nhost_key_checking = False\\n\" \u003e\u003e /etc/ansible/ansible.cfg \u0026\u0026 \\\nprintf \"\\n[cassandras]\\n\" \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\ncat cassandra-vip.txt \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\nprintf \"\\n[managers]\\n\" \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\ncat manager-vip.txt \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\nprintf \"\\n[dockers]\\n\" \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\ncat manager-vip.txt \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\ncat scrps-vips.txt \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\ncat cassandra-vip.txt \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\nprintf \"\\n[scrps]\\n\" \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\ncat scrps-vips.txt \u003e\u003e /etc/ansible/hosts \u0026\u0026 \\\nansible dockers -a \"uptime\" \u0026\u0026 \\\nprintf \"\\n            $(cat join.sh | awk '{print $0}')\" \u003e\u003e swarm-init.yml \u0026\u0026 \\\nansible-playbook swarm-init.yml \u0026\u0026 \\\nansible dockers -a \"docker stats --no-stream\" \u0026\u0026 \\\ndocker node ls \u0026\u0026 \\\ndocker node update --label-add cassandra=true cassandra1 \u0026\u0026 \\\ndocker network create -d overlay --attachable forenet --subnet 192.168.9.0/24 \u0026\u0026 \\\nansible-playbook cassandras-init.yml \u0026\u0026 \\\ndocker secret create schema.1.cql schema.1.cql \u0026\u0026 \\\ndocker stack deploy -c scrp-docker-compose.yml scrp\n```\nFinally run a query (see above for more complex examples):\n```\ndocker run -it --net=forenet sfproductlabs/scrp /app/scrp/gcli scrp_scrp:50551 https://httpbin.org/delay/2\n```\n\n### Deploy to a swarm\n*Important: First make sure you deploy the [schema](https://github.com/sfproductlabs/scrp/blob/master/.setup/schema.1.cql) to cassandra somewhere.*\n\nEx. ```cqlsh --ssl -f ./.setup/schema.1.cql ```\n\n\n[Checkout and use the swarm-config example](https://github.com/sfproductlabs/scrp/blob/master/scrp-docker-compose.yml) then on your docker swarm manager:\n```\ndocker stack deploy -c scrp-docker-compose.yml scrp\n```\nThen follow the logs to see if you need to update anything:\n```\ndocker service logs scrp_scrp -f\n```\n\nThen issue a query to the swarm (as above):\n```\ndocker run -it --net=forenet sfproductlabs/scrp /app/scrp/gcli scrp_scrp:50551 https://httpbin.org/delay/2\n```\n#### Deploying swarm on Hetzner\n```\n$ = Your client/development machine, run them from this git repository root\n# = As root on the dockermanager\nd# = As root on the docker swarm drone\n```\n\n##### Setting up Hetzner\n**Remember to run the $ commands from the git repository root**\n\n* Install hetzner cli:\n```\n$sudo apt install hcloud-cli\n```\n* Go to the cloud console and create a project (important! make sure it's a new one, we will be deleting every server in here when we are done)\n* Then click on project-\u003eaccess-\u003eapi tokens-\u003egenerate token\n* Setup access on your local machine to the datacenters/project:\n```\n$hcloud context create scrp\n```\n* Make sure there are no servers here (yet)\n```\n$hcloud server list  \n```\n* Add your local machine to ssh auth\n```\n$hcloud ssh-key create --name andy --public-key-from-file ~/.ssh/id_rsa.pub  \n```\n* Choose a server-type\n```\n$hcloud server-type list\n$hcloud image list\n$hcloud datacenter list\n```\n* Create a network\n```\n$hcloud network create --ip-range=10.1.0.0/16 --name=aftnet\n$hcloud network add-subnet --ip-range=10.1.0.0/16 --type=server --network-zone=eu-central aftnet\n```\n* Create 30 servers (replace the type with your instance preference Ex. cx41)\n```\n$for n in {1..30}; do (hcloud server create --name scrp$RANDOM$RANDOM$RANDOM$RANDOM --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy \u0026) ; done\n```\n* In a SEPARATE terminal see the status of your booting machines (you can delete them all using the command below if something bad happens):\n```\n$watch -n 5 \"echo \"Press Ctrl-c to exit when your server count meets the desired amount\" \u0026\u0026 hcloud server list | grep 'running' | awk 'END {print NR}'\"\n```\n* Get a list of them. IMPORTANT. This will be used to delete the servers later. Check them!\n```\n$rm *.txt\n$hcloud server list -o columns=name -o noheader \u003e scrps-names.txt\n$hcloud server list -o columns=ipv4 -o noheader \u003e scrps-ips.txt\n$cat scrps-names.txt | xargs -I {} hcloud server describe -o json {} | jq -r '.private_net[0].ip' \u003e\u003e scrps-vips.txt\n```\n* Create a cassandra server (16GB ram):\n```\n$hcloud server create --name cassandra1 --type cx41 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy\n$hcloud server describe -o json cassandra1 | jq -r '.private_net[0].ip' \u003e cassandra-vip.txt\n```\n* Create a manager node, copy some files to it and login:\nAddtional step required *only* on a mac:\n```\neval `ssh-agent`\nssh-add ~/.ssh/id_rsa\n```\nNow create a manager, and get to it:\n```\n$hcloud server create --name manager1 --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy\n$hcloud server describe -o json manager1 | jq -r '.private_net[0].ip' \u003e manager-vip.txt\n$scp -o StrictHostKeyChecking=no *.txt root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\n$scp -o StrictHostKeyChecking=no ansible/* root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\n$scp -o StrictHostKeyChecking=no scrp-docker-compose.yml root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\n$scp -o StrictHostKeyChecking=no .setup/schema* root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/\n$ssh -l root -A $(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}')\n```\n##### Initializing a Docker Swarm\nhttps://docs.docker.com/engine/install/debian/\n\nFrom the #docker manager1 (last ssh command above) as root **(it's important to make sure this runs perfectly)** run:\n\n```\napt-get update \u0026\u0026 \\\napt-get upgrade -y \u0026\u0026 \\\napt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y \u0026\u0026 \\\ncurl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - \u0026\u0026 \\\napt-key fingerprint 0EBFCD88 \u0026\u0026 \\\nadd-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable\" \u0026\u0026 \\\napt-get update \u0026\u0026 \\\napt-get install docker-ce docker-ce-cli containerd.io ansible -y \u0026\u0026 \\\ndocker swarm init --advertise-addr=ens10 \u0026\u0026 \\\ndocker swarm join-token worker | xargs | sed -r 's/^.*(docker.*).*$/\\1/' \u003e join.sh \u0026\u0026 \\\nchmod +x join.sh\n```\nNow we can setup docker on all the client machines using ansible (still in the docker manager1):\n```\nprintf \"\\n[defaults]\\nhost_key_checking = False\\n\" \u003e\u003e /etc/ansible/ansible.cfg\n\nprintf \"\\n[cassandras]\\n\" \u003e\u003e /etc/ansible/hosts\ncat cassandra-vip.txt \u003e\u003e /etc/ansible/hosts\n\nprintf \"\\n[managers]\\n\" \u003e\u003e /etc/ansible/hosts\ncat manager-vip.txt \u003e\u003e /etc/ansible/hosts\n\nprintf \"\\n[dockers]\\n\" \u003e\u003e /etc/ansible/hosts\ncat manager-vip.txt \u003e\u003e /etc/ansible/hosts\ncat scrps-vips.txt \u003e\u003e /etc/ansible/hosts\ncat cassandra-vip.txt \u003e\u003e /etc/ansible/hosts\n\nprintf \"\\n[scrps]\\n\" \u003e\u003e /etc/ansible/hosts\ncat scrps-vips.txt \u003e\u003e /etc/ansible/hosts\n```\n\nTest the machines are contactable:\n```\nansible dockers -a \"uptime\"\n```\n\nIf that worked, install docker on all the machines:\n```\nprintf \"\\n            $(cat join.sh | awk '{print $0}')\" \u003e\u003e swarm-init.yml\nansible-playbook swarm-init.yml\n```\n\nTest the dockers are up:\n```\nansible dockers -a \"docker stats --no-stream\"\ndocker node ls\n```\nNow deploy the swarm stack:\n```\ndocker node update --label-add cassandra=true cassandra1\nansible-playbook cassandras-init.yml\ndocker network create -d overlay --attachable forenet --subnet 192.168.9.0/24 \ndocker secret create schema.1.cql schema.1.cql\ndocker stack deploy -c scrp-docker-compose.yml scrp\n```\nGive it a few minutes to boot, the scrps will take a while and likely fail a few times before they finally connect to cassandra, to debug example:\n```\ndocker service ps scrp_cassandra --no-trunc\ndocker service logs scrp_cassandra -f\n```\nFinally run a query (see above for more complex examples):\n```\ndocker run -it --net=forenet sfproductlabs/scrp /app/scrp/gcli scrp_scrp:50551 https://httpbin.org/delay/2\n```\n\n##### Getting my data out\nIf you are new to cassandra heres a quick intro:\n```\n$ssh -l root -A $(hcloud server list -o columns=ipv4,name -o noheader | grep cassandra1 | awk '{print $1}')\n#docker ps\n#docker exec -it c41 bash #replace c41 with your container\n## cqlsh --ssl\ncqlsh\u003e select count(*) from scrp.content;\ncqlsh\u003e COPY scrp.content  TO 'content.csv' WITH HEADER = FALSE;\n```\n\n\n\n##### Deleting Machines\n\n* DELETE THEM. Yes. Let's get used to it, and make sure we know what we're doing. Double check everything before executing these commands.\n```\n$cat scrps-names.txt | xargs -I {} hcloud server delete {}\n```\nor DANGEROUS (but great for cleaning up, will include cassandra), in parallel:\n```\nhcloud server list -o columns=name -o noheader | xargs -P 8 -I {} hcloud server delete {}\n```\n\n##### Misc\nExample commands (https://docs.hetzner.cloud/):\n```\n$source \u003c(hcloud completion bash)   # bash\n$source \u003c(hcloud completion zsh)   # zsh\n$hcloud server list\n$hcloud ssh-key create --name demo --public-key-from-file ~/.ssh/id_rsa.pub         \n$hcloud server create --name demoserver --type cx11 --image debian-9 --ssh-key demo                 \n$hcloud server list             \n$hcloud server list | grep -E \"[0-9]+.[0-9]+.[0-9]+.[0-9]+\" | sed -r 's/.*(\\w[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+).*$/\\1/' \u003e scrps-ips.txt\n$hcloud server list | grep -E '^[^ID]' | sed -r 's/^[0-9]+ +([^ ]+).*$/\\1/ig' \u003e scrps-names.txt\n```\n\n##### Issues\n\nSometimes I get a network issue running a new docker container(out of swarm mode using ```docker run```) in a node in a drained state, try and run this to fix things:\n```\ndocker node update --availability active manager1\n```\nFor example:\n```\ndocker node update --availability active manager1 \u0026\u0026 docker run -it --net=forenet sfproductlabs/scrp /app/scrp/gcli scrp_scrp:50551 https://www.bf.com/?country=us bf.com,www.bf.com\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsfproductlabs%2Fscrp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsfproductlabs%2Fscrp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsfproductlabs%2Fscrp/lists"}