{"id":21310311,"url":"https://github.com/wiringbits/simple-http-proxy","last_synced_at":"2025-07-11T23:31:25.106Z","repository":{"id":99290663,"uuid":"270460266","full_name":"wiringbits/simple-http-proxy","owner":"wiringbits","description":"A very simple http proxy that runs on a Raspberry Pi","archived":false,"fork":false,"pushed_at":"2020-06-08T04:38:42.000Z","size":6,"stargazers_count":25,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-06T10:43:23.306Z","etag":null,"topics":["http-proxy","playframework","raspberry-pi","scala","web-scraping"],"latest_commit_sha":null,"homepage":"https://wiringbits.net","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wiringbits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-07T23:20:51.000Z","updated_at":"2024-06-10T08:39:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"fd4c0901-38b0-45cf-a809-6e1ed1c62811","html_url":"https://github.com/wiringbits/simple-http-proxy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wiringbits/simple-http-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiringbits%2Fsimple-http-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiringbits%2Fsimple-http-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiringbits%2Fsimple-http-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiringbits%2Fsimple-http-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wiringbits","download_url":"https://codeload.github.com/wiringbits/simple-http-proxy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wiringbits%2Fsimple-http-proxy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264913468,"owners_count":23682657,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http-proxy","playframework","raspberry-pi","scala","web-scraping"],"created_at":"2024-11-21T17:13:02.291Z","updated_at":"2025-07-11T23:31:25.083Z","avatar_url":"https://github.com/wiringbits.png","language":"Scala","readme":"# simple-proxy\nThe purpose for this simple-proxy is to keep it running on a residential server so that we can access servers blocking known cloud IPs (like AWS).\n\nThis has been running on an old Raspberry Pi model B for a while, find the deployment instructions to do so below.\n\nSurprisingly, 256 MB in RAM have been enough to maintain the app running for months until now.\n\n\n## Why\nWhen scrapping websites, you may find that everything just works until you deploy your scrapper to the cloud, a common reason is that some websites tend to have bot-detectors which usually blacklist any IP belonging to common cloud providers (like AWS, DigitalOcean, etc).\n\nThis project is a simple workaround to allow your scrapping server to scrape the problematic websites.\n\nThere is a dedicated [post](https://wiringbits.net/wiringbits/2020/06/07/a-raspberry-pi-as-a-decent-residential-proxy.html) with more details.\n\n\n## Run\nTo run locally, [sbt](https://www.scala-sbt.org/) is required, once installed, just run `sbt run`, and start sending requests to `localhost:9000`, for example:\n\n```shell script\ncurl -X POST \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\": \"https://wiringbits.net\", \"headers\": { \"DNT\": \"1\" }}' \\\n   http://localhost:9000\n```\n\nThis request asks the proxy to issue a `GET` request to `https://wiringbits.net`, by sending the custom header `DNT: 1`.\n\n\n## Deploy\nDeploying the app requires three steps:\n- Install the dependencies (Java 8).\n- The actual app.\n- A ssh tunnel to expose the app to the remote server where you will actually use it.\n\nThe simplified step list is:\n- Build the app: `sbt dist`\n- Copy the app to the residential server: `scp target/universal/simple-proxy-0.1.0-SNAPSHOT.zip pi@raspberrypi.local:~/`\n- ssh into the server, then, run `unzip simple-proxy-0.1.0-SNAPSHOT.zip` and `cd simple-proxy-0.1.0-SNAPSHOT/ \u0026\u0026 ./bin/simple-proxy`\n- ssh into the server, then, tunnel to our real server: `ssh -nNT -R 9999:localhost:9000 ubuntu@cazadescuentos.net`\n\n\n### Deploy with systemd\n\n#### simple-proxy\nOn the server, save it as `/etc/systemd/system/simple-proxy.service`:\n\n```\n[Unit]\nDescription=The simple-proxy\nAfter=network.target\n\n[Service]\nExecStart=/home/pi/simple-proxy-0.1.0-SNAPSHOT/bin/simple-proxy -Dhttp.port=9000 -Dpidfile.path=/dev/null\nWorkingDirectory=/home/pi/simple-proxy-0.1.0-SNAPSHOT\nStandardOutput=inherit\nStandardError=inherit\nRestartSec=1\nStartLimitIntervalSec=0\nRestart=always\nUser=pi\n\n[Install]\nWantedBy=multi-user.target\n```\n\nThen:\n- Try it with `sudo service simple-proxy start`\n- Enable it to run on the server startup: `sudo systemctl enable simple-proxy.service`\n\n\n#### simple-proxy-tunnel\nOn the server, save it as `/etc/systemd/system/simple-proxy-tunnel.service`:\n\n```\n[Unit]\nDescription=The simple-proxy-tunnel\nAfter=network.target\n\n[Service]\nExecStart=/usr/bin/ssh -nNT -R 9999:localhost:9000 -o ConnectTimeout=10 -o ExitOnForwardFailure=yes -o ServerAliveInterval=180 ubuntu@cazadescuentos.net\nWorkingDirectory=/home/pi\nStandardOutput=inherit\nStandardError=inherit\nRestartSec=1\nStartLimitIntervalSec=0\nRestart=always\nUser=pi\n\n[Install]\nWantedBy=multi-user.target\n```\n\nThen:\n- Try it with `sudo service simple-proxy-tunnel start`\n- Enable it to run on the server startup: `sudo systemctl enable simple-proxy-tunnel.service`\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwiringbits%2Fsimple-http-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwiringbits%2Fsimple-http-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwiringbits%2Fsimple-http-proxy/lists"}