{"id":22120062,"url":"https://github.com/maxpleaner/showscraper","last_synced_at":"2025-10-25T11:22:00.002Z","repository":{"id":41137934,"uuid":"485831183","full_name":"MaxPleaner/ShowScraper","owner":"MaxPleaner","description":"automated listing of concerts in san francisco area","archived":false,"fork":false,"pushed_at":"2025-04-28T04:40:54.000Z","size":90015,"stargazers_count":5,"open_issues_count":5,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-28T05:19:53.376Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://bayareashows.org","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaxPleaner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-04-26T14:53:39.000Z","updated_at":"2025-04-28T04:01:02.000Z","dependencies_parsed_at":"2022-09-08T00:52:14.465Z","dependency_job_id":"d09ef32e-ae03-4de3-803f-3a03338c7aeb","html_url":"https://github.com/MaxPleaner/ShowScraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MaxPleaner/ShowScraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2FShowScraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2FShowScraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2FShowScraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2FShowScraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaxPleaner","download_url":"https://codeload.github.com/MaxPleaner/ShowScraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2FShowScraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267007597,"owners_count":24020261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T14:20:11.709Z","updated_at":"2025-10-25T11:21:59.916Z","avatar_url":"https://github.com/MaxPleaner.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ShowScraper\n\nThis is a collection of a bunch of scrapers which fetch concert listings from\nvarious venues in the Bay Area.\n\nIt also is a website to view the listings.\n\n## Tech Stack\n\nThe scraper uses headless [Selenium](https://www.selenium.dev/)\n([Chromedriver](https://chromedriver.chromium.org/downloads))\nand is written in [Ruby](https://www.ruby-lang.org/en/).\n\nResults are converted to [JSON](https://www.json.org/json-en.html)\nand pushed to a [Google Cloud Storage](https://cloud.google.com/storage)\nbucket.\n\nThe front end is written in [React](https://reactjs.org/)\nand styled with [Bulma CSS framework](https://bulma.io/).\n\n## Setup - Scraper\n\n1. Run `bundle install` in the root of the repo using a stable Ruby version\n\n1. You're gonna need to install Chromedriver.\n     - **On Raspberry Pi**:  you can do this with `sudo apt-get install chromium-chromedriver` and this will happen automatically (the scraper will check your system architecture and use the right \n       chromedriver path)\n     - **on OSX and Windows** you can find all versions at [https://chromedriver.chromium.org/](https://chromedriver.chromium.org/) and then add it to your PATH.\n       For example, add this to your `~/.bash_profile`:\n         ```\n         export PATH=$PATH:/path/to/folder/containing/chromedriver/\n         ```\n\n1. Run `cp .env.example .env` in the root of the repo. The `.env` file sets environment variables which can be used to customize the application's behavior. This file can be edited directly.\n\n1. Make a new \"project\" on google cloud. Create a GCS bucket in the project. Add the credentials to `.env`:\n  \n    ```\n    STORAGE_PROJECT = \"my-project-id\"\n    STORAGE_CREDENTIALS = \"path/to/keyfile.json\"\n    ```\n\n1. Change the GCS bucket permissions so all files are publicly available by default.\n\n1. Configure `gsutils` to use your new project, then upload the CORS file which I've included in the repo:\n\n    ```\n    gsutil cors set cors-json-file.json gs://\u003cBUCKET_NAME\u003e\n    ```\n\n## Setup - Frontend\n\n1. Make sure you're using a stable Node version\n2. `cd frontend/react-app`\n3. run `yarn install` to get dependencies\n4. `yarn start` and then open `localhost:3000`\n\nTo build the project for production, use `yarn build`\n\n_Note_: There is also a script which you can run from the root of the repo to start the react server:\n\n```\nbin/run_frontend\n```\n\nThis runs `nvm use 14; cd frontend/react_app \u0026\u0026 yarn install \u0026 yarn start`\n\nNote you will probably have to change this `nvm use 14` if you are using a different Node version.\n\n## Running Scraper\n\nThere is a command line tool at `bin/run_scraper`.\nBy default it will run all scrapers (each will fetch a maximum of 200 events) \nand then upload the results to GCS.\n\n**Options** (note that most of these can also be set from `.env`)\n\n```\n# Limits each scraper to N results\n--limit=10\n\n# Just print the results, don't upload them to GCS\n--skip-persist\n\n# Don't rescue scraping errors - one broken scraper will stop the whole script\n# If rescue=true (the default) then broken scrapers will just be skipped\n--rescue=false\n\n# Just update the list of venues. Don't actually scrape any events.\n--no-scrape\n\n# Limit the scrape to a set of venues. Comma-separated list.\n--sources=GreyArea,Cornerstone\n\n# Run headlessly, or not\n--headless=true\n--headless=false\n```\n\nFor example, combining options:\n\n```\nbin/run-scraper --headless=true --limit=5 --skip-persist --rescue=false sources=ElboRoom,Knockout\n```\n\nThere are some other configuration options done through ENV, see `.env.example`\n\nNote that every time you run a scraper, it will completely overwrite the list of events for that venue.\n\n## Testing\n\nThere are some basic automated tests for the scrapers. Run `bundle exec rspec` from the root of the repo.\nUsing rspec you can also isolate certain tests to run (left as an exercise to the reader).\n\n## Adding a new scraper\n\n1. Add a new entry to `sources.json`. You can get `latlng` from Google Maps\n(right click the marker on the map and the coords will pop up). For `desc` you can\njust copy the blurb from Google Maps as well.\n\n2. Create a new file `scraper/lib/sources/venue_name.rb` (replacing `venue_name`, obviously).\n\n3. You can copy one of the existing scraper classes as a starting point.\n   Note that there are a few different types of websites (calendar view, infinite scroll, all-on-one-page)\n   so it's best to find another scraper that is similar in that regard.\n\n4. Make sure the class name is the exact same as the `name` value in `sources.json`\n\n5. Fill out the contents of the scraper, using `binding.pry` and the `HEADLESS=false`\n   environment variable as needed for debugging.\n\n6. Add a test case to `scraper_spec.rb` (can just use `generic_run_test` like the other scrapers)\n\nNote, there is no need to explicitly `require` the scraper class anywhere into the codebase.\nAutoloading is already set up based on `sources.json`.\n\n## A note about the unused/server and unused/db folders\n\nThese are both unused. I kept them here in case I want to have a dedicated backend at some point.\n\nFor now it suffices to go backend-less and just host the results on GCS.\n\n## Development - TODOS\n\n- [ ] Map View\n- [ ] Add more meta-scrapers (e.g. scrape other scrapers/aggregators), especially for electronic shows which aren't really captured by the current venue list or \"The List\"\n- [ ] Add more venues (have specifically received requests for South Bay, but probably there are new SF / East Bay venues as well). \n- [ ] Add Venue Events List view (accessible from Venue List View)\n- [x] Find a way to handle events that don't have an explicit year in their date\n- [x] Add Submit Event / About pages\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxpleaner%2Fshowscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxpleaner%2Fshowscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxpleaner%2Fshowscraper/lists"}