{"id":21836633,"url":"https://github.com/teemukoivisto/varjo-sport-lambda-scraper","last_synced_at":"2026-04-05T20:36:57.807Z","repository":{"id":51669635,"uuid":"221411106","full_name":"TeemuKoivisto/varjo-sport-lambda-scraper","owner":"TeemuKoivisto","description":"Website and scraper with Scrapy + AWS Lambda to scrape https://unisport.fi 's crappy website","archived":false,"fork":false,"pushed_at":"2021-05-10T19:29:43.000Z","size":289,"stargazers_count":3,"open_issues_count":6,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-26T10:11:21.673Z","etag":null,"topics":["aws","aws-lambda","nodejs","preact","s3","scrapy","serverless","unisport"],"latest_commit_sha":null,"homepage":"https://varjosport.net/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TeemuKoivisto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-13T08:36:33.000Z","updated_at":"2020-05-09T04:20:46.000Z","dependencies_parsed_at":"2022-08-22T19:20:17.254Z","dependency_job_id":null,"html_url":"https://github.com/TeemuKoivisto/varjo-sport-lambda-scraper","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeemuKoivisto%2Fvarjo-sport-lambda-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeemuKoivisto%2Fvarjo-sport-lambda-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeemuKoivisto%2Fvarjo-sport-lambda-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeemuKoivisto%2Fvarjo-sport-lambda-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TeemuKoivisto","download_url":"https://codeload.github.com/TeemuKoivisto/varjo-sport-lambda-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244815737,"owners_count":20514985,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","nodejs","preact","s3","scrapy","serverless","unisport"],"created_at":"2024-11-27T20:41:02.073Z","updated_at":"2025-12-30T23:57:25.371Z","avatar_url":"https://github.com/TeemuKoivisto.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [Varjo Sport Lambda Scraper](https://varjosport.net)\n\nScrapes the basic info (mainly business hours) of all Unisport gyms.\n\nWhy you ask? Because as of now you can't view them all on one page and they are two clicks away from the frontpage. Who designed such moronic system, I don't know, but to me it's annoying since I have no interest in other stuff than seeing how long its open today.\n\n# How to run the scraper locally\n\nRequires Python \u003e=3.6 with preferably virtualenv. I use `virtualenv-wrapper`, some have said `pyenv` is pretty good too.\n\n1. Activate your virtualenv eg `workon varjos`\n2. Install dependencies: `pip install -r requirements.txt` ([Scrapy](https://scrapy.org/) and Twisted)\n3. Load the dev commands: `. cmds.sh`\n4. Run the spider: `crawl`\n\nYou should get `./frontend/unisport_gyms.json` file with the data scraped.\n\nUse `shell` to open [interactive Scrapy shell](https://docs.scrapy.org/en/latest/topics/shell.html) to test CSS selectors without having to run the spider.\n\n# How to run the test server\n\nRequires Node.js \u003e=10.\n\n1. Run: `node server.js`\n2. The server should run at http://localhost:4040/\n\nReload the page after making changes to the files inside `frontend`-folder.\n\n# How to deploy the fronted\n\nRequires AWS account and one S3 bucket.\n\n1. Set the bucket's permissions to allow public bucket access and enable static website hosting\n2. Configure your local AWS user with access to that bucket\n3. You should replace the bucket name in my `cmds.sh` script with your own\n4. Then deploy the code with `AWS_PROFILE=varjosport.net-ci deploy_front` where AWS_PROFILE is your local AWS profile.\n\nGo to the bucket's website URL to see the app running eg http://varjosport.net.s3-website.eu-north-1.amazonaws.com\n\nIf you want to deploy it to your own domain incase this one dies out for some reason, you have to configure Route 53, Certificate Manager, and CloudFront too. Pretty basic configuration, so I didn't bother writing it down as a template.\n\n# How to run the serverless lambda\n\nRequires Node.js \u003e=10. Docker if you want to deploy it. AWS account with one local AWS user with admin privileges (I'm lazy) and one S3 bucket (the same bucket you use to host the frontend). \n\n1. Install dependencies: `npm i`\n2. Run `npm run invoke` to execute the lambda. It will most probably fail because I hard-coded the profile and the bucket. Change them to your own\n3. Similar to the previous command, `npm run deploy` will deploy the lambda but I've hard-coded the parameters\n\nOnce deployed, instead of waiting 24 hours for the lambda to run, you can trigger it manually by going to your AWS console's Lambda page for this function and creating \u0026 sending a test event.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteemukoivisto%2Fvarjo-sport-lambda-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteemukoivisto%2Fvarjo-sport-lambda-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteemukoivisto%2Fvarjo-sport-lambda-scraper/lists"}