{"id":45658846,"url":"https://github.com/codeforboston/cornerwise-scrapers","last_synced_at":"2026-02-24T08:57:52.196Z","repository":{"id":25786535,"uuid":"104963032","full_name":"codeforboston/cornerwise-scrapers","owner":"codeforboston","description":null,"archived":false,"fork":false,"pushed_at":"2022-12-08T11:48:32.000Z","size":2039,"stargazers_count":0,"open_issues_count":6,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-10-08T18:46:32.359Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codeforboston.png","metadata":{"funding":{"custom":"https://www.codeforamerica.org/donate-to-a-brigade?utm_campaign=CodeforBoston\u0026utm_source=github"},"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-27T02:39:09.000Z","updated_at":"2018-06-11T22:32:36.000Z","dependencies_parsed_at":"2023-01-14T03:24:55.522Z","dependency_job_id":null,"html_url":"https://github.com/codeforboston/cornerwise-scrapers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/codeforboston/cornerwise-scrapers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforboston%2Fcornerwise-scrapers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforboston%2Fcornerwise-scrapers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforboston%2Fcornerwise-scrapers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforboston%2Fcornerwise-scrapers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codeforboston","download_url":"https://codeload.github.com/codeforboston/cornerwise-scrapers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforboston%2Fcornerwise-scrapers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29777480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-24T04:54:30.205Z","status":"ssl_error","status_checked_at":"2026-02-24T04:53:58.628Z","response_time":75,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-24T08:57:50.161Z","updated_at":"2026-02-24T08:57:52.167Z","avatar_url":"https://github.com/codeforboston.png","language":"Python","funding_links":["https://www.codeforamerica.org/donate-to-a-brigade?utm_campaign=CodeforBoston\u0026utm_source=github"],"categories":[],"sub_categories":[],"readme":"* Cornerwise Scrapers\n** Introduction\n   Implementation of the \"standard\" [[https://github.com/codeforboston/cornerwise][Cornerwise]] scrapers using [[https://aws.amazon.com/lambda/][AWS Lambda]]\n   infrastructure.\n\n** Setup \n   - Install ~Node.js~, preferably using [[https://github.com/creationix/nvm#installation][NVM]].\n   - ~npm install serverless~\n     Installs [[https://serverless.com][Serverless]], a command line utility that simplifies the deployment\n     of services that run on AWS Lambda, Azure Functions, or others.\n\n   - ~npm install --save serverless-python-requirements~\n     Installs a Serverless plugin that will download the PIP requirements\n     specified in ~requirements.txt~ before deploying to AWS. (Note: you need\n     Docker installed for this to run correctly.)\n\n** Deploying\n   - To deploy to AWS, you'll need to set up an AWS account, if you haven't\n     already. You should also configure a ~cornerwise~ profile in your AWS\n     credentials. See [[https://serverless.com/framework/docs/providers/aws/guide/credentials/][here]] for details about setting up a profile and the\n     privileges the AWS user requires\n   - Copy ~credentials.example.json~ to ~credentials.json~ and modify the\n     variables to use your Socrata credentials.\n   - If everything is correctly configured, you should be able to ~cd~ to this\n     directory and type ~serverless deploy -v~ to fully deploy the lambda\n     function and corresponding API Gateway interface to AWS.\n\n* Scrapers\n** Somerville, MA Reports and Decisions\n   - URL :: https://scraper.cornerwise.org/somervillema\n   - Types :: Cases\n   - Source :: [[./somervillema.py][somervillema.py]]\n   - Description :: Scrapes the OSPCD's [[https://www.somervillema.gov/departments/ospcd/planning-and-zoning/reports-and-decisions][Reports and Decisions]] page.\n** Somerville, MA PB/ZBA Event Scraper\n   - URL :: https://scraper.cornerwise.org/somervillema_events\n   - Types :: Events\n   - Source :: [[file:somervillema_events.py][somervillema_events.py]]\n   - Description :: Scrapes the city's [[https://www.somervillema.gov/event-documents][events]] page, finds events for the\n                    Planning Board and Zoning Board of Appeals, and scrapes the\n                    attached Agenda for related case numbers.\n** Cambridge, MA\n   - URL :: https://scraper.cornerwise.org/cambridgema\n   - Types :: Cases\n   - Source :: [[./cambridgema.py][cambridgema.py]]\n\n** Somerville, MA Capital Projects\n   - URL :: https://scraper.cornerwise.org/somerville_projects\n   - Source :: [[./somervillema_projects.py][somervillema_projects.py]]\n   - Types :: Projects\n   - Description :: Published annually by the Somerville Capital Projects\n                    Committee, the dataset [[https://data.somervillema.gov/Finance/Capital-Investment-Plan-Projects-FY16-26/wz6k-gm5k][includes]] \"infrastructure projects,\n                    building improvements, park redesigns, and equipment\n                    purchases.\"\n\n** Green Line Extension\n   - URL :: https://scraper.cornerwise.org/greenline\n   - Types :: Events\n   - Description :: Scrapes the \"Upcoming Meetings\" section of the\n                    [[http://greenlineextension.org/][Green Line Extension]] home page.\n\n* Using the Scrapers\n  The interface to the scrapers is intentionally simple. Place a GET request to\n  the scraper's URL. You may optionally supply a ~since~ query parameter\n  formatted as ~yyyymmdd~. The scraper will respond with JSON conforming to the\n  [[http://lbovet.github.io/docson/index.html#https://raw.githubusercontent.com/codeforboston/cornerwise/master/docs/scraper-schema.json][Cornerwise scraper schema]].\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeforboston%2Fcornerwise-scrapers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodeforboston%2Fcornerwise-scrapers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeforboston%2Fcornerwise-scrapers/lists"}