{"id":25408312,"url":"https://github.com/rahuletto/goscraper","last_synced_at":"2025-10-31T03:30:44.809Z","repository":{"id":277577114,"uuid":"915746566","full_name":"Rahuletto/goscraper","owner":"Rahuletto","description":"The SRM Academia scraper used by ClassPro, written in Go with Fiber","archived":false,"fork":false,"pushed_at":"2025-02-14T17:03:43.000Z","size":227,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-14T18:20:39.638Z","etag":null,"topics":["classpro","fiber","go","scraper"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rahuletto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-12T17:37:15.000Z","updated_at":"2025-02-14T17:03:47.000Z","dependencies_parsed_at":"2025-02-14T18:31:34.976Z","dependency_job_id":null,"html_url":"https://github.com/Rahuletto/goscraper","commit_stats":null,"previous_names":["rahuletto/goscraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rahuletto%2Fgoscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rahuletto%2Fgoscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rahuletto%2Fgoscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rahuletto%2Fgoscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rahuletto","download_url":"https://codeload.github.com/Rahuletto/goscraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239102837,"owners_count":19582051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["classpro","fiber","go","scraper"],"created_at":"2025-02-16T07:30:55.465Z","updated_at":"2025-10-31T03:30:44.788Z","avatar_url":"https://github.com/Rahuletto.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# goscraper\n\nGoscraper is a backend that fetches and scrapes data from SRM Academia and parses it dynamically. The API is designed for performance, speed, and accuracy, accommodating changes in the HTML structure.\n\n\u003e [!IMPORTANT]\n\u003e This is part of a **monorepo** containing both frontend and backend components. Please fork the entire monorepo instead of cloning this backend separately.\n\n\n## Prerequisites\n\nBefore starting the application, you need to set up Supabase and configure the required environment variables.\n\n### 1. Supabase Setup\n\nFirst, create a new Supabase project and set up the required database tables:\n\n#### Create the `goscrape` table:\n```sql \ncreate table public.goscrape (\n  \"regNumber\" text not null,\n  \"user\" text null,\n  timetable text null,\n  courses text null,\n  attendance text null,\n  marks text null,\n  \"lastUpdated\" numeric null,\n  token text not null,\n  ophour text null default ''::text,\n  constraint goscrape_pkey primary key (\"regNumber\", token),\n  constraint goscrape_regNumber_key unique (\"regNumber\")\n);\n```\n\n#### Create the `gocal` table:\n```sql \ncreate table public.gocal (\n  id bigint generated by default as identity not null,\n  date text null,\n  month text null,\n  day text null,\n  \"order\" text null,\n  event text null,\n  created_at numeric null\n);\n```\n\n#### Set up CRON Jobs\n\n\u003e [!WARNING]\n\u003e Install CRON Integration in Supabase first\n\u003e \u003cimg width=\"891\" alt=\"image\" src=\"https://github.com/user-attachments/assets/ad98da2f-f84c-43a9-8d87-8c512cdcaaa3\" /\u003e\n\n**Deleting old user data:**\n```sql\nUPDATE goscrape\nSET \"user\" = NULL, timetable = NULL, attendance = NULL, marks = NULL, courses = NULL\nWHERE \"lastUpdated\" \u003c (EXTRACT(EPOCH FROM NOW()) * 1000) - (12 * 60 * 60 * 1000);\n```\n\n**Deleting old calendar:**\n```sql\nCREATE OR REPLACE FUNCTION delete_from_gocal()\nRETURNS void AS $$\nBEGIN\n    DELETE FROM gocal;\nEND;\n$$ LANGUAGE plpgsql SECURITY INVOKER;\n\nselect\n  cron.schedule (\n    '0 0 * * *',\n    'SELECT delete_old_calendar_events()'\n  );\n```\n\n### 2. Environment Variables\n\nEnvironment variables are handled by the monorepo. Please refer to the main repository's [README.md](https://github.com/rahuletto/classpro) for complete setup instructions.\n\n---\n\n### WHEN DEPLOYING\nGo to `globals/DevMode` and set the variable to false\n\n---\n\n## Setup Instructions\n\n1. **Fork the monorepo:**\n   ```\n   git clone --recurse-submodules https://github.com/rahuletto/classpro\n   cd classpro/backend\n   ```\n\n2. **Install dependencies:**\n   ```\n   go mod tidy\n   ```\n\n3. **Development Run the application:** (DEV SERVER)\n   ```\n   go run main.go\n   ```\n\n4. **Build and Run the application:** (BUILD SERVER)\n   ```\n   go build main.go\n   ./main\n   ```\n\n## Docker Setup\n```\ndocker compose up --build\n```\n\n## Contributing\n\nContributions are welcome! Please open an issue or submit a pull request for any enhancements or bug fixes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahuletto%2Fgoscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frahuletto%2Fgoscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahuletto%2Fgoscraper/lists"}