https://github.com/rahuletto/goscraper

The SRM Academia scraper used by ClassPro, written in Go with Fiber
https://github.com/rahuletto/goscraper

classpro fiber go scraper

Last synced: 9 months ago
JSON representation

The SRM Academia scraper used by ClassPro, written in Go with Fiber

• Host: GitHub
• URL: https://github.com/rahuletto/goscraper
• Owner: Rahuletto
• Created: 2025-01-12T17:37:15.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-02-14T17:03:43.000Z (over 1 year ago)
• Last Synced: 2025-02-14T18:20:39.638Z (over 1 year ago)
• Topics: classpro, fiber, go, scraper
• Language: Go
• Homepage:
• Size: 222 KB
• Stars: 0
• Watchers: 2
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# goscraper

Goscraper is a backend that fetches and scrapes data from SRM Academia and parses it dynamically. The API is designed for performance, speed, and accuracy, accommodating changes in the HTML structure.

> [!IMPORTANT]

> This is part of a **monorepo** containing both frontend and backend components. Please fork the entire monorepo instead of cloning this backend separately.

## Prerequisites

Before starting the application, you need to set up Supabase and configure the required environment variables.

### 1. Supabase Setup

First, create a new Supabase project and set up the required database tables:

#### Create the `goscrape` table:

```sql 

create table public.goscrape (

  "regNumber" text not null,

  "user" text null,

  timetable text null,

  courses text null,

  attendance text null,

  marks text null,

  "lastUpdated" numeric null,

  token text not null,

  ophour text null default ''::text,

  constraint goscrape_pkey primary key ("regNumber", token),

  constraint goscrape_regNumber_key unique ("regNumber")

);

```

#### Create the `gocal` table:

```sql 

create table public.gocal (

  id bigint generated by default as identity not null,

  date text null,

  month text null,

  day text null,

  "order" text null,

  event text null,

  created_at numeric null

);

```

#### Set up CRON Jobs

> [!WARNING]

> Install CRON Integration in Supabase first

> 

**Deleting old user data:**

```sql

UPDATE goscrape

SET "user" = NULL, timetable = NULL, attendance = NULL, marks = NULL, courses = NULL

WHERE "lastUpdated" < (EXTRACT(EPOCH FROM NOW()) * 1000) - (12 * 60 * 60 * 1000);

```

**Deleting old calendar:**

```sql

CREATE OR REPLACE FUNCTION delete_from_gocal()

RETURNS void AS $$

BEGIN

    DELETE FROM gocal;

END;

$$ LANGUAGE plpgsql SECURITY INVOKER;

select

  cron.schedule (

    '0 0 * * *',

    'SELECT delete_old_calendar_events()'

  );

```

### 2. Environment Variables

Environment variables are handled by the monorepo. Please refer to the main repository's [README.md](https://github.com/rahuletto/classpro) for complete setup instructions.

---

### WHEN DEPLOYING

Go to `globals/DevMode` and set the variable to false

---

## Setup Instructions

1. **Fork the monorepo:**

   ```

   git clone --recurse-submodules https://github.com/rahuletto/classpro

   cd classpro/backend

   ```

2. **Install dependencies:**

   ```

   go mod tidy

   ```

3. **Development Run the application:** (DEV SERVER)

   ```

   go run main.go

   ```

4. **Build and Run the application:** (BUILD SERVER)

   ```

   go build main.go

   ./main

   ```

## Docker Setup

```

docker compose up --build

```

## Contributing

Contributions are welcome! Please open an issue or submit a pull request for any enhancements or bug fixes.