https://github.com/finki-hub/services-scraper
Scraper for many FCSE services and sending data through webhooks
https://github.com/finki-hub/services-scraper
fcse finki ukim
Last synced: 4 months ago
JSON representation
Scraper for many FCSE services and sending data through webhooks
- Host: GitHub
- URL: https://github.com/finki-hub/services-scraper
- Owner: finki-hub
- License: mit
- Created: 2022-09-07T00:56:30.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2026-02-28T01:30:21.000Z (4 months ago)
- Last Synced: 2026-02-28T07:16:01.949Z (4 months ago)
- Topics: fcse, finki, ukim
- Language: TypeScript
- Homepage:
- Size: 2.15 MB
- Stars: 3
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# FINKI Hub / Services Scraper
Tooling for scraping and providing publicly available data from FCSE services. The data is provided using a REST API or webhooks. Requires Node.js 24+.
## Architecture
The scrapers are implemented as classes (called strategies) which contain several selectors and methods for fetching the data from each container (post, announcement, etc). Adding a new service requires creating a new strategy and linking it. See [the example strategy](./src/strategies/ExampleStrategy.ts) for more info.
## Quick Setup (Production)
To run the scraper:
1. Clone the repository: `git clone https://github.com/finki-hub/services-scraper.git`
2. Prepare configuration by copying `config/config.sample.json` to `config/config.json`
3. Install dependencies: `npm i`
4. Run the scraper `npm run start`
It's also available as a Docker image:
```sh
docker run -d \
--name services-scraper \
--restart unless-stopped \
-v ./cache:/app/cache \
-v ./config:/app/config \
-v ./logs:/app/logs \
ghcr.io/finki-hub/services-scraper:latest
```
Or Docker Compose: `docker compose up -d`
You can select which scrapers to run declaratively (in the configuration with the `enabled` flag) or imperatively: `npm run start scraper_1 scraper_2 ... scraper_n`
## Quick Setup (Development)
1. Clone the repository: `git clone https://github.com/finki-hub/services-scraper.git`
2. Install dependencies: `npm i`
3. Prepare configuration: `cp config/config.sample.json config/config.json`
4. Build the project: `npm run build`
5. Run it: `npm run start`
## Configuration
There is an example configuration file available at [`config/config.sample.json`](./config/config.sample.json). Copy it to `config/config.json` and edit it to your liking.
## License
This project is licensed under the terms of the MIT license.