https://github.com/michaeltoohig/unofficial-vmgd-api

Unofficial API for the Vanuatu Meteorology and Geo-Hazards Department
https://github.com/michaeltoohig/unofficial-vmgd-api

fastapi webscraping

Last synced: 16 days ago
JSON representation

Unofficial API for the Vanuatu Meteorology and Geo-Hazards Department

• Host: GitHub
• URL: https://github.com/michaeltoohig/unofficial-vmgd-api
• Owner: michaeltoohig
• Created: 2023-03-22T23:16:57.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2024-07-01T15:57:26.000Z (7 months ago)
• Last Synced: 2024-11-27T19:16:55.345Z (3 months ago)
• Topics: fastapi, webscraping
• Language: HTML
• Homepage: http://unofficial-vmgd-api.michaeltoohig.com
• Size: 1.09 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Unofficial VMGD API

A web scraper and API for the Vanuatu Meteorology & Geo-Hazards Department (VMGD) website.

The goal of this project is to provide VMGD data in a machine readable format.

However, for no technical reason, I've limited the scope of the project to only the `forecast` and `warnings` section of the VMGD website.

## Development

First setup the environment file.

Update the values as needed.

```

cp ./data/.env.template ./data/.env.development

```

Then build and start the container.

```

docker compose -f docker-compose.development.yml build

docker compose -f docker-compose.development.yml up

```

The dev server for the API will started.

Run database migrations as needed.

```

docker compose -f docker-compose.development.yml exec app alembic upgrade head

```

Then run the scraper.

```

docker compose -f docker-compose.development.yml exec app python run_scraper.py

```

#### SSR HTML

I'm thinking of removing this due to it being ugly, lol.

I can show I can do better.

But, if you want to continue to develop the SSR application run the following to compile scss.

```

docker compose -f docker-compose.development.yml exec app boussole {compile|watch}

```

## Tests

Run pytest.

```

docker compose -f docker-compose.development.yml exec app pytest tests/

```

## TODO

Features

- [ ] consider offering option to allow users to query by `fetched_at`, `issued_at` or other date value

- [ ] fun api endpoints like "do I need an umbrella today" endpoint

Improvements

- [ ] fix forecast and other datetime responses that should be implicit VU timezone instead of returning UTC for **everything**.

      Current tasks

Tasks

- [ ] improve `/raw/pages` endpoint

  - I removed these endpoints for now

## Roadmap

- [ ] track volcano activity

- [ ] track cyclone activity

- [ ] track el nino level