https://github.com/as2811-project/au-elec-data-pipeline
A Kafka data pipeline to fetch data from an API and store it in a Postgres database
https://github.com/as2811-project/au-elec-data-pipeline
aws-ec2 data-science kafka postgres supabase
Last synced: 2 months ago
JSON representation
A Kafka data pipeline to fetch data from an API and store it in a Postgres database
- Host: GitHub
- URL: https://github.com/as2811-project/au-elec-data-pipeline
- Owner: as2811-project
- Created: 2024-07-16T10:17:01.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-01T23:50:21.000Z (almost 2 years ago)
- Last Synced: 2025-06-02T20:37:49.993Z (about 1 year ago)
- Topics: aws-ec2, data-science, kafka, postgres, supabase
- Language: Jupyter Notebook
- Homepage:
- Size: 12.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# AEMO Electricity Generation Data Pipeline
This is a data pipeline created for streaming state-wise electricity data from AEMO to a PostgreSQL database (Supabase) using Kafka.
## Feature
- Live data updates: The producer fetches data from AEMO's API and updates the corresponding state's table in the database once every 5 minutes.
## Installation
1. Clone the repo
2. Install the required dependencies from the requirements.txt file
## Usage
- Store your Postgres URI in a .env file as "DB_URL"
- Spin up an AWS EC2 instance (Ubuntu). Install Kafka and Java.
- Configure Kafka in the EC2 instance. Uncomment 'advertised_listeners' and add your EC2 instance's public IP address here. Start Zookeeper and Kafka server.
- In the instance's security rules, allow incoming traffic from your IP address only. Then, start your producer and consumer.
- Replace the IP address here with your instance's IP address.
- Run the initial_batch.py script to fetch the first batch of data into your database.
- Run the producer and consumer scripts for updates.
**Acknowledgement: The data is from AEMO's wonderful dashboard. Link below:**
https://aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/data-dashboard-nem