https://github.com/aust21/crime-data-digitisation
Simulation of migrating from storing data on files to cloud storage
https://github.com/aust21/crime-data-digitisation
airflow aws data-engineering python streamlit
Last synced: 7 months ago
JSON representation
Simulation of migrating from storing data on files to cloud storage
- Host: GitHub
- URL: https://github.com/aust21/crime-data-digitisation
- Owner: aust21
- Created: 2025-02-20T20:28:24.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-28T13:15:20.000Z (about 1 year ago)
- Last Synced: 2025-03-28T14:24:17.523Z (about 1 year ago)
- Topics: airflow, aws, data-engineering, python, streamlit
- Language: Python
- Homepage: https://crimedata.streamlit.app/
- Size: 577 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# CrimeStats Digitization
A project to simulate the digitization of South African crime statistics, effectively storing them on secure cloud systems (AWS).
# Features
- Data Ingestion: Reads crime data from a flat file stored in an AWS bucket.
- Data Storage: Loads processed data into a PostgreSQL database on AWS.
- Real-Time Processing: Fast access to frequently queried records using Redis caching.
# Tech Stack
- Python: Core programming language for data processing and pipeline management.
- PostgreSQL: Relational database to store digitized crime data.
- Airflow: For scheduling and automating the pipeline
# Getting Started
## Prerequisites
Make sure you have the following installed:
- Python 3.x
### Installation
1. Clone the repo
```bash
git clone https://github.com/aust21/crime-stats-digitisation.git
cd crime-stats-digitisation
```
2. Create a virtual environment
- On Linux
```bash
python3 -m venv venv
source venv/bin/activate
```
- Windows
```bash
python -m venv venv
venv\Scripts\activate
```
3. Install dependencies
```bash
pip install -r requirements.txt
```