https://github.com/vietdoo/sg-property-hub
SG Property Hub is a comprehensive platform for managing and analyzing property data.
https://github.com/vietdoo/sg-property-hub
airflow celery-redis crawler etl etl-pipeline fastapi minio mongodb nextjs postgresql s3 spark webscraping
Last synced: 4 months ago
JSON representation
SG Property Hub is a comprehensive platform for managing and analyzing property data.
- Host: GitHub
- URL: https://github.com/vietdoo/sg-property-hub
- Owner: vietdoo
- Created: 2023-10-05T17:06:23.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-11-09T07:14:30.000Z (over 1 year ago)
- Last Synced: 2025-02-07T00:28:59.454Z (over 1 year ago)
- Topics: airflow, celery-redis, crawler, etl, etl-pipeline, fastapi, minio, mongodb, nextjs, postgresql, s3, spark, webscraping
- Language: Python
- Homepage: https://sgpcloud.site/
- Size: 3.99 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.MD
Awesome Lists containing this project
README
# SG Property Hub
## Overview
SG Property Hub is a comprehensive platform for managing and analyzing property data. It consists of multiple components including a back-end API, a data pipeline, a front-end application, and a property crawler worker. Each component is containerized using Docker for easy deployment and orchestration.
## Components
### Data Pipeline
The data pipeline component handles data processing tasks using Apache Spark and Apache Airflow.
- **DAGs**: Contains Airflow DAGs for orchestrating data workflows.
- **Spark**: Contains Spark jobs for data processing.
- **MinIO**: Used for object storage.
- **Dependencies**: Managed using `requirements.txt`.
### Back-End
The back-end component provides APIs and database management functionalities.
- **API**: Contains the API logic.
- **DB**: Contains database-related scripts and configurations.
- **Dependencies**: Managed using `requirements.txt`.
### Next-Hub
The front-end application built with Next.js.
- **Components**: Reusable UI components.
- **Utilities**: Helper functions and utilities.
- **Configuration**: Next.js and Tailwind CSS configurations.
- **Dependencies**: Managed using `package.json`.
### Property Crawler Worker
The property crawler worker is responsible for web scraping property data.
- **Scripts**: Contains scripts for crawling different property websites.
- **Dependencies**: Managed using `requirements.txt`.
## Getting Started
### Prerequisites
- Docker
- Docker Compose
- Node.js (for Next.js application)
### Setup
1. **Clone the repository**:
```sh
git clone https://github.com/vietdoo/sg-property-hub.git
cd sg-property-hub
```
2. **Back-End**:
```sh
cd back-end
docker-compose up --build
```
3. **Data Pipeline**:
```sh
cd data-pipeline
docker-compose up --build
```
4. **Next-Hub**:
```sh
cd next-hub
npm install
npm run dev
```
5. **Property Crawler Worker**:
```sh
cd property-crawler-worker
docker-compose up --build
```
## Usage
### Back-End
Access the API at `http://localhost:8000`.
### Data Pipeline
Access the Airflow UI at `http://localhost:8080`.
### Next-Hub
Access the front-end application at `http://localhost:3000`.
### Property Crawler Worker
Run the crawler scripts as needed.
## Contributing
Contributions are welcome! Please open an issue or submit a pull request.
## License
This project is licensed under the MIT License.
```
Feel free to customize this README file further based on your specific requirements.