{"id":22889667,"url":"https://github.com/vietdoo/sg-property-hub","last_synced_at":"2026-04-08T20:50:35.594Z","repository":{"id":259695564,"uuid":"700981226","full_name":"vietdoo/sg-property-hub","owner":"vietdoo","description":"SG Property Hub is a comprehensive platform for managing and analyzing property data.","archived":false,"fork":false,"pushed_at":"2024-11-09T07:14:30.000Z","size":4186,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-07T00:28:59.454Z","etag":null,"topics":["airflow","celery-redis","crawler","etl","etl-pipeline","fastapi","minio","mongodb","nextjs","postgresql","s3","spark","webscraping"],"latest_commit_sha":null,"homepage":"https://sgpcloud.site/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vietdoo.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-05T17:06:23.000Z","updated_at":"2024-11-09T07:14:34.000Z","dependencies_parsed_at":"2025-02-07T00:36:29.260Z","dependency_job_id":null,"html_url":"https://github.com/vietdoo/sg-property-hub","commit_stats":null,"previous_names":["vietdoo/nextjs-properties-ai"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vietdoo%2Fsg-property-hub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vietdoo%2Fsg-property-hub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vietdoo%2Fsg-property-hub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vietdoo%2Fsg-property-hub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vietdoo","download_url":"https://codeload.github.com/vietdoo/sg-property-hub/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246531974,"owners_count":20792736,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","celery-redis","crawler","etl","etl-pipeline","fastapi","minio","mongodb","nextjs","postgresql","s3","spark","webscraping"],"created_at":"2024-12-13T21:56:35.517Z","updated_at":"2026-04-08T20:50:30.565Z","avatar_url":"https://github.com/vietdoo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# SG Property Hub\n\n## Overview\n\nSG Property Hub is a comprehensive platform for managing and analyzing property data. It consists of multiple components including a back-end API, a data pipeline, a front-end application, and a property crawler worker. Each component is containerized using Docker for easy deployment and orchestration.\n\n![Main Pipeline](public/main-pipeline.png)\n\n## Components\n\n### Data Pipeline\n\nThe data pipeline component handles data processing tasks using Apache Spark and Apache Airflow.\n![etl-pipeline](public/etl-pipeline.png)\n\n- **DAGs**: Contains Airflow DAGs for orchestrating data workflows.\n- **Spark**: Contains Spark jobs for data processing.\n- **MinIO**: Used for object storage.\n- **Dependencies**: Managed using `requirements.txt`.\n\n\n### Back-End\n\nThe back-end component provides APIs and database management functionalities.\n![backend-pipeline](public/backend-pipeline.png)\n\n- **API**: Contains the API logic.\n- **DB**: Contains database-related scripts and configurations.\n- **Dependencies**: Managed using `requirements.txt`.\n\n\n### Next-Hub\n\nThe front-end application built with Next.js.\n![app-screenshot1](public/app-screenshot1.png)\n- **Components**: Reusable UI components.\n- **Utilities**: Helper functions and utilities.\n- **Configuration**: Next.js and Tailwind CSS configurations.\n- **Dependencies**: Managed using `package.json`.\n\n### Property Crawler Worker\n\nThe property crawler worker is responsible for web scraping property data.\n\n![crawler-system-pipeline](public/crawler-system-pipeline.png)\n\n- **Scripts**: Contains scripts for crawling different property websites.\n- **Dependencies**: Managed using `requirements.txt`.\n\n## Getting Started\n\n### Prerequisites\n\n- Docker\n- Docker Compose\n- Node.js (for Next.js application)\n\n### Setup\n\n1. **Clone the repository**:\n    ```sh\n    git clone https://github.com/vietdoo/sg-property-hub.git\n    cd sg-property-hub\n    ```\n\n2. **Back-End**:\n    ```sh\n    cd back-end\n    docker-compose up --build\n    ```\n\n3. **Data Pipeline**:\n    ```sh\n    cd data-pipeline\n    docker-compose up --build\n    ```\n\n4. **Next-Hub**:\n    ```sh\n    cd next-hub\n    npm install\n    npm run dev\n    ```\n\n5. **Property Crawler Worker**:\n    ```sh\n    cd property-crawler-worker\n    docker-compose up --build\n    ```\n\n## Usage\n\n### Back-End\n\nAccess the API at `http://localhost:8000`.\n\n### Data Pipeline\n\nAccess the Airflow UI at `http://localhost:8080`.\n\n### Next-Hub\n\nAccess the front-end application at `http://localhost:3000`.\n\n### Property Crawler Worker\n\nRun the crawler scripts as needed.\n\n## Contributing\n\nContributions are welcome! Please open an issue or submit a pull request.\n\n## License\n\nThis project is licensed under the MIT License.\n```\n\nFeel free to customize this README file further based on your specific requirements.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvietdoo%2Fsg-property-hub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvietdoo%2Fsg-property-hub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvietdoo%2Fsg-property-hub/lists"}