{"id":15154321,"url":"https://github.com/mirvald-space/haraj_scrap","last_synced_at":"2025-07-25T04:09:50.157Z","repository":{"id":249665906,"uuid":"832174807","full_name":"mirvald-space/haraj_scrap","owner":"mirvald-space","description":"It handles asynchronous data fetching, city-based search filtering, and storage in a MongoDB database, supporting real-time updates and easy front-end integration.","archived":false,"fork":false,"pushed_at":"2024-08-17T12:25:03.000Z","size":54,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-09T12:33:46.295Z","etag":null,"topics":["async","asynchronous","asyncio","db","fastapi","mongo","python","scraping","semphore"],"latest_commit_sha":null,"homepage":"http://haraj.com.sa","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mirvald-space.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-22T13:31:02.000Z","updated_at":"2024-08-17T12:25:06.000Z","dependencies_parsed_at":"2024-07-22T16:12:21.885Z","dependency_job_id":"c5c1665e-28f9-44b9-a2af-f682234553bc","html_url":"https://github.com/mirvald-space/haraj_scrap","commit_stats":null,"previous_names":["mirvald-design/haraj_scrap"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirvald-space%2Fharaj_scrap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirvald-space%2Fharaj_scrap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirvald-space%2Fharaj_scrap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirvald-space%2Fharaj_scrap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mirvald-space","download_url":"https://codeload.github.com/mirvald-space/haraj_scrap/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247866127,"owners_count":21009238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","asynchronous","asyncio","db","fastapi","mongo","python","scraping","semphore"],"created_at":"2024-09-26T17:21:44.705Z","updated_at":"2025-04-08T14:46:52.007Z","avatar_url":"https://github.com/mirvald-space.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Haraj.com.sa Scraper Backend\n\nThis project is a backend service for scraping and managing data from Haraj.com.sa. It provides an API for searching posts, retrieving specific posts, and updating the database with new data.\n\n## Features\n\n- Asynchronous data fetching from Haraj.com.sa API\n- MongoDB integration for data storage\n- FastAPI-based RESTful API\n- Scheduled background tasks for data updates\n- City-based filtering and pagination support\n\n## Prerequisites\n\n- Python 3.7+\n- MongoDB\n- pip (Python package manager)\n\n## Installation\n\n1. Clone the repository:\n\n   ```\n   git clone https://github.com/yourusername/haraj-scraper.git\n   cd haraj-scraper\n   ```\n\n2. Create a virtual environment:\n\n   ```\n   python -m venv .myenv\n   source .myenv/bin/activate  # On Windows use `.myenv\\Scripts\\activate`\n   ```\n\n3. Install dependencies:\n\n   ```\n   pip install -r requirements.txt\n   ```\n\n4. Set up environment variables:\n   Create a `.env` file in the root directory with the following content:\n   ```\n   MONGODB_URI=your_mongodb_connection_string\n   DB_NAME=haraj-db\n   API_URL=https://graphql.haraj.com.sa/?queryName=search\u0026lang=en\u0026clientId=5PfyC2s3-xlv8-ManX-RCdf-LqhFYr5SkUh3v3\u0026version=N9.0.38%20,%206/28/2024/\n   PORT=8000\n   ```\n\n## Running Locally\n\nTo run the server locally:\n\n```\npython main.py\n```\n\nThe server will start on `http://localhost:8000`.\n\n## API Endpoints\n\n### Root Endpoint\n\n- `GET /`: Root endpoint to check if the server is running\n\n  Example request:\n\n  ```\n  curl http://localhost:8000/\n  ```\n\n  Example response:\n\n  ```json\n  {\n  \t\"message\": \"The server is running successfully\"\n  }\n  ```\n\n### Search Posts\n\n- `GET /search/`: Search for posts with optional city filtering and pagination\n\n  Example request:\n\n  ```\n  curl \"http://localhost:8000/search/?query=car\u0026city=Riyadh\u0026page=1\u0026limit=10\"\n  ```\n\n  Example response (when data is found):\n\n  ```json\n  {\n  \t\"posts\": [\n  \t\t{\n  \t\t\t\"id\": \"140507182\",\n  \t\t\t\"bodyHTML\": \"\u003cp\u003eCar for sale in excellent condition...\u003c/p\u003e\",\n  \t\t\t\"title\": \"2020 Toyota Camry for Sale\",\n  \t\t\t\"URL\": \"https://haraj.com.sa/en/11140507182/toyota_camry_2020\",\n  \t\t\t\"city\": \"Riyadh\",\n  \t\t\t\"postDate\": \"2024-07-23T10:22:23\",\n  \t\t\t\"updateDate\": \"2024-07-23T10:22:23\",\n  \t\t\t\"firstImage\": \"https://mimg6cdn.haraj.com.sa/userfiles30/2024-07-23/1350x1800_CE6D1043-6A81-44F8-B36B-818BF2E1922C.jpg\",\n  \t\t\t\"commentCount\": 0,\n  \t\t\t\"tags\": [\"car\", \"toyota\", \"camry\"],\n  \t\t\t\"authorUsername\": \"cardealer123\",\n  \t\t\t\"price\": {\n  \t\t\t\t\"formattedPrice\": \"50,000 SAR\"\n  \t\t\t}\n  \t\t}\n  \t\t// ... more posts\n  \t],\n  \t\"is_complete\": true,\n  \t\"total_count\": 100,\n  \t\"current_page\": 1,\n  \t\"total_pages\": 10,\n  \t\"has_previous_page\": false,\n  \t\"has_next_page\": true,\n  \t\"message\": \"Data successfully retrieved.\"\n  }\n  ```\n\n  Example response (when no data is found):\n\n  ```json\n  {\n  \t\"posts\": [],\n  \t\"is_complete\": true,\n  \t\"total_count\": 0,\n  \t\"current_page\": 1,\n  \t\"total_pages\": 0,\n  \t\"has_previous_page\": false,\n  \t\"has_next_page\": false,\n  \t\"message\": \"No posts available for this query.\"\n  }\n  ```\n\n### Get Specific Post\n\n- `GET /post/{post_id}`: Retrieve a specific post by ID\n\n  Example request:\n\n  ```\n  curl http://localhost:8000/post/140507182\n  ```\n\n  Example response:\n\n  ```json\n  {\n  \t\"id\": \"140507182\",\n  \t\"bodyHTML\": \"\u003cp\u003eCar for sale in excellent condition...\u003c/p\u003e\",\n  \t\"title\": \"2020 Toyota Camry for Sale\",\n  \t\"URL\": \"https://haraj.com.sa/en/11140507182/toyota_camry_2020\",\n  \t\"city\": \"Riyadh\",\n  \t\"postDate\": \"2024-07-23T10:22:23\",\n  \t\"updateDate\": \"2024-07-23T10:22:23\",\n  \t\"firstImage\": \"https://mimg6cdn.haraj.com.sa/userfiles30/2024-07-23/1350x1800_CE6D1043-6A81-44F8-B36B-818BF2E1922C.jpg\",\n  \t\"commentCount\": 0,\n  \t\"tags\": [\"car\", \"toyota\", \"camry\"],\n  \t\"authorUsername\": \"cardealer123\",\n  \t\"price\": {\n  \t\t\"formattedPrice\": \"50,000 SAR\"\n  \t}\n  }\n  ```\n\n### Update Database\n\n- `POST /update`: Trigger a manual database update (background task)\n\n  Example request:\n\n  ```\n  curl -X POST http://localhost:8000/update\n  ```\n\n  Example response:\n\n  ```json\n  {\n  \t\"message\": \"Database update initiated\"\n  }\n  ```\n\n## Deployment\n\nTo deploy on a server:\n\n1. Set up a MongoDB instance (e.g., MongoDB Atlas)\n2. Clone the repository on your server\n3. Install dependencies: `pip install -r requirements.txt`\n4. Set up environment variables (see Installation step 4)\n5. Run the application using a process manager like PM2 or supervisord:\n   ```\n   pm2 start main.py --interpreter python3\n   ```\n\n## Development\n\n- The project structure is as follows:\n  ```\n  scrapper/\n  ├── main.py\n  ├── config.py\n  ├── db/\n  │   ├── database.py\n  │   └── models.py\n  └── utils/\n      ├── routes.py\n      ├── handlers.py\n      └── services.py\n  ```\n- To add new features, extend the `PostService` class in `utils/services.py`\n- Add new routes in `utils/routes.py`\n- Modify database operations in `db/database.py`\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirvald-space%2Fharaj_scrap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmirvald-space%2Fharaj_scrap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirvald-space%2Fharaj_scrap/lists"}