{"id":27085467,"url":"https://github.com/mindscope-world/global-data-scraping","last_synced_at":"2025-10-26T19:18:37.044Z","repository":{"id":285514766,"uuid":"958391375","full_name":"mindscope-world/Global-Data-Scraping","owner":"mindscope-world","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-01T07:21:37.000Z","size":81,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-01T07:23:02.882Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mindscope-world.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-04-01T05:58:20.000Z","updated_at":"2025-04-01T07:21:40.000Z","dependencies_parsed_at":"2025-04-01T07:23:06.197Z","dependency_job_id":"77f5925c-00fc-4273-9633-ac6d92747c27","html_url":"https://github.com/mindscope-world/Global-Data-Scraping","commit_stats":null,"previous_names":["mindscope-world/global-data-scraping"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindscope-world%2FGlobal-Data-Scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindscope-world%2FGlobal-Data-Scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindscope-world%2FGlobal-Data-Scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindscope-world%2FGlobal-Data-Scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mindscope-world","download_url":"https://codeload.github.com/mindscope-world/Global-Data-Scraping/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247436116,"owners_count":20938532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-06T04:37:15.071Z","updated_at":"2025-10-26T19:18:36.957Z","avatar_url":"https://github.com/mindscope-world.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# World Disaster Center API\n\nA FastAPI application that provides real-time information about global disasters, funding appeals, and affected populations using AI-powered web crawling and structured data extraction.\n\n## Features\n\n- **Real-time Disaster Information**: Crawls and analyzes recent disaster reports from around the world\n- **Structured Data Extraction**: Uses AI to extract key information about:\n  - Disaster types and locations\n  - Affected populations and casualties\n  - Funding appeals and humanitarian response plans\n- **Multi-Region Coverage**: Implements 20 specialized search queries targeting diverse disaster types and regions\n- **Priority Source Handling**: Prioritizes authoritative humanitarian sources (UNOCHA, UNHCR, WHO, etc.)\n- **Comprehensive API Endpoints**:\n  - `/disasters/by_country/{country_name}`: Get recent disasters for a specific country\n  - `/funding/stats`: Get aggregated funding statistics from recent reports\n  - `/people/stats`: Get aggregated affected population statistics\n  - `/chat`: Conversational AI assistant for disaster information\n\n## Prerequisites\n\n1. Python 3.8 or higher\n2. OpenAI API key\n3. (Optional) Docker for local containerization\n\n## Local Development\n\n1. Clone this repository and navigate to the project directory\n\n2. Create a virtual environment and activate it:\n   ```bash\n   python -m venv venv\n   # Windows\n   .\\venv\\Scripts\\activate\n   # Linux/MacOS\n   source venv/bin/activate\n   ```\n\n3. Install dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. Create a `.env` file with your OpenAI API key:\n   ```\n   OPENAI_API_KEY=your-openai-api-key\n   ```\n\n5. Run the FastAPI application:\n   ```bash\n   uvicorn app:app --reload\n   ```\n\n6. Access the API documentation at http://localhost:8000/docs\n\n## Docker Deployment\n\n```bash\n# Build the Docker image\ndocker build -t disaster-api .\n\n# Run the container\ndocker run -p 8080:8080 -e OPENAI_API_KEY=your-openai-api-key disaster-api\n\n# Access the API at http://localhost:8080/docs\n```\n\n## Deployment to Render\n\nThe application is configured for deployment on Render using the provided `render.yaml`:\n\n1. Push your code to a Git repository (GitHub, GitLab, etc.)\n\n2. Create a new Web Service on Render:\n   - Connect your repository\n   - Select the Docker environment\n   - Add your environment variables:\n     ```\n     OPENAI_API_KEY=your-openai-api-key\n     ```\n\n3. Deploy! Render will automatically build and deploy your application\n\n## API Usage\n\n### Get Recent Disasters by Country\n```http\nGET /disasters/by_country/{country_name}\n```\n\n### Get Funding Statistics\n```http\nGET /funding/stats\n```\n\n### Get Affected People Statistics\n```http\nGET /people/stats\n```\n\n### Chat with Disaster Assistant\n```http\nPOST /chat\nContent-Type: application/json\n\n{\n    \"message\": \"What are the recent disasters in Japan?\"\n}\n```\n\n## Data Quality and Limitations\n\n- Data is sampled from crawled web pages and may be incomplete\n- Statistics are not comprehensive global figures\n- Information accuracy depends on source availability and AI interpretation\n- Always verify critical information through official channels\n\n## Troubleshooting\n\nIf you encounter issues:\n\n1. Check the application logs for error messages\n2. Verify your OpenAI API key is correctly set\n3. Ensure all required dependencies are installed\n4. Check for rate limiting issues from search providers\n5. Verify network connectivity for web crawling\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Commit your changes\n4. Push to the branch\n5. Create a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindscope-world%2Fglobal-data-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmindscope-world%2Fglobal-data-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindscope-world%2Fglobal-data-scraping/lists"}