{"id":48908706,"url":"https://github.com/aojdevstudio/insurance-web-scraper","last_synced_at":"2026-04-16T22:03:56.959Z","repository":{"id":282778617,"uuid":"949626945","full_name":"AojdevStudio/insurance-web-scraper","owner":"AojdevStudio","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-16T22:24:14.000Z","size":108,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-16T22:27:25.461Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AojdevStudio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-16T21:25:35.000Z","updated_at":"2025-03-16T22:24:17.000Z","dependencies_parsed_at":"2025-03-16T22:37:44.398Z","dependency_job_id":null,"html_url":"https://github.com/AojdevStudio/insurance-web-scraper","commit_stats":null,"previous_names":["aojdevstudio/insurance-web-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AojdevStudio/insurance-web-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AojdevStudio%2Finsurance-web-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AojdevStudio%2Finsurance-web-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AojdevStudio%2Finsurance-web-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AojdevStudio%2Finsurance-web-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AojdevStudio","download_url":"https://codeload.github.com/AojdevStudio/insurance-web-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AojdevStudio%2Finsurance-web-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31905896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"ssl_error","status_checked_at":"2026-04-16T18:21:47.142Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-16T22:03:38.728Z","updated_at":"2026-04-16T22:03:56.948Z","avatar_url":"https://github.com/AojdevStudio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dental Insurance Guidelines Web Scraper\n\nA Python-based web scraper for extracting dental insurance guidelines from major carriers' provider portals.\n\n## Features\n\n- Automated extraction of dental insurance guidelines\n- Support for major carriers (Aetna, Cigna, MetLife, UHC)\n- PDF processing and text extraction\n- Structured data output in JSON format\n- Rate limiting and proxy support\n- Comprehensive error handling\n\n## Requirements\n\n- Python 3.11+\n- Docker (optional, for containerized deployment)\n- Access to provider portals\n\n## Quick Start\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/yourusername/insurance-web-scraper.git\n   cd insurance-web-scraper\n   ```\n\n2. Install dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. Configure your credentials:\n   ```bash\n   cp .env.example .env\n   # Edit .env with your credentials\n   ```\n\n4. Run the scraper:\n   ```bash\n   python -m scraper run --carrier aetna\n   ```\n\n## Documentation\n\nDetailed documentation is available in the `docs` directory:\n\n- [Installation Guide](docs/installation.md)\n- [Configuration Guide](docs/configuration.md)\n- [Usage Examples](docs/usage.md)\n- [Troubleshooting](docs/troubleshooting.md)\n- [API Documentation](docs/api.md)\n\n## Configuration\n\nThe scraper uses YAML configuration files for easy setup and maintenance. See the [Configuration Guide](docs/configuration.md) for details.\n\nExample configuration:\n```yaml\nscraper:\n  rate_limit: 5  # requests per second\n  output_dir: \"./data\"\n  proxy:\n    enabled: true\n    rotate_every: 60  # seconds\n\ncarriers:\n  aetna:\n    enabled: true\n    base_url: \"https://www.aetna.com/health-care-professionals\"\n  cigna:\n    enabled: true\n    base_url: \"https://www.cigna.com/healthcare-providers\"\n```\n\n## Support\n\nFor issues and feature requests, please use the [GitHub issue tracker](https://github.com/yourusername/insurance-web-scraper/issues).\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## System Requirements\n\n- Python 3.11+\n- Docker (optional for containerized deployment)\n- Docker Compose (optional for containerized deployment)\n- At least 4GB RAM available\n- At least 10GB disk space\n\n## Project Structure\n\n```\n.\n├── dental_scraper/         # Main Python package\n│   ├── spiders/           # Spider implementations for each carrier\n│   ├── scrapers/          # Base scraping functionality\n│   ├── models/            # Data models and validation\n│   ├── middlewares/       # Request middleware (rate limiting, etc.)\n│   ├── utils/             # Utility functions and helpers\n│   ├── app.py            # Application entry point\n│   ├── exceptions.py     # Custom exceptions\n│   └── pdf/              # PDF processing functionality\n├── tests/                 # Test suite\n│   └── spiders/          # Spider-specific tests\n├── data/                 # Data storage\n│   ├── raw/             # Raw downloaded PDFs\n│   └── processed/       # Processed data files\n├── logs/                 # Application logs\n└── cache/               # Cache storage\n```\n\n## Local Development Setup\n\n1. Create a virtual environment:\n   ```bash\n   python -m venv venv\n   source venv/bin/activate  # On Windows: venv\\Scripts\\activate\n   ```\n\n2. Install dependencies:\n   ```bash\n   pip install -r config/requirements.txt\n   ```\n\n## Docker Deployment\n\nThe system uses the following services when deployed with Docker:\n\n- **Scraper Service**: Python 3.11-based web scraper\n- **MongoDB**: Document storage for structured data\n- **Qdrant**: Vector database for semantic search\n- **Rotating Proxy**: TOR-based proxy rotation system\n\n### Environment Variables\n\n- `MONGO_URI`: MongoDB connection string\n- `QDRANT_HOST`: Qdrant service hostname\n- `QDRANT_PORT`: Qdrant service port\n- `TOR_PROXY`: TOR proxy connection string\n\n### Docker Setup\n\n1. Start the services:\n   ```bash\n   docker-compose up -d\n   ```\n\n2. Monitor the logs:\n   ```bash\n   docker-compose logs -f scraper\n   ```\n\n## Usage\n\n### Local Development\n```bash\npython src/main.py\n```\n\n### Testing\n\nRun tests with pytest:\n```bash\npytest tests/\n```\n\n## Resource Management\n\n### Docker Resource Limits\n- Scraper Service: 2GB RAM, 2 CPU cores\n- MongoDB: Uses default resource allocation\n- Qdrant: Uses default resource allocation\n- TOR Proxy: Uses default resource allocation\n\n### Network Configuration\n- Custom DNS servers: 8.8.8.8, 8.8.4.4\n- Bridge network mode for service isolation\n- TOR proxy with 5 instances and 60-second rotation\n\n## Maintenance\n\n- Logs are stored in the `logs` directory\n- PDFs are stored in the `data/raw` directory\n- Extracted data is stored in the `data/processed` directory\n- Cache is stored in the `cache` directory\n\n## Troubleshooting\n\n1. If the scraper service fails to start:\n   - Check if all required directories exist\n   - Verify MongoDB and Qdrant services are running (if using Docker)\n   - Check the logs for specific error messages\n\n2. If proxy rotation isn't working:\n   - Verify TOR service is running\n   - Check TOR logs for connection issues\n   - Ensure proxy settings are correctly configured\n\n3. For memory issues:\n   - Monitor container resource usage\n   - Adjust memory limits in docker-compose.yml\n   - Clear cache directory if needed\n\n## Contributing\n\n1. Create a new branch for your feature\n2. Write tests for new functionality\n3. Submit a pull request\n\n## License\n\nMIT License\n\nCopyright (c) 2024 AojdevStudio\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n## Authentication Configuration System\n\nThe authentication configuration system provides a secure and flexible way to manage authentication credentials and handle different authentication methods for web scraping tasks.\n\n### Features\n\n- Secure credential storage using system keyring and encrypted files\n- Support for multiple authentication methods:\n  - Basic Authentication\n  - Form-based Authentication\n  - Token-based Authentication\n- Configuration validation using Pydantic\n- Utility functions for handling CSRF tokens, form fields, and response validation\n- Comprehensive test coverage\n\n### Installation\n\nThe authentication system requires additional dependencies. Install them using:\n\n```bash\npip install -r requirements.txt\n```\n\n### Usage Examples\n\n#### Basic Authentication\n\n```python\nfrom auth import CredentialManager, AuthConfigManager, create_auth_handler\n\n# Initialize managers\ncredential_manager = CredentialManager(\"my_service\")\nconfig_manager = AuthConfigManager(credential_manager)\n\n# Add configuration\nconfig = config_manager.add_config(\"my_service\", {\n    \"auth_type\": \"basic\",\n    \"service_name\": \"my_service\",\n    \"base_url\": \"https://example.com\",\n    \"auth_url\": \"https://example.com/auth\"\n})\n\n# Create auth handler\nauth_handler = create_auth_handler(\"basic\", credential_manager)\n\n# Store credentials\ncredential_manager.store_credential(\"username\", \"my_username\")\ncredential_manager.store_credential(\"password\", \"my_password\")\n\n# Authenticate\nsuccess = auth_handler.authenticate(\n    username=credential_manager.get_credential(\"username\"),\n    password=credential_manager.get_credential(\"password\"),\n    url=config.auth_url\n)\n```\n\n#### Form Authentication\n\n```python\n# Add form auth configuration\nconfig = config_manager.add_config(\"my_service\", {\n    \"auth_type\": \"form\",\n    \"service_name\": \"my_service\",\n    \"base_url\": \"https://example.com\",\n    \"form_url\": \"https://example.com/login\",\n    \"form_data\": {\n        \"remember_me\": \"true\"\n    }\n})\n\n# Create form auth handler\nauth_handler = create_auth_handler(\"form\", credential_manager)\n\n# Authenticate with form\nsuccess = auth_handler.authenticate(\n    username=credential_manager.get_credential(\"username\"),\n    password=credential_manager.get_credential(\"password\"),\n    form_url=config.form_url,\n    form_data=config.form_data\n)\n```\n\n#### Token Authentication\n\n```python\n# Add token auth configuration\nconfig = config_manager.add_config(\"my_service\", {\n    \"auth_type\": \"token\",\n    \"service_name\": \"my_service\",\n    \"base_url\": \"https://example.com\",\n    \"token_url\": \"https://example.com/token\",\n    \"token_field\": \"access_token\",\n    \"expiry_field\": \"expires_in\"\n})\n\n# Create token auth handler\nauth_handler = create_auth_handler(\"token\", credential_manager)\n\n# Authenticate and get token\nsuccess = auth_handler.authenticate(\n    username=credential_manager.get_credential(\"username\"),\n    password=credential_manager.get_credential(\"password\"),\n    token_url=config.token_url\n)\n```\n\n### Security Considerations\n\n1. Credentials are stored securely using the system keyring when available\n2. Fallback to encrypted file storage when keyring is not available\n3. Sensitive data is never logged or stored in plain text\n4. SSL certificate verification is enabled by default\n5. CSRF tokens are automatically handled for form authentication\n6. Token expiration is tracked and handled appropriately\n\n### Testing\n\nRun the test suite:\n\n```bash\npytest tests/auth/\n``` \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faojdevstudio%2Finsurance-web-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faojdevstudio%2Finsurance-web-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faojdevstudio%2Finsurance-web-scraper/lists"}