{"id":37074469,"url":"https://github.com/bradsjm/byte-blaster","last_synced_at":"2026-01-14T08:47:02.992Z","repository":{"id":297726327,"uuid":"997724112","full_name":"bradsjm/byte-blaster","owner":"bradsjm","description":"A comprehensive Python implementation of the EMWIN Quick Block Transfer (QBT) protocol for receiving real-time weather data from the National Weather Service. This protocol is also commonly known as ByteBlaster.","archived":false,"fork":false,"pushed_at":"2025-07-05T12:39:35.000Z","size":332,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-04T07:53:03.794Z","etag":null,"topics":["byteblaster","emwin","noaa","quickblocktransfer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bradsjm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-07T03:38:48.000Z","updated_at":"2025-09-22T19:26:55.000Z","dependencies_parsed_at":"2025-06-07T04:27:30.096Z","dependency_job_id":"b67df7cb-e438-462f-97a4-0a3b2ae23cc1","html_url":"https://github.com/bradsjm/byte-blaster","commit_stats":null,"previous_names":["bradsjm/byte-blaster"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bradsjm/byte-blaster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradsjm%2Fbyte-blaster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradsjm%2Fbyte-blaster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradsjm%2Fbyte-blaster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradsjm%2Fbyte-blaster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bradsjm","download_url":"https://codeload.github.com/bradsjm/byte-blaster/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bradsjm%2Fbyte-blaster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28414693,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:38:59.149Z","status":"ssl_error","status_checked_at":"2026-01-14T08:38:43.588Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["byteblaster","emwin","noaa","quickblocktransfer"],"created_at":"2026-01-14T08:47:02.305Z","updated_at":"2026-01-14T08:47:02.980Z","avatar_url":"https://github.com/bradsjm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EMWIN QBT Client (ByteBlaster)\n\nA comprehensive Python implementation of the **EMWIN Quick Block Transfer (QBT)** protocol for receiving real-time weather data from the National Weather Service. This protocol is also commonly known as **ByteBlaster**.\n\n## Requirements\n\n- **Python 3.12+** (Python 3.13 recommended for best performance)\n- Zero runtime dependencies - pure Python implementation\n\n## Overview\n\nThe EMWIN QBT protocol uses Quick Block Transfer (QBT) to efficiently deliver weather data, forecasts, warnings, and satellite imagery. This Python client provides a robust, asynchronous implementation with automatic reconnection, server failover, and comprehensive error handling.\n\n## Features\n\n- **Complete Protocol Implementation**: Full support for EMWIN QBT protocol (v1 and v2)\n- **High-Level File Manager**: Simplified interface for receiving complete, reconstructed files, abstracting away low-level data segments.\n- **Async Iterator Support**: Modern async/await patterns with `stream_files()` and `stream_segments()` methods\n- **Automatic Reconnection**: Intelligent failover across multiple servers\n- **Server List Management**: Dynamic server list updates with persistence\n- **Async/Await Support**: Built on asyncio for high performance\n- **Data Validation**: Checksum verification and error detection\n- **Compression Support**: Automatic handling of zlib-compressed v2 data\n- **Watchdog Monitoring**: Connection health monitoring with configurable timeouts\n- **Observable Pattern**: Easy subscription to data events\n- **Production Ready**: Comprehensive logging, error handling, and type hints\n\n## Installation\n\n### From PyPI (Recommended)\n\nThe package is available on PyPI and works with all modern Python package managers:\n\n#### Using pip\n```bash\n# Install latest version\npip install byte-blaster\n\n# Install with optional dependencies\npip install byte-blaster[dev,test]\n```\n\n#### Using UV (Fastest)\n```bash\n# Install in current environment\nuv pip install byte-blaster\n\n# Create new project with byte-blaster\nuv init my-weather-app\ncd my-weather-app\nuv add byte-blaster\n\n# Install with dependency groups\nuv sync --group dev --group test\n```\n\n#### Using Poetry\n```bash\n# Add to existing project\npoetry add byte-blaster\n\n# Add development dependencies\npoetry add --group dev byte-blaster[dev]\npoetry add --group test byte-blaster[test]\n```\n\n#### Using PDM\n```bash\n# Add to project\npdm add byte-blaster\n\n# Add with dependency groups\npdm add -dG dev byte-blaster[dev]\npdm add -dG test byte-blaster[test]\n```\n\n#### Using Conda/Mamba\n```bash\n# Install from conda-forge (if available) or pip\nconda install byte-blaster\n# or\nconda install pip \u0026\u0026 pip install byte-blaster\n```\n\n### From Source (Development)\n\n```bash\n# Clone the repository\ngit clone \u003crepository-url\u003e\ncd byte-blaster\n\n# Using UV (recommended for development)\nuv sync --group dev --group test\nsource .venv/bin/activate  # or `.venv\\Scripts\\activate` on Windows\n\n# Using pip\npip install -e .\npip install -e \".[dev,test]\"\n\n# Using Poetry\npoetry install --with dev,test\n\n# Using PDM\npdm install -dG dev -dG test\n```\n\n### Verify Installation\n\n```bash\n# Run comprehensive verification script\npython scripts/verify_installation.py\n\n# Test basic import\npython -c \"from byteblaster import ByteBlasterClient; print('✅ Installation successful')\"\n\n# Run examples\npython examples/example.py\n```\n\n### Package Manager Compatibility\n\nThis project uses modern Python packaging standards and is compatible with:\n\n| Package Manager | Version | Installation Command | Dependency Groups |\n|-----------------|---------|---------------------|-------------------|\n| **pip** | 21.3+ | `pip install byte-blaster` | `pip install byte-blaster[dev,test]` |\n| **UV** | 0.1.0+ | `uv add byte-blaster` | `uv sync --group dev --group test` |\n| **Poetry** | 1.2+ | `poetry add byte-blaster` | `poetry install --with dev,test` |\n| **PDM** | 2.0+ | `pdm add byte-blaster` | `pdm install -dG dev -dG test` |\n| **Conda** | Any | `pip install byte-blaster` | Via pip in conda env |\n| **Pipenv** | Any | `pipenv install byte-blaster` | `pipenv install --dev` |\n\n**Requirements:**\n- Python 3.12+ (Python 3.13 recommended)\n- Modern package manager supporting PEP 621 (pyproject.toml)\n- No system dependencies required\n\n**Features:**\n- ✅ Pure Python (no compiled extensions)\n- ✅ Type hints included (`py.typed` marker)\n- ✅ Dependency groups for dev/test separation\n- ✅ Wheel and source distributions available\n- ✅ Cross-platform compatibility (Windows, macOS, Linux)\n- ✅ PEP 621 compliant (pyproject.toml-based configuration)\n- ✅ PEP 517/518 build system (setuptools backend)\n- ✅ PEP 440 version scheme\n- ✅ PEP 508 dependency specifications\n\n### Troubleshooting Installation\n\nIf you encounter issues with specific package managers:\n\n#### UV Issues\n```bash\n# Update UV to latest version\nuv self update\n\n# Clear cache and retry\nuv cache clean\nuv add byte-blaster\n\n# Force reinstall\nuv pip install --force-reinstall byte-blaster\n```\n\n#### pip Issues\n```bash\n# Update pip and build tools\npip install --upgrade pip setuptools wheel\n\n# Install with verbose output for debugging\npip install -v byte-blaster\n\n# Clear pip cache\npip cache purge\n```\n\n#### Poetry Issues\n```bash\n# Update Poetry\npoetry self update\n\n# Clear cache\npoetry cache clear pypi --all\n\n# Force update lock file\npoetry lock --no-update\npoetry install\n```\n\n#### General Issues\n- **Python Version**: Ensure Python 3.12+ is installed\n- **Virtual Environment**: Always use a virtual environment\n- **Dependencies**: Check for conflicting packages with `pip check`\n- **Permissions**: Use `--user` flag if encountering permission errors\n- **Network**: Check proxy settings if behind corporate firewall\n\n### Package Manager Summary\n\nByteBlaster is designed to work seamlessly with all modern Python package managers:\n\n| Feature | Status | Notes |\n|---------|--------|-------|\n| **pip compatibility** | ✅ Full | Standard `pip install byte-blaster` |\n| **UV compatibility** | ✅ Full | Fastest installation with `uv add byte-blaster` |\n| **Poetry compatibility** | ✅ Full | `poetry add byte-blaster` |\n| **PDM compatibility** | ✅ Full | `pdm add byte-blaster` |\n| **Conda compatibility** | ✅ Via pip | `pip install byte-blaster` in conda env |\n| **Pipenv compatibility** | ✅ Full | `pipenv install byte-blaster` |\n| **Zero dependencies** | ✅ Yes | Pure Python, no runtime dependencies |\n| **Type hints** | ✅ Full | Complete type annotations included |\n| **Wheels available** | ✅ Yes | Fast binary installation |\n| **Source builds** | ✅ Yes | `pip install` from source works |\n\n**Recommended installation methods:**\n- **For new projects**: `uv add byte-blaster` (fastest)\n- **For existing pip projects**: `pip install byte-blaster`\n- **For Poetry projects**: `poetry add byte-blaster`\n- **For development**: `uv sync --group dev --group test` (after cloning)\n\n## Quick Start\n\n```python\nimport asyncio\nfrom byteblaster import ByteBlasterClientOptions, ByteBlasterFileManager, CompletedFile\n\nasync def handle_file(file: CompletedFile):\n    \"\"\"Handler for completed files.\"\"\"\n    print(f\"Received file: {file.filename}, Size: {len(file.data)} bytes\")\n\nasync def main():\n    # Create client options - email is required for authentication\n    options = ByteBlasterClientOptions(email=\"your-email@example.com\")\n    \n    # Use the high-level file manager, the recommended client for most use cases.\n    manager = ByteBlasterFileManager(options)\n    \n    # Subscribe to completed file events\n    manager.subscribe(handle_file)\n    \n    # Start receiving data\n    await manager.start()\n    \n    # Keep running (use Ctrl+C to stop)\n    try:\n        # Wait indefinitely until the program is interrupted\n        await asyncio.Event().wait()\n    except asyncio.CancelledError:\n        print(\"\\nClient shutting down...\")\n    finally:\n        await manager.stop()\n        print(\"Client stopped.\")\n\nif __name__ == \"__main__\":\n    try:\n        asyncio.run(main())\n    except KeyboardInterrupt:\n        pass\n```\n\n## Async Iterator Patterns\n\nThe library supports modern async iterator patterns for reactive programming:\n\n### File Streaming\n\n```python\nimport asyncio\nfrom byteblaster import ByteBlasterClientOptions, ByteBlasterFileManager\n\nasync def main():\n    options = ByteBlasterClientOptions(email=\"your-email@example.com\")\n    manager = ByteBlasterFileManager(options)\n    \n    await manager.start()\n    \n    # Use async iterator pattern\n    async with manager.stream_files() as files:\n        async for completed_file in files:\n            print(f\"Processing: {completed_file.filename}\")\n            # Process file here\n    \n    await manager.stop()\n\nasyncio.run(main())\n```\n\n### Segment Streaming\n\n```python\nimport asyncio\nfrom byteblaster import ByteBlasterClientOptions, ByteBlasterClient\n\nasync def main():\n    options = ByteBlasterClientOptions(email=\"your-email@example.com\")\n    client = ByteBlasterClient(options)\n    \n    await client.start()\n    \n    # Stream individual segments\n    async with client.stream_segments() as segments:\n        async for segment in segments:\n            print(f\"Segment: {segment.filename} ({segment.block_number}/{segment.total_blocks})\")\n    \n    await client.stop()\n\nasyncio.run(main())\n```\n\n## Complete Example\n\nSee `examples/example.py` for a comprehensive example that demonstrates:\n- File reconstruction from multiple segments\n- Data persistence to disk\n- Graceful shutdown handling\n- Progress monitoring\n- Concurrent processing with multiple handlers\n\n```bash\npython examples/example.py\n```\n\nFor advanced async iterator examples, see:\n```bash\npython examples/example_async_iterators.py\n```\n\n## Protocol Details\n\n### Authentication\n- Uses email-based authentication with XOR obfuscation\n- Automatic re-authentication every 2 minutes\n- Configurable authentication parameters\n\n### Data Flow\n1. **Connection**: Client connects to ByteBlaster servers\n2. **Authentication**: Sends XOR-encoded logon message\n3. **Data Reception**: Receives and decodes data segments\n4. **Reconstruction**: Assembles complete files from segments\n5. **Validation**: Verifies checksums and data integrity\n\n### Frame Types\n- **Data Blocks**: Weather data, forecasts, imagery\n- **Server Lists**: Dynamic server list updates\n\n## API Reference\n\n### ByteBlasterFileManager\n\nThe recommended high-level client for most use cases. It abstracts away segment handling and provides a simple interface for receiving complete files.\n\n```python\nfrom byteblaster import ByteBlasterFileManager, ByteBlasterClientOptions\n\n# Configure the client with your email\noptions = ByteBlasterClientOptions(email=\"your-email@example.com\")\n\n# Create the file manager\nmanager = ByteBlasterFileManager(options)\n```\n\n#### Methods\n\n- `subscribe(handler)`: Subscribe to `CompletedFile` events.\n- `unsubscribe(handler)`: Remove event subscription.\n- `stream_files(max_queue_size=100)`: Create async iterator for streaming files.\n- `start()`: Start the client (async).\n- `stop(shutdown_timeout=None)`: Stop the client (async).\n\n#### Properties\n- `client`: Access the underlying `ByteBlasterClient` instance.\n- `assembler`: Access the `FileAssembler` instance.\n\n### ByteBlasterClient\n\nThe low-level client for handling the EMWIN QBT protocol. Use this if you need to work directly with data segments instead of complete files.\n\n```python\nfrom byteblaster import ByteBlasterClient, ByteBlasterClientOptions\n\nclient = ByteBlasterClient(\n    options=ByteBlasterClientOptions(\n        email=\"user@example.com\",\n        server_list_path=\"servers.json\",      # Server persistence file\n        watchdog_timeout=20.0,                # Connection timeout\n        max_exceptions=10,                    # Max errors before reconnect\n        reconnect_delay=5.0,                  # Delay between reconnects\n        connection_timeout=10.0,              # TCP connection timeout\n    )\n)\n```\n\n#### Methods\n\n- `subscribe(handler)`: Subscribe to data segment events.\n- `unsubscribe(handler)`: Remove event subscription.\n- `stream_segments(max_queue_size=1000)`: Create async iterator for streaming segments.\n- `start()`: Start the client (async).\n- `stop(shutdown_timeout=None)`: Stop the client (async).\n- `get_server_list()`: Get the current `ByteBlasterServerList` instance.\n\n#### Properties\n\n- `is_connected`: Connection status.\n- `is_running`: Client running status.\n- `server_count`: Number of available servers.\n- `email`: Authentication email.\n\n### ByteBlasterClientOptions\n\nConfiguration class for client initialization.\n\n```python\nfrom byteblaster import ByteBlasterClientOptions\n\noptions = ByteBlasterClientOptions(\n    email=\"user@example.com\",                 # Required: Email for authentication\n    server_list_path=\"servers.json\",          # Server persistence file path\n    watchdog_timeout=20.0,                    # Connection watchdog timeout\n    max_exceptions=10,                        # Max errors before reconnect\n    reconnect_delay=5.0,                      # Delay between reconnection attempts\n    connection_timeout=10.0,                  # TCP connection establishment timeout\n)\n```\n\n### QBTSegment\n\nData structure representing a single QBT data block.\n\n```python\n@dataclass\nclass QBTSegment:\n    filename: str           # Original filename\n    block_number: int       # Block sequence number\n    total_blocks: int       # Total blocks in file\n    content: bytes          # Block data\n    checksum: int           # Block checksum\n    length: int             # Block length\n    version: int            # Protocol version (1 or 2)\n    timestamp: datetime     # File timestamp\n    received_at: datetime   # Reception timestamp\n    header: str             # Raw header\n    source: str             # Server address\n```\n\n### CompletedFile\n\nData structure representing a fully reconstructed file.\n\n```python\nclass CompletedFile(NamedTuple):\n    filename: str    # Original filename\n    data: bytes      # Complete file content\n```\n\n### Async Iterators\n\nThe library provides async iterator classes for streaming data:\n\n- `FileStream`: Async iterator for completed files (from `stream_files()`)\n- `SegmentStream`: Async iterator for data segments (from `stream_segments()`)\n\nBoth support context manager protocol and backpressure handling.\n\n### Server Management\n\nThe client automatically manages server lists. You can access the list through the client instance:\n\n```python\n# Get current server list from the file manager\nserver_list = manager.client.get_server_list()\n\n# Or directly from a client instance\n# server_list = client.get_server_list()\n```\n\n## Configuration\n\n### Server List Persistence\n\nServer lists are automatically saved to `servers.json` (configurable):\n\n```json\n{\n  \"servers\": [\n    \"w2.2y.net:2211\",\n    \"2.pool.iemwin.net:2211\"\n  ],\n  \"sat_servers\": [],\n  \"received_at\": \"2024-01-01T12:00:00\",\n  \"version\": \"1.0\"\n}\n```\n\n### Logging\n\nConfigure logging to monitor client behavior:\n\n```python\nimport logging\n\nlogging.basicConfig(\n    level=logging.INFO,\n    format=\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\"\n)\n```\n\n## Protocol Specification\n\nBased on the EMWIN ByteBlaster protocol:\n\n- **Frame Sync**: 6 consecutive 0xFF bytes (for TCP stream)\n- **XOR Encoding**: All data XOR'ed with 0xFF\n- **Header Format**: 80-byte ASCII header with metadata\n- **Compression**: V2 uses zlib compression\n- **Checksum**: Simple byte sum validation\n\n### Header Format\n\n```\n/PF\u003cfilename\u003e /PN \u003cblock_num\u003e /PT \u003ctotal_blocks\u003e /CS \u003cchecksum\u003e /FD\u003cdate\u003e [/DL\u003clength\u003e]\n```\n\n## Error Handling\n\nThe client includes comprehensive error handling:\n\n- **Connection Errors**: Automatic reconnection with exponential backoff\n- **Protocol Errors**: State machine reset and resynchronization\n- **Data Errors**: Checksum validation and corruption detection\n- **Timeout Handling**: Watchdog monitoring with configurable timeouts\n\n## Testing\n\n```bash\n# Run tests\npytest\n\n# Run with coverage\npytest --cov=byteblaster\n\n# Run specific test categories\npytest -m unit          # Unit tests only\npytest -m integration   # Integration tests only\n```\n\n## Development\n\n### Code Quality\n\nThis project follows strict code quality standards:\n\n- **Type Hints**: Complete type annotations with Python 3.12+ syntax\n- **Linting**: Ruff for code formatting and linting\n- **Type Checking**: Pyright/basedpyright for static analysis\n- **Testing**: Comprehensive test suite with pytest\n\n```bash\n# Format code\nruff format .\n\n# Check code quality\nruff check --fix .\n\n# Type checking\nbasedpyright\n```\n\n### Architecture\n\nThe client is built with a modular architecture:\n\n- `client.py`: Main client implementation\n- `file_manager.py`: High-level file management interface\n- `protocol/`: Protocol implementation\n  - `decoder.py`: State machine decoder\n  - `models.py`: Data models\n  - `auth.py`: Authentication handling\n- `utils/`: Utility functions\n  - `crypto.py`: XOR encoding and compression\n  - `serverlist.py`: Server list management\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests for new functionality\n5. Ensure all tests pass\n6. Submit a pull request\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## References\n\n- [EMWIN Protocol Documentation](https://www.weather.gov/emwin/)\n- [Quick Block Transfer Specification](https://www.nws.noaa.gov/emwin/winpro.htm)\n- [NOAAPort and EMWIN](http://www.noaaport.net/)\n\n## Support\n\nFor questions, issues, or contributions:\n\n- Create an issue on GitHub\n- Check the examples and documentation\n- Review the comprehensive logging output\n\n---\n\n## Acknowledgments\n\n### AI Assistance\n\nThis project has been developed with assistance from Large Language Models (LLMs), and we acknowledge their significant contributions to both the codebase and documentation:\n\n- **Anthropic Claude** - Contributed to code architecture, implementation patterns, documentation structure, async/await patterns, error handling strategies, and comprehensive testing approaches\n- **OpenAI GPT** - Assisted with protocol implementation details, API design decisions, code optimization suggestions, and example development\n- **Google Gemini** - Provided insights on Python best practices, type annotation improvements, and packaging standards compliance\n\nThe LLMs have been instrumental in:\n- **Code Quality**: Implementing modern Python 3.12+ features, type hints, and async patterns\n- **Documentation**: Creating comprehensive README, API documentation, and example code\n- **Architecture**: Designing modular, testable, and maintainable code structure\n- **Standards Compliance**: Ensuring adherence to PEP standards and modern packaging practices\n- **Error Handling**: Implementing robust error recovery and logging strategies\n\nWhile AI has significantly accelerated development and improved code quality, all code has been reviewed, tested, and validated by human developers. The final implementation decisions, architecture choices, and quality standards remain under human oversight.\n\n### Human Contributors\n\nWe also acknowledge the human developers, domain experts, and community members who have contributed to the project through code review, testing, feedback, and domain expertise in weather data protocols.\n\n---\n\n**Note**: Replace `your-email@example.com` with your actual email address when using the client. Some ByteBlaster servers may require registration.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbradsjm%2Fbyte-blaster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbradsjm%2Fbyte-blaster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbradsjm%2Fbyte-blaster/lists"}