{"id":26925375,"url":"https://github.com/xatuke/czds-service","last_synced_at":"2025-04-02T02:36:30.987Z","repository":{"id":285561464,"uuid":"942910078","full_name":"xatuke/czds-service","owner":"xatuke","description":"CZDS Zone Database Service","archived":false,"fork":false,"pushed_at":"2025-03-05T09:04:11.000Z","size":58,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-01T12:57:56.264Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xatuke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-04T21:50:16.000Z","updated_at":"2025-03-25T08:54:36.000Z","dependencies_parsed_at":"2025-04-01T12:57:59.359Z","dependency_job_id":"68bd9bec-c992-40d1-a28e-15e4f05af67d","html_url":"https://github.com/xatuke/czds-service","commit_stats":null,"previous_names":["xatuke/czds-service"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xatuke%2Fczds-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xatuke%2Fczds-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xatuke%2Fczds-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xatuke%2Fczds-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xatuke","download_url":"https://codeload.github.com/xatuke/czds-service/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246744749,"owners_count":20826822,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-02T02:36:30.235Z","updated_at":"2025-04-02T02:36:30.969Z","avatar_url":"https://github.com/xatuke.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"CZDS API Client in Python\n===========\n\nThis repository provides a Python example of how to download zone files via CZDS (Centralized Zone Data Service) REST API. \nA detail API Specs can be found [here.](https://github.com/icann/czds-api-client-java/tree/master/docs)\n\nThere is also an example provided in Java. It can be found in [this repo.](https://github.com/icann/czds-api-client-java)\n\nInstallation\n------------\n\nThis script requires Python 3. It has been tested with Python 3.7.1. \n\nIt requires the `requests` extension library. Please checkout here to see how to install it - https://github.com/requests/requests\n\nRun\n---------------------\n\n1. Make a copy of the `config.sample.json` file and name it `config.json`\n2. Edit `config.json` and fill in your information\n3. Run `python download.py` to download zonefiles manually\n\nFor using the automated service:\n\n```bash\n# Start the service - runs in background\npython zone_service.py --start\n\n# Check the service status\npython zone_service.py --status\n\n# Run once immediately (regardless of time window)\npython zone_service.py --run-once\n\n# Stop the service\npython zone_service.py --stop\n```\n\nAll the zone files will be saved in `working-directory`/zonefiles, `working-directory` is specified in `config.json`, \nor default to current directory if not specified in `config.json`\n\nDatabase Integration, Change Tracking and Webhook Notifications\n------------------------------------------------------------------\n\nThis client now supports extracting zone file data, tracking changes, and sending webhook notifications with memory-efficient processing for large zonefiles and robust resumability:\n\n1. Set `database.enabled` to `true` in your `config.json` file\n2. Optionally configure these additional parameters:\n   - `database.path`: Custom path for the SQLite database\n   - `database.batch_size`: Number of records to process in each batch (default: 10000)\n   - `database.max_connections`: Maximum number of database connections (default: 1)\n   - `log.interval_seconds`: How often to log progress updates (default: 5)\n   - `resume.enabled`: Enable resumable downloads (default: true)\n   - `resume.processing`: Enable resumable processing (default: true)\n   - `resume.checkpoint_dir`: Directory for checkpoint files (default: working_directory/checkpoints)\n\n3. For large zone files (like .com), enable caching of uncompressed files:\n   - `processing.use_cached_uncompressed`: Enable caching of uncompressed files (default: true)\n   - `processing.cached_uncompressed_dir`: Directory to store uncompressed files (default: working_directory/uncompressed_cache)\n\n4. To enable automated service that tracks changes and sends webhooks:\n   - `service.enabled`: Enable the automated zonefile service (default: false)\n   - `service.download_start_time`: Start of download window (default: \"00:15:00\" UTC)\n   - `service.download_end_time`: End of download window (default: \"05:45:00\" UTC)\n   - `service.timezone`: Timezone for the download window (default: \"UTC\")\n   - `service.change_tracking.enabled`: Enable change tracking (default: true)\n   - Configure `webhooks` array with endpoints to receive notifications\n\n5. When you run the script or service, it will:\n   - Download the zone files (resuming interrupted downloads if applicable)\n   - Stream and process each zonefile line-by-line without loading it entirely into memory\n   - Insert records in efficient batches with optimized transactions\n   - Create checkpoints to enable resuming processing in case of failure\n   - Track changes between different versions of zonefiles\n   - Send webhook notifications about new, changed, or deleted domains\n   - Provide regular progress updates during processing\n   - Log the processing status\n\nThe database contains several tables with performance-optimized indexes:\n- `zonefiles`: Stores metadata about each downloaded zone file\n- `dns_records`: Stores individual DNS records extracted from the zone files\n- `download_status`: Tracks download progress for resumability\n- `processing_checkpoints`: Records processing progress for resumability\n- `domain_states`: Tracks each domain's current state for change detection\n- `change_events`: Records detected changes (new, modified, or deleted domains)\n- `service_status`: Tracks automated service runs and statuses\n\nPerformance Optimizations:\n- **Cached Uncompressed Files**: Optionally caches uncompressed versions of zone files for much faster processing\n- **True Streaming**: Files are processed in binary chunks without decompressing the entire file at once\n- **Memory Efficiency**: Advanced buffering approach to handle multi-gigabyte files with minimal memory\n- **Batch Processing**: Records are processed in configurable batches to optimize memory usage\n- **Database Optimization**: Uses SQLite's WAL mode, optimized settings, and proper indexing\n- **Progress Tracking**: Regular status updates including file processing percentage\n- **Error Handling**: Robust error handling to process partial files and skip malformed records\n- **Domain Change Detection**: Efficiently tracks changes between zonefile versions using SHA-256 hashing\n\nResumability Features:\n- **Download Resumption**: Automatically resumes interrupted downloads from the last byte received\n- **Processing Resumption**: Restarts processing from the last successfully processed line\n- **Checkpoint System**: Creates both database and file-based checkpoints for redundancy\n- **Status Tracking**: Maintains detailed status information about each download and processing job\n- **Error Recovery**: Can recover from network errors, application crashes, or system failures\n\nService Capabilities:\n- **Scheduled Downloads**: Automatically downloads zonefiles during ICANN's update window (00:00-06:00 UTC)\n- **Change Detection**: Efficiently identifies new, modified, and deleted domains across versions\n- **Webhook Notifications**: Sends notifications about changes to configured endpoints\n- **Filtering**: Supports filtering notifications by TLD and event type\n- **Batching**: Configurable batch sizes for webhook notifications\n- **Archiving**: Maintains archives of zonefile versions for comparison\n- **Command-line Control**: Start, stop, check status, or run once using command-line interface\n\nDocumentation\n-------------\n\n* CZDS REST API Specs - https://github.com/icann/czds-api-client-java/blob/master/docs/ICANN_CZDS_api.pdf\n\nContributing\n------------\n\nContributions are welcome.\n\nOther\n-----\n\nReference Implementation in Java: https://github.com/icann/czds-api-client-java","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxatuke%2Fczds-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxatuke%2Fczds-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxatuke%2Fczds-service/lists"}