{"id":17793382,"url":"https://github.com/centminmod/wordpress-plugin-mirror-poc","last_synced_at":"2025-03-16T19:30:32.905Z","repository":{"id":258125716,"uuid":"866270183","full_name":"centminmod/wordpress-plugin-mirror-poc","owner":"centminmod","description":"WordPress Plugin Mirror System Proof Of Concept Leveraging Cloudflare","archived":false,"fork":false,"pushed_at":"2024-10-16T12:41:15.000Z","size":2363,"stargazers_count":45,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-10-18T22:18:04.561Z","etag":null,"topics":["cloudflare","cloudflare-r2","cloudflare-worker","cloudflare-workers","wordpress","wordpress-plugin","wordpress-plugin-mirror","wordpress-svn","wp","wp-plugin"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/centminmod.png","metadata":{"files":{"readme":"readme-component-summary.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-02T00:12:51.000Z","updated_at":"2024-10-16T12:41:18.000Z","dependencies_parsed_at":"2024-10-18T06:01:58.314Z","dependency_job_id":null,"html_url":"https://github.com/centminmod/wordpress-plugin-mirror-poc","commit_stats":null,"previous_names":["centminmod/wordpress-plugin-mirror-poc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fwordpress-plugin-mirror-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fwordpress-plugin-mirror-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fwordpress-plugin-mirror-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fwordpress-plugin-mirror-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/centminmod","download_url":"https://codeload.github.com/centminmod/wordpress-plugin-mirror-poc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221667118,"owners_count":16860553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudflare","cloudflare-r2","cloudflare-worker","cloudflare-workers","wordpress","wordpress-plugin","wordpress-plugin-mirror","wordpress-svn","wp","wp-plugin"],"created_at":"2024-10-27T11:08:16.364Z","updated_at":"2024-10-27T11:08:17.055Z","avatar_url":"https://github.com/centminmod.png","language":null,"readme":"1. WordPress Plugin Mirror (get_plugins_r2.js):\n\nThis Cloudflare Worker is the cornerstone of the system, acting as an intelligent intermediary between clients and data sources (WordPress.org and R2 storage).\n\n- Caching Strategy: It implements a sophisticated caching mechanism using Cloudflare R2 storage. This approach significantly reduces the load on WordPress.org servers and improves response times for frequently requested plugins.\n\n- Content Type Handling: The worker intelligently handles different content types - ZIP files, JSON metadata, and checksums - each with its own storage and retrieval logic.\n\n- Rate Limiting and User Agent Rotation: To prevent abuse and mimic diverse client requests, the worker implements rate limiting and rotates through a weighted list of user agents. This helps in avoiding detection as a bot and ensures fair usage of resources.\n\n- Compression Support: The worker uses gzip compression for responses when supported by the client, reducing bandwidth usage and improving download speeds.\n\n- Force Update and Cache-Only Modes: These features provide flexibility in managing cached data. Force update ensures the latest data is always retrieved, while cache-only mode allows for system checks and preemptive caching without actual downloads.\n\n2. Bash Script (get_plugins_r2.sh):\n\nThis script serves as the client-side orchestrator for the plugin download process.\n\n- Flexible Plugin Selection: It can work with plugins installed in a WordPress directory, a predefined list, or fetch all available WordPress plugins.\n\n- Parallel Processing: The script can spawn multiple instances of the download process, significantly reducing the time required for bulk updates.\n\n- Selective Downloading: It compares local and remote versions to download only when necessary, optimizing bandwidth usage and processing time.\n\n- Delay Functionality: The ability to introduce delays between downloads helps in managing request rates and avoiding potential rate limiting from the source.\n\n- Comprehensive Logging: Detailed logging facilitates troubleshooting and performance analysis.\n\n3. D1 Database Sync Worker (scan_plugins_update_d1.js):\n\nThis worker bridges the gap between the mirrored data in R2 and a queryable database in Cloudflare D1.\n\n- Flexible Processing Modes: It supports processing all plugins, a single plugin, a batch of plugins, or a specific range, providing versatility in synchronization tasks.\n\n- Batched Processing: To work within D1's query limits, the worker processes plugins in smaller chunks, implementing delays between batches to avoid rate limiting.\n\n- Database Optimization: The worker creates optimized indexes and a virtual FTS (Full-Text Search) table, enabling efficient querying for sorting, pagination, and text-based searches.\n\n- Selective Updates: It only updates D1 when plugin data has changed, reducing unnecessary writes and improving performance.\n\n- Error Handling: The worker implements robust error handling, including logging plugins that are too large for D1, allowing for manual inspection and future handling strategies.\n\n4. D1 Sync Shell Script (scan_plugins_update_d1.sh):\n\nThis script provides a user-friendly interface to interact with the D1 Sync Worker.\n\n- Flexible Configuration: It offers numerous command-line options to control various aspects of the synchronization process.\n\n- Multi-threading: The script can distribute the workload across multiple threads, enabling parallel processing of plugin batches and significantly speeding up the overall synchronization process.\n\n- Database Management: It includes options for creating and resetting the database schema, facilitating easy setup and maintenance.\n\nInnovative Aspects:\n\n1. Edge Computing Utilization: By leveraging Cloudflare Workers, the system pushes computation to the edge, reducing latency and improving global performance.\n\n2. Tiered Storage Architecture: The use of R2 for bulk storage and D1 for queryable data creates an efficient, tiered storage system that balances performance and cost.\n\n3. Adaptive Mirroring: The system's ability to selectively update and cache data reduces unnecessary data transfer and storage costs.\n\n4. Scalability: The architecture is designed to handle a large number of plugins efficiently, making it suitable for large-scale WordPress deployments or plugin directories.\n\n5. Future-Proofing: The inclusion of full-text search capabilities and optimized indexing in the D1 database prepares the system for advanced querying needs in future applications.\n\n6. Compliance and Courtesy: Features like rate limiting and user agent rotation ensure the system interacts responsibly with WordPress.org, adhering to best practices for mirroring services.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentminmod%2Fwordpress-plugin-mirror-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcentminmod%2Fwordpress-plugin-mirror-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentminmod%2Fwordpress-plugin-mirror-poc/lists"}