Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/centminmod/wordpress-plugin-mirror-poc

WordPress Plugin Mirror System Proof Of Concept Leveraging Cloudflare
https://github.com/centminmod/wordpress-plugin-mirror-poc

cloudflare cloudflare-r2 cloudflare-worker cloudflare-workers wordpress wordpress-plugin wordpress-plugin-mirror wordpress-svn wp wp-plugin

Last synced: 3 months ago
JSON representation

WordPress Plugin Mirror System Proof Of Concept Leveraging Cloudflare

Awesome Lists containing this project

README 

        
1. WordPress Plugin Mirror (get_plugins_r2.js):



This Cloudflare Worker is the cornerstone of the system, acting as an intelligent intermediary between clients and data sources (WordPress.org and R2 storage).



- Caching Strategy: It implements a sophisticated caching mechanism using Cloudflare R2 storage. This approach significantly reduces the load on WordPress.org servers and improves response times for frequently requested plugins.



- Content Type Handling: The worker intelligently handles different content types - ZIP files, JSON metadata, and checksums - each with its own storage and retrieval logic.



- Rate Limiting and User Agent Rotation: To prevent abuse and mimic diverse client requests, the worker implements rate limiting and rotates through a weighted list of user agents. This helps in avoiding detection as a bot and ensures fair usage of resources.



- Compression Support: The worker uses gzip compression for responses when supported by the client, reducing bandwidth usage and improving download speeds.



- Force Update and Cache-Only Modes: These features provide flexibility in managing cached data. Force update ensures the latest data is always retrieved, while cache-only mode allows for system checks and preemptive caching without actual downloads.



2. Bash Script (get_plugins_r2.sh):



This script serves as the client-side orchestrator for the plugin download process.



- Flexible Plugin Selection: It can work with plugins installed in a WordPress directory, a predefined list, or fetch all available WordPress plugins.



- Parallel Processing: The script can spawn multiple instances of the download process, significantly reducing the time required for bulk updates.



- Selective Downloading: It compares local and remote versions to download only when necessary, optimizing bandwidth usage and processing time.



- Delay Functionality: The ability to introduce delays between downloads helps in managing request rates and avoiding potential rate limiting from the source.



- Comprehensive Logging: Detailed logging facilitates troubleshooting and performance analysis.



3. D1 Database Sync Worker (scan_plugins_update_d1.js):



This worker bridges the gap between the mirrored data in R2 and a queryable database in Cloudflare D1.



- Flexible Processing Modes: It supports processing all plugins, a single plugin, a batch of plugins, or a specific range, providing versatility in synchronization tasks.



- Batched Processing: To work within D1's query limits, the worker processes plugins in smaller chunks, implementing delays between batches to avoid rate limiting.



- Database Optimization: The worker creates optimized indexes and a virtual FTS (Full-Text Search) table, enabling efficient querying for sorting, pagination, and text-based searches.



- Selective Updates: It only updates D1 when plugin data has changed, reducing unnecessary writes and improving performance.



- Error Handling: The worker implements robust error handling, including logging plugins that are too large for D1, allowing for manual inspection and future handling strategies.



4. D1 Sync Shell Script (scan_plugins_update_d1.sh):



This script provides a user-friendly interface to interact with the D1 Sync Worker.



- Flexible Configuration: It offers numerous command-line options to control various aspects of the synchronization process.



- Multi-threading: The script can distribute the workload across multiple threads, enabling parallel processing of plugin batches and significantly speeding up the overall synchronization process.



- Database Management: It includes options for creating and resetting the database schema, facilitating easy setup and maintenance.



Innovative Aspects:



1. Edge Computing Utilization: By leveraging Cloudflare Workers, the system pushes computation to the edge, reducing latency and improving global performance.



2. Tiered Storage Architecture: The use of R2 for bulk storage and D1 for queryable data creates an efficient, tiered storage system that balances performance and cost.



3. Adaptive Mirroring: The system's ability to selectively update and cache data reduces unnecessary data transfer and storage costs.



4. Scalability: The architecture is designed to handle a large number of plugins efficiently, making it suitable for large-scale WordPress deployments or plugin directories.



5. Future-Proofing: The inclusion of full-text search capabilities and optimized indexing in the D1 database prepares the system for advanced querying needs in future applications.



6. Compliance and Courtesy: Features like rate limiting and user agent rotation ensure the system interacts responsibly with WordPress.org, adhering to best practices for mirroring services.