{"id":15685729,"url":"https://github.com/johnbillion/slurpetta","last_synced_at":"2025-05-07T14:44:37.785Z","repository":{"id":230288115,"uuid":"778992930","full_name":"johnbillion/slurpetta","owner":"johnbillion","description":"Slurps down the most popular plugins and themes from WordPress.org","archived":false,"fork":false,"pushed_at":"2024-09-15T14:49:35.000Z","size":143,"stargazers_count":13,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T11:03:36.346Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnbillion.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-28T20:24:53.000Z","updated_at":"2024-11-27T18:27:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"10544ba5-c601-4118-aa73-7051f03f16ac","html_url":"https://github.com/johnbillion/slurpetta","commit_stats":null,"previous_names":["johnbillion/slurpetta"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbillion%2Fslurpetta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbillion%2Fslurpetta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbillion%2Fslurpetta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbillion%2Fslurpetta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnbillion","download_url":"https://codeload.github.com/johnbillion/slurpetta/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252897957,"owners_count":21821536,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T17:29:40.534Z","updated_at":"2025-05-07T14:44:37.759Z","avatar_url":"https://github.com/johnbillion.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Slurpetta\n\nA command line PHP script that downloads and updates a copy of the latest stable\nversion of:\n\n* Every plugin in the WordPress.org directory with at least 10,000 active installations\n* Every theme in the WordPress.org directory with at least 1,000 active installations\n* WordPress core latest and nightly\n\nAs of August 2024 this is **around 2,500 plugins** and **700** themes.\n\nSlurping and updating just these plugins and themes is at least 20x faster and smaller than\nslurping the entire plugin and theme repos which would otherwise total over 100,000 items.\n\nReally handy for doing local searches across popular WordPress plugins, themes, and core.\n\n## Requirements\n\n* Unix system (tested on macOS and Linux)\n* PHP 8.0 or higher\n* `wget` and `svn` command-line executables installed\n\n## Instructions\n\nRun this from within the `slurpetta` directory:\n\n```sh\n./update plugins\n./update themes\n./update core\n```\n\nWhen the script is done:\n\n* The `plugins` and `themes` directories contain all the plugins and themes\n* The `popular` directory contains symlinks to all plugins with over 1M active installations\n* The `top` directory contains symlinks to all plugins with over 5M active installations\n* The `core` directory contains the latest release of WordPress core\n\n## Scanning the results\n\n### Simple scanning\n\nYou'll likely have the best experience using [ripgrep](https://github.com/BurntSushi/ripgrep) to search for files. It's available via package managers for macOS, Linux, and Windows, and it's just about the fastest tool available for regex searching across a large number of files.\n\nExamples:\n\n```sh\nrg --type php 'rest_get_date_with_gmt' plugins\n```\n\n### Finding files\n\n```sh\nfind plugins -name 'foo.php'\n```\n\n### Advanced scanning\n\nIt's possible to perform more powerful searches that are aware of language syntax and semantics using [Semgrep](https://github.com/semgrep/semgrep). It's available via package managers or via Docker. You don't need to sign into the Semgrep Code service on the CLI despite what its documentation says.\n\nSemgrep allows you to perform searches using [its language-aware pattern syntax](https://semgrep.dev/docs/writing-rules/pattern-syntax/). Benefits include ignoring code comments and being aware of multi-line matches because it's aware of the semantics of the code beyond simple static analysis. Note that complex searches will take a lot more time than standard searches with ripgrep.\n\nExamples:\n\n```sh\nsemgrep -e 'printf(esc_attr__(...), ...)' --lang=php --no-git-ignore plugins\n```\n\nThere is a built-in ruleset for PHP that you can use, but running it across all plugins will give you a very large number of results so you may want to restrict it to a sub-directory or a single plugin.\n\n```sh\nsemgrep --config \"p/php\" --no-git-ignore plugins/a\nsemgrep --config \"p/php\" --no-git-ignore plugins/a/akismet\n```\n\nThere is also a built-in ruleset specifically for vulnerabilities in WordPress code. You can scan all the plugins with this rule because at the time of writing it only shows around 50 results.\n\n```sh\nsemgrep --config \"p/wordpress\" --no-git-ignore plugins\n```\n\n### Generating scan summaries\n\nThis repository also includes a script to show a summary of a scan.  For example:\n\n```sh\nrg --type php 'rest_get_date_with_gmt' plugins themes | tee scans/rest_get_date_with_gmt.txt\n./summarize-scan.php scans/rest_get_date_with_gmt.txt\n```\n\n```\nMatching plugins: 83\nMatches  Slug                               Active installs\n=======  ====                               ===============\n     11  woocommerce                             5,000,000+\n      1  mailchimp-for-wp                        2,000,000+\n      1  advanced-custom-fields                  2,000,000+\n     52  custom-post-type-ui                     1,000,000+\n      5  astra-sites                             1,000,000+\n      1  better-wp-security                        900,000+\n      1  woocommerce-gateway-stripe                800,000+\n      8  imagify                                   800,000+\n      1  woocommerce-payments                      700,000+\n      1  premium-addons-for-elementor              700,000+\n\nMatching themes: 0\n\nMatching core: 2\nMatches  Slug  Active installs\n=======  ====  ===============\n      5  latest              -\n      5  nightly             -\n```\n\n## FAQ\n\n### What can I use this for?\n\n* Scanning (SAST)\n* Producing stats\n* Training an LLM\n\n### Why download the zip files? Why not use SVN?\n\nAn SVN checkout of the entire repository is a BEAST of a thing. You don't want it,\ntrust me. Updates and cleanups can take **hours** or even **days** to complete.\n\n### How long will it take?\n\nYour first update will take a while but depends entirely on your connection and\ndisk speeds. On a fast modern machine with a fast internet connection it may take\nas little as 15 minutes, but be prepared for it to take hours on a machine with\na slower connection or disk speeds.\n\nBut subsequent updates are smarter. The script tracks the SVN revision numbers\nof your latest updates and then asks the SVN repositories for a list of plugins\nand themes that have changed since. Only those changed are updated after the\ninitial sync.\n\n### How much disk space do I need?\n\nAs of August 2024:\n\n* Around 12 GB of disk space for plugins\n* Around 3 GB of disk space for themes\n* A few MB of disk space for WordPress core\n\n### Something went wrong, how do I do a partial update?\n\nThe last successful update revision numbers are stored in `plugins/.last-revision`.\nand `themes/.last-revision`. You can just overwrite one of those and the next `update`\nwill start after that revision.\n\n### What is this thing actually doing to my computer?\n\nOnce downloads have started, you can use a command like this to monitor the\ntasks being executed by this tool:\n\n```sh\nwatch -n .5 \"pstree -pa `pgrep -f '^xargs -n 1 -P .+ ./download'`\"\n```\n\n## Many thanks\n\nThis is an adaptation of [the WordPress Plugin Directory Slurper](https://github.com/markjaquith/WordPress-Plugin-Directory-Slurper) by Mark Jaquith. The majority of the code was originally written by Mark and the other contributors to that library. If you need the entire plugin directory rather than just those with at least 10,000 active installations, then use that instead.\n\n## Copyright \u0026 License\n\nCopyright © 2011-2020 Mark Jaquith, 2024 John Blackbourn\n\nThis program is free software; you can redistribute it and/or\nmodify it under the terms of the GNU General Public License\nas published by the Free Software Foundation; either version 2\nof the License, or (at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program; if not, write to the Free Software\nFoundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnbillion%2Fslurpetta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnbillion%2Fslurpetta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnbillion%2Fslurpetta/lists"}