{"id":13518566,"url":"https://github.com/zepgram/module-multi-threading","last_synced_at":"2026-02-04T12:06:17.104Z","repository":{"id":65433456,"uuid":"592257961","full_name":"zepgram/module-multi-threading","owner":"zepgram","description":"This module is a powerful tool for developers who want to process large data sets in a short amount of time.","archived":false,"fork":false,"pushed_at":"2026-02-03T20:40:39.000Z","size":63,"stargazers_count":81,"open_issues_count":0,"forks_count":10,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-02-03T22:02:46.331Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zepgram.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-01-23T10:34:17.000Z","updated_at":"2026-02-03T20:40:43.000Z","dependencies_parsed_at":"2025-07-19T10:09:46.235Z","dependency_job_id":"59b17d91-07da-4a6d-a2d1-0eb2a5f62bc5","html_url":"https://github.com/zepgram/module-multi-threading","commit_stats":{"total_commits":10,"total_committers":2,"mean_commits":5.0,"dds":0.09999999999999998,"last_synced_commit":"e8ea1d25abe2ed56f38d74641d6002f93a86d386"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/zepgram/module-multi-threading","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zepgram%2Fmodule-multi-threading","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zepgram%2Fmodule-multi-threading/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zepgram%2Fmodule-multi-threading/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zepgram%2Fmodule-multi-threading/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zepgram","download_url":"https://codeload.github.com/zepgram/module-multi-threading/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zepgram%2Fmodule-multi-threading/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29084088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-04T03:31:03.593Z","status":"ssl_error","status_checked_at":"2026-02-04T03:29:50.742Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T05:01:46.310Z","updated_at":"2026-02-04T12:06:17.095Z","avatar_url":"https://github.com/zepgram.png","language":"PHP","funding_links":[],"categories":["PHP"],"sub_categories":[],"readme":"# Multi-Threading for Magento 2\n\nThis module is a powerful tool for developers who want to process large data sets in\na short amount of time. It allows you to process large collections of data in parallel\nusing multiple child processes, improving performance and reducing processing time.\n\n## Installation\n```php\ncomposer require zepgram/module-multi-threading\nbin/magento module:enable Zepgram_MultiThreading\nbin/magento setup:upgrade\n```\n\n## Usage\n\nThese classes allows you to process a search criteria, a collection or an array using multi-threading.\n\n### ForkedSearchResultProcessor\n\n```php\nuse Zepgram\\MultiThreading\\Model\\ForkedSearchResultProcessor;\nuse Magento\\Catalog\\Api\\ProductRepositoryInterface;\nuse Magento\\Framework\\Api\\SearchCriteriaBuilder;\n\nclass MyAwesomeClass\n{\n    /** @var ForkedSearchResultProcessor */\n    private $forkedSearchResultProcessor;\n    \n    /** @var ProductRepositoryInterface */\n    private $productRepository;\n    \n    public function __construct(\n        ForkedSearchResultProcessor $forkedSearchResultProcessor,\n        ProductRepositoryInterface $productRepository,\n        SearchCriteriaBuilder $searchCriteriaBuilder \n    ) {\n        $this-\u003eforkedSearchResultProcessor = $forkedSearchResultProcessor;\n        $this-\u003eproductRepository = $productRepository;\n        $this-\u003esearchCriteriaBuilder = $searchCriteriaBuilder;\n    }\n    \n    $searchCriteria = $this-\u003esearchCriteriaBuilder-\u003ecreate();\n    $productRepository = $this-\u003eproductRepository;\n    $callback = function ($item) {\n        $item-\u003egetData();\n        // do your business logic here\n    };\n    \n    $this-\u003eforkedSearchResultProcessor-\u003eprocess(\n        $searchCriteria,\n        $productRepository,\n        $callback,\n        $pageSize = 1000,\n        $maxChildrenProcess = 10,\n        $isIdempotent = true\n    );\n}\n```\n\n### ForkedCollectionProcessor\n```php\nuse Zepgram\\MultiThreading\\Model\\ForkedCollectionProcessor;\nuse Magento\\Catalog\\Model\\ResourceModel\\Product\\CollectionFactory;\n\nclass MyAwesomeClass\n{\n    /** @var ForkedCollectionProcessor */\n    private $forkedCollectionProcessor;\n\n    public function __construct(\n        ForkedCollectionProcessor $forkedCollectionProcessor,\n        CollectionFactory $collectionFactory\n    ) {\n        $this-\u003eforkedCollectionProcessor = $forkedCollectionProcessor;\n        $this-\u003ecollectionFactory = $collectionFactory;\n    }\n\n    $collection = $this-\u003ecollectionFactory-\u003ecreate();\n    $callback = function ($item) {\n        $item-\u003egetData();\n        // do your business logic here\n    };\n\n    $this-\u003eforkedCollectionProcessor-\u003eprocess(\n        $collection,\n        $callback,\n        $pageSize = 1000,\n        $maxChildrenProcess = 10,\n        $isIdempotent = true\n    );\n}\n```\n\n### ForkedArrayProcessor\nThis class allows you to process an array of data using multi-threading.\n\n```php\nuse Zepgram\\MultiThreading\\Model\\ForkedArrayProcessor;\n\nclass MyAwesomeClass\n{\n    /** @var ForkedArrayProcessor */\n    private $forkedArrayProcessor;\n    \n    public function __construct(ForkedArrayProcessor $forkedArrayProcessor)\n    {\n        $this-\u003eforkedArrayProcessor = $forkedArrayProcessor;\n    }\n    \n    $array = [1,2,3,4,5,...];\n    $callback = function ($item) {\n        echo $item;\n        // do your business logic here\n    };\n    \n    $this-\u003eforkedArrayProcessor-\u003eprocess(\n        $array,\n        $callback,\n        $pageSize = 2,\n        $maxChildrenProcess = 2\n    );\n}\n```\n\n### ParallelStoreProcessor or ParallelWebsiteProcessor\n\n```php\nuse Zepgram\\MultiThreading\\Model\\Dimension\\ParallelStoreProcessor;\nuse Magento\\Catalog\\Model\\ResourceModel\\Product\\Collection;\nuse Magento\\Catalog\\Model\\ResourceModel\\Product\\CollectionFactory;\n\nclass MyAwesomeClass\n{\n    /** @var ParallelStoreProcessor */\n    private $parallelStoreProcessor;\n    \n    /** @var CollectionFactory */\n    private $collectionFactory;\n    \n    public function __construct(\n        ParallelStoreProcessor $parallelStoreProcessor,\n        CollectionFactory $collectionFactory\n    ) {\n        $this-\u003eparallelStoreProcessor = $parallelStoreProcessor;\n        $this-\u003ecollectionFactory = $collectionFactory;\n    }\n    \n    $array = [1,2,3,4,5,...];\n    $callback = function (StoreInterface $store) {\n        // retrieve data from database foreach stores (do not load the collection !)\n        $collection = $this-\u003ecollectionFactory-\u003ecreate();\n        $collection-\u003eaddFieldToFilter('type_id', 'simple')\n            -\u003eaddFieldToSelect(['sku', 'description', 'created_at'])\n            -\u003esetStoreId($store-\u003egetId())\n            -\u003eaddStoreFilter($store-\u003egetId())\n            -\u003edistinct(true);\n            \n        // handle pagination system to avoid memory leak\n        $currentPage = 1;\n        $pageSize = 1000;\n        $collection-\u003esetPageSize($pageSize);\n        $totalPages = $collection-\u003egetLastPageNumber();\n        while ($currentPage \u003c= $totalPages) {\n            $collection-\u003eclear();\n            $collection-\u003esetCurPage($currentPage);\n            foreach ($collection-\u003egetItems() as $product) {\n                // do your business logic here\n            }\n            $currentPage++;\n        }\n    };\n    \n    // your collection will be processed foreach store by a dedicated child process\n    $this-\u003eparallelStoreProcessor-\u003eprocess(\n        $callback,\n        $maxChildrenProcess = null,\n        $onlyActiveStores = true,\n        $withDefaultStore = false\n    );\n}\n```\n\n### bin/magento thread:processor command\n\nThis command allows running a command indefinitely in a dedicated thread using \nthe Process Symfony Component.\n```php\nbin/magento thread:processor \u003ccommand_name\u003e [--timeout=\u003ctimeout\u003e] [--iterations=\u003citerations\u003e] [--environment=\u003cenvironment\u003e] [--progress]\n```\n\n#### Options\n\n- `timeout`: Define the process timeout in seconds (default: 300)\n- `iterations`: Define the number of iteration (default: 0)\n- `environment`: Set environment variables separate by comma\n- `progress`: Show progress bar while executing command\n\n### How it works\nThe `thread:processor` command creates a dedicated child process to execute existing command line.\nThe child process runs the command specified by the user, while the parent process\nmonitors the child process and can act accordingly. You can define iterations and execute the same command\nmultiple times with a dedicated child foreach execution.\n\nThe `ForkedSearchResultProcessor`,`ForkedCollectionProcessor` and `ForkedArrayProcessor` classes\nuse a similar approach to process a search criteria or a collection. The process is divided\ninto several pages, and for each page, a child process is created to run the callback\nfunction specified by the user on each item of that page.\n\nThe `ParallelStoreProcessor` and `ParallelWebsiteProcessor` classes are designed to make it easier \nto process a list of stores or websites in parallel. To use either of these classes, you'll need to\nprovide a callback function that will be called for each store or website in the list. The callback \nfunction should take one parameter, which will be a single store or website object.\u003cbr\u003e\nEach store or website will be passed to the callback function in a separate process, \nallowing faster processing times.\nThe number of children process cannot exceed the number of stores or websites: for example,\nif you have 10 stores, the maximum number of child processes that can be created in parallel is 10.\n\n#### Here is a breakdown of the parameters:\n- `$collection`/`$searchCriteria`/`$array`: The first parameter is the data source,\n  either a `Magento\\Framework\\Api\\SearchCriteriaInterface` for ForkedSearchResultProcessor or\n  a `Magento\\Framework\\Data\\Collection` for ForkedCollectionProcessor, and an `array`\n  for ForkedArrayProcessor\n\n- `$callback`: This parameter is a callable that will be executed on each item of the collection.\n  It is a callback function that is passed the current item from the collection to be processed.\n  This function should contain the business logic that should be executed on each item.\n\n- `$pageSize`: This parameter is used to set the number of items per page.\n  It is used to paginate the collection so that it can be processed in smaller chunks.\n\n- `$maxChildrenProcess`: This parameter is used to set the maximum number of child\n  processes that can be run simultaneously. This is used to control the number of threads\n  that will be used by the multi-threading process. If set to 1, by definition you will have no parallelization, \n  the parent process will wait the child process to finish before creating another one.\n\n- `$isIdempotent`: This parameter is a flag set to `true` by default and can be used for `ForkedSearchResultProcessor` \n  or `ForkedCollectionProcessor` when your `$maxChildrenProcess` is greater than one.\n  While fetching data from database with `ForkedSearchResult` and `ForkedCollectionProcessor` you may change values\n  queried: by modifying items on columns queried you will change the nature of the initial collection query and at the end, \n  the OFFSET limit in the query will be invalid because the native pagination system expect the pagination to be\n  processed by only one process. To avoid that, set `$isIdempotent` to `false`.\u003cbr\u003e\n  E.G.: In your collection query, you request all products `disabled`, in your callback method you `enable` and save \n  them in database, then in this particular case you are modifying the column that you request in your collection,\n  your query is not idempotent.\n\n### Memory Limit\nThis module allows to bypass the limitation of the memory limit, because the memory\nlimit is reset on each child process creation. This means that even if the memory limit\nis set to a low value, this module can still process large amounts of data without\nrunning out of memory. However, it is important to keep in mind that this also means\nthat the overall resource usage will be higher, so it is important to monitor the\nsystem and adjust the parameters accordingly.\n\n### Limitations\nThis module uses `pcntl_fork()` function which is not available on Windows.\n\n### Conclusion\nThis module provides a useful tool for running commands or processing collections\nand search criteria in a multi-threaded way, making it a great solution for improving\nperformance and reducing execution time.\nThe module is easy to install and use, and provides options for controlling the number\nof child processes, timeout, and environment variables.\n\n### Disclaimer\nThe Multi-Threading for Magento 2 module is provided as is, without any guarantees or warranties.\nWhile this module has been tested and is believed to be functional, it is important to note\nthat the use of multi-threading in PHP can be complex and may have unintended consequences.\nAs such, it is the responsibility of the user of this module to thoroughly test it in a\ndevelopment environment before deploying it to a production environment.\nI decline all responsibility for any issues or damages that may occur as a result of using\nthis module. With great power comes great responsibility, use it wisely.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzepgram%2Fmodule-multi-threading","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzepgram%2Fmodule-multi-threading","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzepgram%2Fmodule-multi-threading/lists"}