{"id":13725246,"url":"https://github.com/netlify/gatsby-parallel-runner","last_synced_at":"2025-05-04T05:32:19.089Z","repository":{"id":38782410,"uuid":"238360015","full_name":"netlify/gatsby-parallel-runner","owner":"netlify","description":null,"archived":true,"fork":false,"pushed_at":"2025-01-20T16:29:41.000Z","size":419,"stargazers_count":40,"open_issues_count":11,"forks_count":9,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-26T16:59:26.681Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/netlify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-02-05T03:30:01.000Z","updated_at":"2025-04-04T02:24:45.000Z","dependencies_parsed_at":"2024-02-04T07:47:43.074Z","dependency_job_id":"651b16bf-a2e8-4cf7-910e-e96bf945efa1","html_url":"https://github.com/netlify/gatsby-parallel-runner","commit_stats":{"total_commits":79,"total_committers":3,"mean_commits":"26.333333333333332","dds":0.06329113924050633,"last_synced_commit":"2844dc81094a1f7a95cb8a2c94055b4c0fee6b1c"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netlify%2Fgatsby-parallel-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netlify%2Fgatsby-parallel-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netlify%2Fgatsby-parallel-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netlify%2Fgatsby-parallel-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/netlify","download_url":"https://codeload.github.com/netlify/gatsby-parallel-runner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252293082,"owners_count":21724960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T01:02:16.910Z","updated_at":"2025-05-04T05:32:18.495Z","avatar_url":"https://github.com/netlify.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Gatsby Parallel Runner\n\nThis is an early a parallel runner for gatsby that allows plugins and core parts of Gatsby to parallelize\nsuited tasks such as image processing.\n\nWhen gatsby is started from a parent process with the environment variable `ENABLE_GATSBY_EXTERNAL_JOBS` set,\nit will communicate some jobs up to the parent process via ipc, instead of running them in it's own internal\nqueue.\n\nThis allows a parent process to orchestrate certain task across multiple workers for better parallelization\nthrough autoscaling cloud functions or the like.\n\nCurrently this plugin includes a processing queue implementation based on Google Cloud Functions, but the\ngeneral abstractions in place should make it easy to add similar runtimes for other cloud providers or via\ndifferent approaches to parallelization.\n\n## Installation and usage\n\nInstall in your gatsby project:\n\n```\nnpm i gatsby-parallel-runner\n```\n\nTo use with Google Cloud, set relevant env variables in your shell:\n\n```\nexport GOOGLE_APPLICATION_CREDENTIALS=~/path/to/your/google-credentials.json\nexport TOPIC=parallel-runner-topic\n```\n\nDeploy the cloud function:\n\n```\nnpx gatsby-parallel-runner deploy\n\n```\n\nThen run your Gatsby build with the parallel runner instead of the default `gatsby build` command.\n\n```\nnpx gatsby-parallel-runner\n```\n\n## Processor Queues, Processors and Implementations\n\nGatsby Parallel Runner comes with a set of core abstractions for parallelizing jobs.\n\nThe main orchestrator is the Processor Queue that gives invididual processors a simple interface for\nsending jobs to cloud functions and getting back results:\n\n```js\nconst result = await queue.process(job)\n```\n\nTo do it's job, the ProcessorQueue needs a `pubSubImplementation` that must provide\n`push(msg)` and `subscribe(handler)` methods for enqueuing new jobs and receiving\nresults.\n\nImplementations are defined in `src/processor-queue/implementations` and there's currently\njust one of them based on Google's Cloud Functions.\n\nThe `src/processors` folder has the different processors that can be triggered via Gatsby's\nexternal job feature.\n\nThe processor folder must be named after the Redux event that should trigger it. Ie, the\nImage Processing processor gets triggered by the sharp plugin via an `IMAGE_PROCESSING` job,\nso the folder is called `image-processing`\n\nEach processor can have a set of implementations based on the Processor Queue implementations\navailable.\n\nThere's currently just one processor (image-processing), with an implementation for `google-functions`.\n\nWhen running `npx gatsby-parallel-runner deploy`, the active processor queue implementation will\nmake sure to deploy all the cloud function needed for the available processors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetlify%2Fgatsby-parallel-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetlify%2Fgatsby-parallel-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetlify%2Fgatsby-parallel-runner/lists"}