{"id":20424993,"url":"https://github.com/databio/shinyqueue","last_synced_at":"2026-04-19T01:37:22.884Z","repository":{"id":84917657,"uuid":"142711349","full_name":"databio/shinyqueue","owner":"databio","description":"An R package that makes it easy to deploy shiny apps that process big data","archived":false,"fork":false,"pushed_at":"2018-10-02T13:03:40.000Z","size":18,"stargazers_count":4,"open_issues_count":3,"forks_count":1,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-12-09T13:03:41.183Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-28T21:29:52.000Z","updated_at":"2024-04-30T11:39:17.000Z","dependencies_parsed_at":"2023-03-10T17:45:45.017Z","dependency_job_id":null,"html_url":"https://github.com/databio/shinyqueue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/databio/shinyqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databio%2Fshinyqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databio%2Fshinyqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databio%2Fshinyqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databio%2Fshinyqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databio","download_url":"https://codeload.github.com/databio/shinyqueue/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databio%2Fshinyqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31991720,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"ssl_error","status_checked_at":"2026-04-18T20:23:29.375Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T07:11:54.569Z","updated_at":"2026-04-19T01:37:22.850Z","avatar_url":"https://github.com/databio.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# shinyqueue\n\nshinyqueue makes it easy to deploy computation-intensive shiny apps. It decouples heavy processing tasks from interactive visualization tasks. \n\nShiny is mostly useful for interactivity, and most effective interactivity happens quickly and therefore doesn't require large computational resources. But we find we want to build apps that provide interactive visualization of results of a more computationally intensive processing, which leads to a problem: \"when Shiny performs long-running calculations or tasks on behalf of one user, it stalls progress for all other Shiny users that are connected to the same process\" ([R Studio Blog](http://blog.rstudio.com/2018/06/26/shiny-1-1-0/)).\n\nFor a standalone shiny app that a user runs locally, this isn't a problem -- the user just runs the compute process, waits for a few minutes, and they interacts with the results. But it becomes an issue for servers because when shiny server is processing the heavy computation, the R process will be using all available CPUs, compromising the lower-intensity interactivity for all other users. Shiny 1.1.0 (from 06/2018) address this issue by enabling [asynchronous operations](http://blog.rstudio.com/2018/06/26/shiny-1-1-0/) using the `future` and `promises` package. This decouples the heavy processing step from the interactive steps and goes a long way toward improving user experience for heavy-computation multi-user shiny apps. But it leaves a few more advanced problems; for example: dealing with duplicated high-memory processes, deploying across server hardware, interfacing with non-R processes, running multiple shiny apps with different requirements, and retaining and sharing results.\n\nshinyqueue is an alternative way to implement asynchronous operations that addresses these issues. It does not rely on using the `future` or `promises` package, making it straightforward to use. It completely decouples the interactive and computation R processes, acting like an ultralight job queuing system, all written in R and interconnected with shiny.\n\n## How shinyqueue works\n\nTo understand how shinyqueue works, let's divide our web app into tasks. A computationally intensive app that will benefit from shinyqueue is one that needs to do 3 things: 1) collect the job request from the user; 2) process the request (heavy computation); and 3) render the results, typically with something reactive. In a typical shiny framework, you would use just a single R process to do all 3 of these tasks. Instead, we can divide them, and then use `shinyqueue` to help the tasks communicate with one another. Here's how each task would use `shinyqueue`:\n\n- Task 1: collect job request from user\n\tUse `shinyqueue::submitJob()` to register a job in the `shinyqueue`. The app should then forward the user to the results page for the job.\n\n- Task 2: heavy computation\n\tIn the background, the computation  uses `shinyqueue::lurk()` to check for submitted jobs, and then runs them when they are found. The user never needs to interact directly with this process, so the high processor use never makes the shiny server unresponsive. This process needs to `load(shinyqueue)` for the `lurk()` function, but it does not depend on `shiny` because the user never interacts with it.\n\n- Task 3: render results visualization\n\tThe results page uses `shinyqueue::retrieveJob()` to retrieve job metadata, which will indicate the status of the job (`queued`, `running`, or `complete`), along with a pointer to the actual results when a job is complete. The process can then read in these results and render them in shiny.\n\nThough there are 3 tasks, in practice, we really only need 2 different types of R process, because Task 1 and Task 3 are both pretty low resource use. Because they both have fairly low resource requirements, they can live in the same process without a major hit to performance. Therefore, all we need to do is divide our app into two types of R process: interactive processes (which will run shiny), and computation processes (which prepare the results shiny will render). In a minimal use case, you could have just one of each type of process, but on a busy server, it's also possible to imagine multiple process of each type. As a bonus, each process can also live in its own container, dividing responsibilities and allowing multiple different apps to use the same `shinyqueue`.\n\nBoth process types will `load(shinyqueue)` and use `shinyqueue` functions that will enable these processes to really easily communicate with one another.\n\n\n## Example of task 1: submit job\n\n```\njob = list()\njob$uploadedFile = \"path/to/file\"\njob$referenceGenome = \"hg38\"\njob$universe = \"blah\"\nshinyqueue::submitJob(depotQueueDir, datalist=job)\n```\n\n## Example of task 2: retrieve job and interactive results viewer\n\n```\n# parse URL\njobID = getJobIDFromURL()\njobStatus = shinyqueue::retrieveJob(jobID)\n\n#display results viewer...\n```\n\n## Example of task 3: process jobs\n\n\n```\n# A function that will run the LOLAweb process for a given \"job\" file.\n# The job file should specify the path to the user uploaded file and\n# any other user-selections provided\n\nprocessLOLAwebJob = function(file, outfolder=\"/path/to/results/\", resources=LWResources) {\n\n\tjob = yaml::yaml.load(file)\n\n\t# Access the pre-loaded data\n\tregionDB = resources$regionDBs[jobs$genome]\n\t\n\tresult = runLOLA(job$query, regionDB)\n\n\t# Now, store that result in the output folder,\n\t# which should then render correctly\n\tsave(result, outfolder)\n}\n\n# Register any job class this container knows how to handle,\n# and map those jobs to the function that can handle it.\nprocessFunctions = list(\"LOLAweb\" = \"processLOLAwebJob\")\n\n\n# First, load up some data to save in this container\nLWResources = list()\nLWResources$regionDBs = list()\nfor (genome in genomes) {\n\tLWResources$regionDBs[[genome]] = LOLA::loadRegionDB(\"/path/to/genome\")\n}\nLWResources$universes = list()\n\nfor (universe in universes) {\n\tLWResources$universes[[universe]] = LOLA::readbed(\"/path/to/universe\")\n}\n\n\n\n# Now, just lurk, waiting for new jobs:\nshinyqueue::lurk(\"/path/to/job/folder\", processFunctions)\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabio%2Fshinyqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabio%2Fshinyqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabio%2Fshinyqueue/lists"}