https://github.com/pzingg/dirq-r
A port of the Perl module Directory::Queue::Simple for the R language
https://github.com/pzingg/dirq-r
fifo-queue filesystem queues
Last synced: 12 months ago
JSON representation
A port of the Perl module Directory::Queue::Simple for the R language
- Host: GitHub
- URL: https://github.com/pzingg/dirq-r
- Owner: pzingg
- License: other
- Created: 2023-12-16T00:30:45.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-16T00:31:12.000Z (over 2 years ago)
- Last Synced: 2025-04-05T04:15:04.833Z (about 1 year ago)
- Topics: fifo-queue, filesystem, queues
- Language: R
- Homepage:
- Size: 9.77 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.Rmd
- License: LICENSE
Awesome Lists containing this project
README
---
output:
github_document:
html_preview: false
---
```{r knitrsetup, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "README-"
)
```
```{r mainexample, echo = FALSE}
suppressMessages(suppressWarnings(library(dirq)))
```
# dirq - A Filesystem Directory Based Queue
A port of Perl module [Directory::Queue::Simple]
(http://search.cpan.org/dist/Directory-Queue/)
The goal of this package is to offer a simple queue system
using the underlying filesystem for storage, security and
to prevent race conditions via atomic operations.
A simple producer:
```{r producer}
# Define a path to your queue
# In production, you wouldn't use a temporary directory!
path <- tempfile()
# Create a new queue or recover an existing one
q <- dirq(path)
cat(paste("top level directory is", q$path(), "\n"))
# Add some messages
for (i in seq.int(1, 6)) {
name <- q$add(list(data = paste("element", i)), format = "json")
cat(paste("added element", name, "\n"))
}
```
A simple consumer:
```{r consumer}
# Recover the previous queue
q <- dirq(path)
# Get the first element in the queue
cursor <- q$iter_first()
while (!is.null(cursor)) {
if (q$lock(cursor)) {
cat(paste("reading element", cursor, "\n"))
data <- q$get(cursor, format = "json")
q$remove(cursor)
# Or use q$unlock(cursor) to just peek at data
}
# Get the next element in the queue
cursor <- q$iter_next(cursor)
}
```
# Similar work
## txtq
[Will Landau](https://github.com/wlandau)'s [`txtq`](https://github.com/wlandau/txtq) is also a pure filesystem-based queue.
## liteq
[Gábor Csárdi](https://github.com/gaborcsardi)'s [`liteq`](https://github.com/r-lib/liteq) package offers essentially the same functionality implemented with SQLite. It has a some additional features, such as the ability to detect crashed workers and re-queue failed messages, but it was in an early stage of development at the time `dirq` was released.
## Other message queues
There is a plethora of message queues beyond R, most notably [ZeroMQ](https://zeromq.org) and [RabbitMQ](https://www.rabbitmq.com/). In fact, [Jeroen Ooms](https://github.com/jeroen) and [Whit Armstrong](https://github.com/armstrtw) maintain [`rzmq`](https://github.com/ropensci/rzmq), a package to work with [ZeroMQ](https://zeromq.org) from R. Even in this landscape, `dirq` has advantages.
1. The `dirq` user interface is friendly, and its internals are simple.
No prior knowledge of sockets or message-passing is required.
2. `dirq` is lightweight, R-focused, and easy to install. It only depends
on R and a few packages on [CRAN](https://cran.r-project.org).
3. Because `dirq` it is file-based, the queue persists even if your work
crashes.