Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/wlandau/autometric
Background resource logging from R
https://github.com/wlandau/autometric
r
Last synced: 8 days ago
JSON representation
Background resource logging from R
- Host: GitHub
- URL: https://github.com/wlandau/autometric
- Owner: wlandau
- License: other
- Created: 2024-09-27T14:23:50.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2024-10-30T19:02:19.000Z (14 days ago)
- Last Synced: 2024-10-30T19:25:54.394Z (14 days ago)
- Topics: r
- Language: R
- Homepage: https://wlandau.github.io/autometric
- Size: 1.39 MB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# autometric 
[](https://CRAN.R-project.org/package=autometric)
[](https://www.repostatus.org/#active)
[](https://github.com/wlandau/autometric/actions?query=workflow%3Acheck)
[](https://app.codecov.io/gh/wlandau/autometric)
[](https://github.com/wlandau/autometric/actions?query=workflow%3Alint)
[](https://github.com/wlandau/autometric/actions?query=workflow%3Apkgdown)
Intense parallel workloads can be difficult to monitor. Packages [`crew.cluster`](https://wlandau.github.io/crew.cluster/), [`clustermq`](https://mschubert.github.io/clustermq/), and [`future.batchtools`](https://future.batchtools.futureverse.org/) distribute hundreds of worker processes over multiple computers. If a worker process exhausts its available memory, it may terminate silently, leaving the underlying problem difficult to detect or troubleshoot. Using the `autometric` package, a worker can proactively monitor itself in a detached POSIX thread. The worker process itself runs normally, and the thread writes to a log every few seconds. If the worker terminates unexpectedly, `autometric` can read and visualize the log file to reveal potential resource-related reasons for the crash.
## Requirements
* A Linux, Mac OS, or Windows operating system.
* POSIX threads. Modern tool chains seem to provide R with POSIX thread support, even on Windows.
* Either POSIX timers or a Windows operating system.
## Installation
You can install the development version of `autometric` from [GitHub](https://github.com/) with:
```{r, eval = FALSE}
remotes::install_github("wlandau/autometric")
```
## Usage
The `log_start()` function in `autometric` starts a non-blocking POSIX thread to write resource usage statistics to a log at periodic intervals. The following example uses the [`callr`](https://callr.r-lib.org/) R package to launch a resource-intensive background process on a Unix-like system. The `autometric` thread prints to standard output, and `callr` directs all its standard output to a temporary text file we define in advance.^[Logging to standard output is useful on clusters like SLURM where workers already redirect standard output to log files, or on the cloud where a service like [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) captures messages instead of directing them to a physical file.] Custom calls to `log_phase_set()` annotate the log data^[See also `log_phase_get()` and `log_phase_reset()`].
```{r}
log_file <- tempfile()
process <- callr::r_bg(
func = function() {
library(autometric)
log_phase_set("setup")
log_start(
path = "/dev/stdout",
pids = c(my_worker = Sys.getpid()),
seconds = 1
)
log_phase_set("warmup")
Sys.sleep(3)
print("You can mix printed messages with regular log data.")
print("Defining a function that guzzles CPU power.")
is_prime <- function(n) {
if (n <= 1) return(FALSE)
for (i in seq(2, sqrt(n))) {
if (n %% i == 0) {
return(FALSE)
}
}
TRUE
}
log_phase_set("large object")
x <- rnorm(1e8)
log_phase_set("heavy cpu")
lapply(seq_len(1e6), is_prime)
log_phase_set("another object")
y <- rnorm(1e8)
log_phase_set("more cpu")
lapply(seq_len(1e6), is_prime)
log_phase_set("third object")
z <- rnorm(1e8)
log_phase_set("done")
},
stdout = log_file
)
```
```{r, include = FALSE}
while(process$is_alive()) {
Sys.sleep(1)
}
```
When we read in the log file, we see messages from both R and the `autometric` thread.
```{r}
writeLines(readLines(log_file))
```
`autometric` knows how to read its log entries even if the log file has other messages. See the documentation of `log_read()` to learn how to interpret the data and customize the measurement units.
```{r}
library(autometric)
log_data <- log_read(log_file)
log_data
```
`autometric` also supports simple visualizations plot performance metrics over time. To avoid depending on many other R packages, `autometric` only uses base plotting functionality. Feel free to create fancier visualizations directly with [`ggplot2`](https://ggplot2.tidyverse.org/).
```{r resident}
log_plot(log_data, metric = "resident")
```
```{r cpu}
log_plot(log_data, phase = "more cpu", metric = "cpu")
```
## Attribution
`autometric` heavily leverages fantastic work on the [`ps`](https://ps.r-lib.org/) R package by Jay Loden, Dave Daeschler, Giampaolo Rodola, Gábor Csárdi, and Posit Software, PBC. The source code of [`ps`](https://ps.r-lib.org/) was especially helpful for identifying appropriate system calls to retrieve resource usage statistics. Attribution is given in the `Authors@R` field of the `DESCRIPTION` file, the `LICENSE.note` file at the top level of the package, and in the comments of the C files in `src/`. Please visit to view the license of [`ps`](https://ps.r-lib.org/). [`ps`](https://ps.r-lib.org/) in turn is based on [`psutil`](https://github.com/giampaolo/psutil), whose license is available at .
## Code of Conduct
Please note that the `autometric` project is released with a Contributor Code of Conduct: `https://contributor-covenant.org/version/2/1/CODE_OF_CONDUCT.html`. By contributing to this project, you agree to abide by its terms.