Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mrc-ide/didehpc

:cloud::computer::cloud: Support for the DIDE cluster
https://github.com/mrc-ide/didehpc

cluster infrastructure

Last synced: about 2 months ago
JSON representation

:cloud::computer::cloud: Support for the DIDE cluster

Host: GitHub
URL: https://github.com/mrc-ide/didehpc
Owner: mrc-ide
License: other
Created: 2015-11-06T16:22:51.000Z (about 9 years ago)
Default Branch: master
Last Pushed: 2023-08-02T20:22:07.000Z (over 1 year ago)
Last Synced: 2023-08-02T21:28:09.721Z (over 1 year ago)
Topics: cluster, infrastructure
Language: R
Homepage: http://mrc-ide.github.io/didehpc
Size: 1.38 MB
Stars: 10
Watchers: 2
Forks: 4
Open Issues: 20
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# didehpc

[![Project Status: WIP – Initial development is in progress, but there has not y
et been a stable, usable release suitable for the public.](https://www.repostatus.org/badges/latest/wip.svg)](https://www.repostatus.org/#wip)
[![R build status](https://github.com/mrc-ide/didehpc/workflows/R-CMD-check/badge.svg)](https://github.com/mrc-ide/didehpc/actions)
[![codecov.io](https://codecov.io/github/mrc-ide/didehpc/coverage.svg?branch=master
)](https://codecov.io/github/mrc-ide/didehpc?branch=master)

> DIDE Cluster Support

**NOTICE**: This will only be of use to people at DIDE, as it uses our [cluster web portal](https://mrcdata.dide.ic.ac.uk/hpc), local cluster, and local network file systems.

## What is this?

This is a package for interfacing with the DIDE cluster directly from R. It is meant make jobs running on the cluster appear as if they are running locally but asynchronously. The idea is to let the cluster appear as an extension of your own computer so you can get using it within an R project easily.

## How does it work?

The steps below are described in more detail in the [vignettes](https://mrc-ide.github.io/didehpc)

1. Ensure that your project is in a directory that the cluster can see (i.e. on one of the network drives). See [notes](https://mrc-ide.github.io/didehpc/articles/didehpc.html#mapping-network-drives) for instructions
2. Set your DIDE credentials up so that you can log in and tell `didehpc` about them.
3. Create a "context" in which future expressions will be evaluated (which will be recreated on the cluster)
4. Create a "queue" that uses that context
5. Queue expressions which will be run at some future time on the cluster
6. Monitor progress, retrieve results, etc.

## Documentation

* New to this? [The main vignette](https://mrc-ide.github.io/didehpc/articles/didehpc.html) contains full instructions and explanations about why some bits are needed.
* Need a reminder? There is a [quickstart guide](https://mrc-ide.github.io/didehpc/articles/quickstart.html) which is much shorter and will be quicker to glance through.
* Trying to install packages on the cluster? Check the [packages vignette](https://mrc-ide.github.io/didehpc/articles/packages.html) for ways of controlling this.
* Having problems? Check the [troubleshooting guide](https://mrc-ide.github.io/didehpc/articles/troubleshooting.html).
* Lots of small jobs to run? Consider using [workers](https://mrc-ide.github.io/didehpc/articles/workers.html) for a fast queue over several cluster nodes.

## Issues

* Check the [issue tracker](https://github.com/mrc-ide/didehpc/issues) for known problems, or to create a new one
* Use the "Cluster" channel on Teams, which Rich and Wes keep an eye on

## Installation

The simplest approach is to run:

```r
# install.packages("drat") # if needed
drat:::add("mrc-ide")
install.packages("didehpc")
```

## License