A Distributed Worker Launcher Framework
Terminate dispatchers and/or workers
Random name
Create a crew
relay object.
Retry code.
Manually terminate a local process.
Get the termination signal.
Create a stateful throttling object.
Configure TLS.
Crew worker.
R6
TLS class.
R6
throttle class.
Create a launcher with local process workers.
Create an abstract launcher.
Crew assertion
Local asynchronous client object.
Create a local monitor object.
R6
async class.
R6
client class.
Controller group class
Controller class
Local process launcher class
Launcher abstract class
Local monitor class
R6
relay class.
Create a client object.
Create a controller group.
Create a controller with a local process launcher.
Create a controller object from a client and launcher.
Deprecate a crew
feature.
Run an asynchronous task in the crew launcher.
Evaluate an R command and return results as a monad.
crew: a distributed worker launcher framework
Objects exported from other packages
In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The 'NNG'-powered 'mirai' R package by Gao (2023) <doi:10.5281/zenodo.7912722> is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The 'crew' package extends 'mirai' with a unifying interface for third-party worker launchers. Inspiration also comes from packages. 'future' by Bengtsson (2021) <doi:10.32614/RJ-2021-048>, 'rrq' by FitzJohn and Ashton (2023) <https://github.com/mrc-ide/rrq>, 'clustermq' by Schubert (2019) <doi:10.1093/bioinformatics/btz284>), and 'batchtools' by Lang, Bischel, and Surmann (2017) <doi:10.21105/joss.00135>.
Useful links