yy_api_task_pull function

Functions for Task Pull Parallelism

Functions for Task Pull Parallelism

These functions are designed for SPMD but assume that rank 0 is a manager and the rest are workers.

task.pull(jids, FUN, ..., rank.manager = .pbd_env$SPMD.CT$rank.root, comm = .pbd_env$SPMD.CT$comm, bcast = .pbd_env$SPMD.TP$bcast, barrier = .pbd_env$SPMD.TP$barrier, try = .pbd_env$SPMD.TP$try, try.silent = .pbd_env$SPMD.TP$try.silent) task.pull.workers(FUN = function(jid, ...){ return(jid) }, ..., rank.manager = .pbd_env$SPMD.CT$rank.root, comm = .pbd_env$SPMD.CT$comm, try = .pbd_env$SPMD.TP$try, try.silent = .pbd_env$SPMD.TP$try.silent) task.pull.manager(jids, rank.manager = .pbd_env$SPMD.CT$rank.root, comm = .pbd_env$SPMD.CT$comm)

Arguments

  • jids: all job ids (a vector of positive integers).
  • FUN: a function to be evaluated by workers.
  • ...: extra parameters for FUN.
  • rank.manager: rank of the manager from where jid is sent.
  • comm: a communicator number.
  • bcast: if bcast to all ranks.
  • barrier: if barrier for all ranks.
  • try: wheter to use try() to avoid crashes. CAUTION: try = FALSE is not safe and can crash all MPI/R jobs.
  • try.silent: turn off error messages from try().

Details

All of these functions are designed to emulate a manager/workers paradigm in an SPMD environment. If your chunk workloads are known and similar, consider a direct SPMD solution.

FUN is a user defined function which has jid as its first argument and other variables are given in ....

The manager will be queried by workers whenever a worker finishes a job to see if more jobs are available.

Returns

A list with length comm.size() - 1

will be returned to the manager and NULL to the workers. Each element of the list contains the returns ret of their FUN

results.

References

Programming with Big Data in R Website: https://pbdr.org/

Author(s)

Wei-Chen Chen wccsnow@gmail.com , George Ostrouchov, Drew Schmidt, Pragneshkumar Patel, and Hao Yu.

See Also

get.jid().

Examples

## Not run: ### Under command mode, run the demo with 2 processors by ### (Use Rscript.exe for windows system) # mpiexec -np 2 Rscript -e "demo(task_pull,'pbdMPI',ask=F,echo=F)" ### Or # execmpi("demo(task_pull,'pbdMPI',ask=F,echo=F)", nranks = 2L) ## End(Not run)
  • Maintainer: Wei-Chen Chen
  • License: Mozilla Public License 2.0
  • Last published: 2024-09-18