Start parallel clusters using parallel package

parallel_start(..., .method = c("parallel", "spark"))

parallel_stop()

Arguments

...

Parameters passed to underlying functions (See Details Section)

.method

The method to create the parallel backend. Supports:

  • "parallel" - Uses the parallel and doParallel packages

  • "spark" - Uses the sparklyr package

Parallel (.method = "parallel")

Performs 3 Steps:

  1. Makes clusters using parallel::makeCluster(...). The parallel_start(...) are passed to parallel::makeCluster(...).

  2. Registers clusters using doParallel::registerDoParallel().

  3. Adds .libPaths() using parallel::clusterCall().

Spark (.method = "spark")

Examples

# Starts 2 clusters parallel_start(2) # Returns to sequential processing parallel_stop()