Start parallel clusters using parallel
package
Usage
parallel_start(..., .method = c("parallel", "spark"))
parallel_stop()
Arguments
- ...
Parameters passed to underlying functions (See Details Section)
- .method
The method to create the parallel backend. Supports:
"parallel" - Uses the
parallel
anddoParallel
packages"spark" - Uses the
sparklyr
package
Parallel (.method = "parallel"
)
Performs 3 Steps:
Makes clusters using
parallel::makeCluster(...)
. Theparallel_start(...)
are passed toparallel::makeCluster(...)
.Registers clusters using
doParallel::registerDoParallel()
.Adds
.libPaths()
usingparallel::clusterCall()
.
Spark (.method = "spark"
)
Important, make sure to create a spark connection using
sparklyr::spark_connect()
.Pass the connection object as the first argument. For example,
parallel_start(sc, .method = "spark")
.The
parallel_start(...)
are passed tosparklyr::registerDoSpark(...)
.