Fits Models in a Modeltime Table to Resamples

Resampled predictions are commonly used for:

1. Analyzing accuracy and stability of models

2. As inputs to Ensemble methods (refer to the modeltime.ensemble package)

Usage

modeltime_fit_resamples(object, resamples, control = control_resamples())

Arguments

object

A Modeltime Table

resamples

An rset resample object. Used to generate sub-model predictions for the meta-learner. See timetk::time_series_cv() or rsample::vfold_cv() for making resamples.

control

A tune::control_resamples() object to provide control over the resampling process.

Value

A Modeltime Table (mdl_time_tbl) object with a column containing resample results (.resample_results)

Details

The function is a wrapper for tune::fit_resamples() to iteratively train and predict models contained in a Modeltime Table on resample objects. One difference between tune::fit_resamples() and modeltime_fit_resamples() is that predictions are always returned (i.e. control = tune::control_resamples(save_pred = TRUE)). This is needed for ensemble_model_spec().

Resampled Prediction Accuracy

Calculating Accuracy Metrics on models fit to resamples can help to understand the model performance and stability under different forecasting windows. See modeltime_resample_accuracy() for getting resampled prediction accuracy for each model.

Ensembles

Fitting and Predicting Resamples is useful in creating Stacked Ensembles using modeltime.ensemble::ensemble_model_spec(). The sub-model cross-validation predictions are used as the input to the meta-learner model.

Examples

library(tidymodels)
#> ── Attaching packages ────────────────────────────────────── tidymodels 1.1.1 ──
#> ✔ broom        1.0.5     ✔ recipes      1.0.9
#> ✔ dials        1.2.0     ✔ rsample      1.2.0
#> ✔ dplyr        1.1.4     ✔ tibble       3.2.1
#> ✔ ggplot2      3.4.4     ✔ tidyr        1.3.0
#> ✔ infer        1.0.5     ✔ tune         1.1.2
#> ✔ modeldata    1.2.0     ✔ workflows    1.1.3
#> ✔ parsnip      1.1.1     ✔ workflowsets 1.0.1
#> ✔ purrr        1.0.2     ✔ yardstick    1.2.0
#> ── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
#> ✖ purrr::discard() masks scales::discard()
#> ✖ dplyr::filter()  masks stats::filter()
#> ✖ dplyr::lag()     masks stats::lag()
#> ✖ recipes::step()  masks stats::step()
#> • Use suppressPackageStartupMessages() to eliminate package startup messages
library(modeltime)
library(timetk)
library(magrittr)
#> 
#> Attaching package: ‘magrittr’
#> The following object is masked from ‘package:tidyr’:
#> 
#>     extract
#> The following object is masked from ‘package:purrr’:
#> 
#>     set_names

# Make resamples
resamples_tscv <- training(m750_splits) %>%
    time_series_cv(
        assess      = "2 years",
        initial     = "5 years",
        skip        = "2 years",
        # Normally we do more than one slice, but this speeds up the example
        slice_limit = 1
    )
#> Using date_var: date

# \donttest{
# Fit and generate resample predictions
m750_models_resample <- m750_models %>%
    modeltime_fit_resamples(
        resamples = resamples_tscv,
        control   = control_resamples(verbose = TRUE)
    )
#> ── Fitting Resamples ────────────────────────────────────────────
#> 
#> • Model ID: 1 ARIMA(0,1,1)(0,1,1)[12]
#> i Slice1: preprocessor 1/1
#> ! Slice1: preprocessor 1/1:
#>   `keep_original_cols` was added to `step_dummy()` after this r...
#>   ℹ Regenerate your recipe to avoid this warning.
#> ✓ Slice1: preprocessor 1/1
#> i Slice1: preprocessor 1/1, model 1/1
#> frequency = 12 observations per 1 year
#> ✓ Slice1: preprocessor 1/1, model 1/1
#> i Slice1: preprocessor 1/1, model 1/1 (extracts)
#> i Slice1: preprocessor 1/1, model 1/1 (predictions)
#> • Model ID: 2 PROPHET
#> i Slice1: preprocessor 1/1
#> ! Slice1: preprocessor 1/1:
#>   `keep_original_cols` was added to `step_dummy()` after this r...
#>   ℹ Regenerate your recipe to avoid this warning.
#> ✓ Slice1: preprocessor 1/1
#> i Slice1: preprocessor 1/1, model 1/1
#> Disabling weekly seasonality. Run prophet with weekly.seasonality=TRUE to override this.
#> Disabling daily seasonality. Run prophet with daily.seasonality=TRUE to override this.
#> ✓ Slice1: preprocessor 1/1, model 1/1
#> i Slice1: preprocessor 1/1, model 1/1 (extracts)
#> i Slice1: preprocessor 1/1, model 1/1 (predictions)
#> • Model ID: 3 GLMNET
#> i Slice1: preprocessor 1/1
#> ! Slice1: preprocessor 1/1:
#>   `keep_original_cols` was added to `step_dummy()` after this r...
#>   ℹ Regenerate your recipe to avoid this warning.
#> ✓ Slice1: preprocessor 1/1
#> i Slice1: preprocessor 1/1, model 1/1
#> ✓ Slice1: preprocessor 1/1, model 1/1
#> i Slice1: preprocessor 1/1, model 1/1 (extracts)
#> i Slice1: preprocessor 1/1, model 1/1 (predictions)
#> 3.587 sec elapsed
#> 

# A new data frame is created from the Modeltime Table
#  with a column labeled, '.resample_results'
m750_models_resample
#> # Modeltime Table
#> # A tibble: 3 × 4
#>   .model_id .model     .model_desc             .resample_results
#>       <int> <list>     <chr>                   <list>           
#> 1         1 <workflow> ARIMA(0,1,1)(0,1,1)[12] <rsmp[+]>        
#> 2         2 <workflow> PROPHET                 <rsmp[+]>        
#> 3         3 <workflow> GLMNET                  <rsmp[+]>        
# }