Calculate Accuracy Metrics from Modeltime Resamples

This is a wrapper for yardstick that simplifies time series regression accuracy metric calculations from a Modeltime Table that has been resampled and fitted using modeltime_fit_resamples().

Usage

modeltime_resample_accuracy(
  object,
  summary_fns = mean,
  metric_set = default_forecast_accuracy_metric_set(),
  ...
)

Arguments

object

a Modeltime Table with a column '.resample_results' (the output of modeltime_fit_resamples())

summary_fns

One or more functions to analyze resamples. The default is mean(). Possible values are:

• NULL, to returns the resamples untransformed.

• A function, e.g. mean.

• A purrr-style lambda, e.g. ~ mean(.x, na.rm = TRUE)

• A list of functions/lambdas, e.g. list(mean = mean, sd = sd)

metric_set

A yardstick::metric_set() that is used to summarize one or more forecast accuracy (regression) metrics.

...

Additional arguments passed to the function calls in summary_fns.

Details

#' Default Accuracy Metrics

The following accuracy metrics are included by default via modeltime::default_forecast_accuracy_metric_set():

• MAE - Mean absolute error, yardstick::mae()

• MAPE - Mean absolute percentage error, yardstick::mape()

• MASE - Mean absolute scaled error, yardstick::mase()

• SMAPE - Symmetric mean absolute percentage error, yardstick::smape()

• RMSE - Root mean squared error, yardstick::rmse()

• RSQ - R-squared, yardstick::rsq()

Summary Functions

By default, modeltime_resample_accuracy() returns the average accuracy metrics for each resample prediction.

The user can change this default behavior using summary_fns. Simply pass one or more Summary Functions. Internally, the functions are passed to dplyr::across(.fns), which applies the summary functions.

Returning Unsummarized Results

You can pass summary_fns = NULL to return unsummarized results by .resample_id.

Professional Tables (Interactive & Static)

Use modeltime::table_modeltime_accuracy() to format the results for reporting in reactable (interactive) or gt (static) formats, which are perfect for Shiny Apps (interactive) and PDF Reports (static).

Examples

library(modeltime)

# Mean (Default)
m750_training_resamples_fitted %>%
    modeltime_resample_accuracy() %>%
    table_modeltime_accuracy(.interactive = FALSE)


  
Accuracy Table
    
.model_id
      
.model_desc
      
.type
      
n
      
mae
      
mape
      
mase
      
smape
      
rmse
      
rsq
    
1
ARIMA(0,1,1)(0,1,1)[12]
Resamples
6
0
0
0
0
0
1
2
PROPHET
Resamples
6
0
0
0
0
0
1
3
GLMNET
Resamples
6
0
0
0
0
0
1

# Mean and Standard Deviation
m750_training_resamples_fitted %>%
    modeltime_resample_accuracy(
        summary_fns = list(mean = mean, sd = sd)
    ) %>%
    table_modeltime_accuracy(.interactive = FALSE)


  
Accuracy Table
    
.model_id
      
.model_desc
      
.type
      
n
      
mae_mean
      
mae_sd
      
mape_mean
      
mape_sd
      
mase_mean
      
mase_sd
      
smape_mean
      
smape_sd
      
rmse_mean
      
rmse_sd
      
rsq_mean
      
rsq_sd
    
1
ARIMA(0,1,1)(0,1,1)[12]
Resamples
6
0
0
0
0
0
0
0
0
0
0
1
0
2
PROPHET
Resamples
6
0
0
0
0
0
0
0
0
0
0
1
0
3
GLMNET
Resamples
6
0
0
0
0
0
0
0
0
0
0
1
0

# When summary_fns = NULL, returns the unsummarized resample results
m750_training_resamples_fitted %>%
    modeltime_resample_accuracy(
        summary_fns = NULL
    )
#> # A tibble: 18 × 10
#>    .model_id .model_desc  .resample_id .type   mae  mape  mase smape  rmse   rsq
#>        <int> <chr>        <chr>        <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#>  1         1 ARIMA(0,1,1… Slice1       Resa…     0     0     0     0     0 1    
#>  2         1 ARIMA(0,1,1… Slice2       Resa…     0     0     0     0     0 1    
#>  3         1 ARIMA(0,1,1… Slice3       Resa…     0     0     0     0     0 1    
#>  4         1 ARIMA(0,1,1… Slice4       Resa…     0     0     0     0     0 1    
#>  5         1 ARIMA(0,1,1… Slice5       Resa…     0     0     0     0     0 1    
#>  6         1 ARIMA(0,1,1… Slice6       Resa…     0     0     0     0     0 1    
#>  7         2 PROPHET      Slice1       Resa…     0     0     0     0     0 1    
#>  8         2 PROPHET      Slice2       Resa…     0     0     0     0     0 1    
#>  9         2 PROPHET      Slice3       Resa…     0     0     0     0     0 1    
#> 10         2 PROPHET      Slice4       Resa…     0     0     0     0     0 1    
#> 11         2 PROPHET      Slice5       Resa…     0     0     0     0     0 1.000
#> 12         2 PROPHET      Slice6       Resa…     0     0     0     0     0 1    
#> 13         3 GLMNET       Slice1       Resa…     0     0     0     0     0 1    
#> 14         3 GLMNET       Slice2       Resa…     0     0     0     0     0 1    
#> 15         3 GLMNET       Slice3       Resa…     0     0     0     0     0 1    
#> 16         3 GLMNET       Slice4       Resa…     0     0     0     0     0 1    
#> 17         3 GLMNET       Slice5       Resa…     0     0     0     0     0 1    
#> 18         3 GLMNET       Slice6       Resa…     0     0     0     0     0 1