augment_expanding

augment_expanding(data, date_column, value_column, window_func='mean', min_periods=None, engine='pandas', threads=1, show_progress=True, reduce_memory=False, **kwargs)

Apply one or more Series-based expanding functions to one or more columns of a DataFrame.

Parameters

Name Type Description Default
data Union[pd.DataFrame, pd.core.groupby.generic.DataFrameGroupBy] Input data to be processed. Can be a Pandas DataFrame or a GroupBy object. required
date_column str Name of the datetime column. Data is sorted by this column within each group. required
value_column Union[str, list] Column(s) to which the expanding window functions should be applied. Can be a single column name or a list. required
window_func Union[str, list, Tuple[str, Callable]] The window_func parameter in the augment_expanding function specifies the function(s) to be applied to the expanding windows of the value column(s). 1. It can be either: - A string representing the name of a standard function (e.g., ‘mean’, ‘sum’). 2. For custom functions: - Provide a list of tuples. Each tuple should contain a custom name for the function and the function itself. - Each custom function should accept a Pandas Series as its input and operate on that series. Example: (“range”, lambda x: x.max() - x.min()) (See more Examples below.) Note: If your function needs to operate on multiple columns (i.e., it requires access to a DataFrame rather than just a Series), consider using the augment_expanding_apply function in this library. 'mean'
min_periods int Minimum observations in the window to have a value. Defaults to the window size. If set, a value will be produced even if fewer observations are present than the window size. None
engine str Specifies the backend computation library for augmenting expanding window functions. The options are: - “pandas” (default): Uses the pandas library. - “polars”: Uses the polars library, which may offer performance benefits for larger datasets. 'pandas'
threads int Number of threads to use for parallel processing. If threads is set to 1, parallel processing will be disabled. Set to -1 to use all available CPU cores. 1
show_progress bool If True, a progress bar will be displayed during parallel processing. True
reduce_memory bool The reduce_memory parameter is used to specify whether to reduce the memory usage of the DataFrame by converting int, float to smaller bytes and str to categorical data. This reduces memory for large data but may impact resolution of float and will change str to categorical. Default is True. False
**kwargs additional keyword arguments Additional arguments passed to the pandas.Series.expanding method when using the Pandas engine. {}

Returns

Type Description
pd.DataFrame The augment_expanding function returns a DataFrame with new columns for each applied function, window size, and value column.

Notes

Performance

Polars Engine (3X faster than Pandas)

In most cases, the polars engine will be faster than the pandas engine. Speed tests indicate 3X or more.

Parallel Processing (Pandas Engine Only)

This function uses parallel processing to speed up computation for large datasets with many time series groups:

Parallel processing has overhead and may not be faster on small datasets.

To use parallel processing, set threads = -1 to use all available processors.

Examples

# Example 1 - Pandas Backend for Expanding Window Functions
# This example demonstrates the use of string-named functions 
# on an expanding window using the Pandas backend for computations.
    
import pytimetk as tk
import pandas as pd
import numpy as np

df = tk.load_dataset("m4_daily", parse_dates = ['date'])

expanded_df = (
    df
        .groupby('id')
        .augment_expanding(
            date_column = 'date', 
            value_column = 'value', 
            window_func = [
                'mean',  # Built-in mean function
                'std',   # Built-in standard deviation function,
                 ('quantile_75', lambda x: pd.Series(x).quantile(0.75)),  # Custom quantile function
                    
            ],
            min_periods = 1,
            engine = 'pandas',  # Utilize pandas for the underlying computations
            threads = 1,  # Disable parallel processing
            show_progress = True,  # Display a progress bar
            )
)
display(expanded_df)
id date value value_expanding_mean value_expanding_std value_expanding_quantile_75
0 D10 2014-07-03 2076.2 2076.200000 NaN 2076.200
1 D10 2014-07-04 2073.4 2074.800000 1.979899 2075.500
2 D10 2014-07-05 2048.7 2066.100000 15.133737 2074.800
3 D10 2014-07-06 2048.9 2061.800000 15.054789 2074.100
4 D10 2014-07-07 2006.4 2050.720000 27.996732 2073.400
... ... ... ... ... ... ...
9738 D500 2012-09-19 9418.8 8286.606679 2456.960489 9903.550
9739 D500 2012-09-20 9365.7 8286.864035 2456.723940 9902.600
9740 D500 2012-09-21 9445.9 8287.140391 2456.496163 9902.075
9741 D500 2012-09-22 9497.9 8287.429011 2456.274422 9901.550
9742 D500 2012-09-23 9545.3 8287.728789 2456.058410 9901.025

9743 rows × 6 columns

# Example 2 - Polars Backend for Expanding Window Functions using Built-Ins 
#             (538X Faster than Pandas)
#  This example demonstrates the use of string-named functions and configurable 
#  functions using the Polars backend for computations. Configurable functions, 
#  like pl_quantile, allow the use of specific parameters associated with their 
#  corresponding polars.Expr.rolling_<function_name> method.
#  For instance, pl_quantile corresponds to polars.Expr.rolling_quantile.
    
import pytimetk as tk
import pandas as pd
import polars as pl
import numpy as np
from pytimetk.utils.polars_helpers import pl_quantile
from pytimetk.utils.pandas_helpers import pd_quantile

df = tk.load_dataset("m4_daily", parse_dates = ['date'])

expanded_df = (
    df
        .groupby('id')
        .augment_expanding(
            date_column = 'date', 
            value_column = 'value', 
            window_func = [
                'mean',  # Built-in mean function
                'std',   # Built-in std function
                ('quantile_75', pl_quantile(quantile=0.75)),  # Configurable with all parameters found in polars.Expr.rolling_quantile
            ],
            min_periods = 1,
            engine = 'polars',  # Utilize Polars for the underlying computations
        )
)
display(expanded_df)
id date value value_expanding_mean value_expanding_std value_expanding_quantile_75
0 D10 2014-07-03 2076.2 2076.200000 NaN 2076.2
1 D10 2014-07-04 2073.4 2074.800000 1.979899 2076.2
2 D10 2014-07-05 2048.7 2066.100000 15.133737 2076.2
3 D10 2014-07-06 2048.9 2061.800000 15.054789 2076.2
4 D10 2014-07-07 2006.4 2050.720000 27.996732 2073.4
... ... ... ... ... ... ...
9738 D500 2012-09-19 9418.8 8286.606679 2456.960489 9906.4
9739 D500 2012-09-20 9365.7 8286.864035 2456.723940 9902.6
9740 D500 2012-09-21 9445.9 8287.140391 2456.496163 9902.6
9741 D500 2012-09-22 9497.9 8287.429011 2456.274422 9902.6
9742 D500 2012-09-23 9545.3 8287.728789 2456.058410 9902.6

9743 rows × 6 columns

# Example 3 - Lambda Functions for Expanding Window Functions are faster in Pandas than Polars
# This example demonstrates the use of lambda functions of the form lambda x: x
# Identity lambda functions, while convenient, have signficantly slower performance.
# When using lambda functions the Pandas backend will likely be faster than Polars.

import pytimetk as tk
import pandas as pd
import numpy as np

df = tk.load_dataset("m4_daily", parse_dates = ['date'])

expanded_df = (
    df
        .groupby('id')
        .augment_expanding(
            date_column = 'date', 
            value_column = 'value', 
            window_func = [
                
                ('range', lambda x: x.max() - x.min()),  # Identity lambda function: can be slower, especially in Polars
            ],
            min_periods = 1,
            engine = 'pandas',  # Utilize pandas for the underlying computations
        )
)
display(expanded_df)
id date value value_expanding_range
0 D10 2014-07-03 2076.2 0.0
1 D10 2014-07-04 2073.4 2.8
2 D10 2014-07-05 2048.7 27.5
3 D10 2014-07-06 2048.9 27.5
4 D10 2014-07-07 2006.4 69.8
... ... ... ... ...
9738 D500 2012-09-19 9418.8 10782.0
9739 D500 2012-09-20 9365.7 10782.0
9740 D500 2012-09-21 9445.9 10782.0
9741 D500 2012-09-22 9497.9 10782.0
9742 D500 2012-09-23 9545.3 10782.0

9743 rows × 4 columns