backtesting-configuration

Backtesting Configuration

When to Use

Use this skill to translate a deployment scenario (retraining cadence, forecast horizon, data budget, ingestion delay) into TimeSeriesFold parameters. TimeSeriesFold is the cross-validation strategy passed via the cv argument to backtesting_forecaster (and its multi-series / stats variants) and to the hyperparameter search functions.

from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold

cv = TimeSeriesFold(
    steps=7,                       # forecast horizon
    initial_train_size=365,        # first training window (required)
    refit=False,                   # train once (default)
    fixed_train_size=True,         # rolling window (default)
    gap=0,
)

metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric='mean_absolute_error',
)

For fast hyperparameter tuning where multi-step realism is not required, use OneStepAheadFold instead: it validates one step ahead (no recursive prediction), so it is much faster but less representative of multi-step performance. Use TimeSeriesFold for realistic multi-step backtesting.

Related skills

• Before: forecasting-single-series / forecasting-multiple-series (have a fitted forecaster before backtesting)
• With: metric-selection (choose the metric(s) backtesting reports)
• With: hyperparameter-optimization (the same cv object drives the search functions)
• After: prediction-intervals (add interval= / interval_method= to backtest uncertainty)

Stop Conditions

Scan before writing code. Each row lists a rule, the symptom when it is broken, and the recovery. Full pitfall catalog: the troubleshooting-common-errors skill.

TimeSeriesFold Parameters

Constraints

• Must produce at least 2 folds: initial_train_size + gap + 2 * steps <= n_observations
• initial_train_size must be large enough for the model to learn patterns (at minimum 2× the lag order for ML models, or 2× steps for statistical models)
• gap simulates real-world delay between data availability and forecast usage
• When fixed_train_size=True, the window rolls forward (oldest data discarded). Use for concept drift or when old data is less relevant.
• When fixed_train_size=False, the window expands (all history retained). Use when more data always helps.

Business Scenario Mapping

Retraining frequency

Data freshness vs volume

Deployment gap

Initial training size

Fold stride (test set overlap)

Examples

"I retrain my model every Monday and forecast the next 7 days"

refit = True
fixed_train_size = False  # Keep all history
gap = 0  # No delay
fold_stride = None  # Non-overlapping weeks

"I want to simulate deploying once and seeing how the model degrades"

refit = False  # Train once; fixed_train_size has no effect without refit
gap = 0

"There's a 2-day lag between data ingestion and when forecasts are needed"

gap = 2
refit = True

"I have limited compute — evaluate every other week"

refit = True
skip_folds = 2  # Keep every 2nd fold

"I want maximum evaluation coverage with a 12-step horizon"

initial_train_size = <minimum viable>  # 2 * max_lag
refit = False  # Faster; trains once, fixed_train_size has no effect
fold_stride = 1  # Sliding window (many overlapping folds)