Hyperparameter Optimization
Developmenthyperparameter-optimizationHyperparameter Optimization is an Agent Skill for development published in the skforecast/skforecast repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Optimizes forecaster hyperparameters using grid search, random search, or Bayesian search (Optuna). Covers single-series and multi-series search, cross-validation configuration, and search space definition. Use when the user wants to find the best model configuration.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/skforecast/skforecast/blob/HEAD/skills/hyperparameter-optimization/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/hyperparameter-optimization, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Hyperparameter Optimization
References
See references/search-parameters.md for
the complete parameter comparison across all 9 search functions, function
routing by forecaster type, and lags_grid / search_space / param_grid
usage details.
When to Use
Use hyperparameter search after establishing a baseline forecaster to improve prediction accuracy. Skforecast supports three strategies:
| Strategy | When to Use | Speed |
|---|---|---|
| Bayesian Search | Recommended default. Smart exploration via Optuna | Fastest to converge |
| Random Search | Large parameter space, limited compute budget | Medium |
| Grid Search | Small parameter space, exhaustive exploration | Slowest |
Related skills
- Before:
autocorrelation-and-lag-selection(narrow thelagssearch space to a statistically informed candidate set) - Before:
feature-selection(run the search on a reduced feature set to make it tractable) - After:
prediction-intervals(add uncertainty quantification once the configuration is fixed)
Stop Conditions
Scan before writing code. Each row lists a rule, the symptom when it is broken, and the recovery. Full pitfall catalog: the troubleshooting-common-errors skill.
| Rule | Symptom | Recovery |
|---|---|---|
The search refits the forecaster in place with the best params when return_best=True (the default) | With return_best=False, the forecaster keeps its pre-search params | Rely on the default, or refit with the best params from the results table |
Use the *_multiseries / *_stats search variant matching the forecaster type | Search function raises on the wrong forecaster type | Call e.g. bayesian_search_forecaster_multiseries / grid_search_stats |
Include lags in the Bayesian search_space() | Suboptimal search; the highest-impact parameter stays fixed | Add trial.suggest_categorical('lags', [...]) to the search space |
Bayesian Search (Recommended)
Always prefer Bayesian search as the default strategy. It uses Optuna to intelligently explore the search space.
from skforecast.recursive import ForecasterRecursive
from skforecast.model_selection import bayesian_search_forecaster, TimeSeriesFold
from lightgbm import LGBMRegressor
forecaster = ForecasterRecursive(
estimator=LGBMRegressor(random_state=123),
lags=24,
)
cv = TimeSeriesFold(
steps=12,
initial_train_size=len(data) - 100,
refit=False,
)
# Define search space as a function — lags CAN be included here
def search_space(trial):
return {
'lags': trial.suggest_categorical('lags', [12, 24, [1, 2, 3, 23, 24]]),
'n_estimators': trial.suggest_int('n_estimators', 50, 500),
'max_depth': trial.suggest_int('max_depth', 3, 15),
'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
}
# n_trials=20 is the default. Increase for better results (50-200 recommended).
results, study = bayesian_search_forecaster(
forecaster=forecaster,
y=data['target'],
exog=exog,
cv=cv,
search_space=search_space,
metric='mean_absolute_error',
n_trials=20,
random_state=123,
return_best=True, # Automatically updates forecaster with best params
n_jobs='auto',
show_progress=True,
output_file='search_results.csv', # Save results incrementally
)
# results is a DataFrame sorted by metric (best first)
# study is the full Optuna Study; access the best trial with study.best_trial
Grid Search
from skforecast.model_selection import grid_search_forecaster
# Different lag configurations to try
lags_grid = [3, 10, 24, [1, 2, 3, 23, 24]]
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [5, 10, 15],
'learning_rate': [0.01, 0.1],
}
results = grid_search_forecaster(
forecaster=forecaster,
y=data['target'],
exog=exog,
cv=cv,
lags_grid=lags_grid,
param_grid=param_grid,
metric='mean_absolute_error',
return_best=True,
n_jobs='auto',
show_progress=True,
)
Random Search
from skforecast.model_selection import random_search_forecaster
# Note: uses param_distributions (not param_grid) and n_iter
param_distributions = {
'n_estimators': [50, 100, 200, 500],
'max_depth': [3, 5, 10, 15],
'learning_rate': [0.01, 0.05, 0.1, 0.3],
}
results = random_search_forecaster(
forecaster=forecaster,
y=data['target'],
exog=exog,
cv=cv,
lags_grid=lags_grid,
param_distributions=param_distributions,
n_iter=10, # Number of random parameter combinations to try
random_state=123,
metric='mean_absolute_error',
return_best=True,
n_jobs='auto',
show_progress=True,
)
Multi-Series Search
from skforecast.recursive import ForecasterRecursiveMultiSeries
from skforecast.model_selection import bayesian_search_forecaster_multiseries
forecaster = ForecasterRecursiveMultiSeries(
estimator=LGBMRegressor(random_state=123),
lags=24,
encoding='ordinal',
)
cv = TimeSeriesFold(
steps=12,
initial_train_size=len(series) - 100,
refit=False,
)
results, study = bayesian_search_forecaster_multiseries(
forecaster=forecaster,
series=series,
exog=exog,
cv=cv,
search_space=search_space,
metric='mean_absolute_error',
aggregate_metric=['weighted_average', 'average', 'pooling'], # Default
levels=None, # None = evaluate all series; or list of series names
n_trials=20,
return_best=True,
n_jobs='auto',
show_progress=True,
)
# Access the best trial with study.best_trial
Statistical Models Search
from skforecast.recursive import ForecasterStats
from skforecast.stats import Arima
from skforecast.model_selection import grid_search_stats
forecaster = ForecasterStats(estimator=Arima(order=(1, 1, 1)))
param_grid = {
'order': [(1, 0, 0), (1, 1, 0), (1, 1, 1), (2, 1, 1)],
'seasonal_order': [(0, 0, 0), (1, 1, 1)],
'm': [12],
}
results = grid_search_stats(
forecaster=forecaster,
y=data['target'],
cv=cv,
param_grid=param_grid,
metric='mean_absolute_error',
return_best=True,
)
Fast Tuning with OneStepAheadFold
from skforecast.model_selection import OneStepAheadFold
# Much faster than TimeSeriesFold — no recursive predictions needed
cv_fast = OneStepAheadFold(
initial_train_size=len(data) - 100,
)
results, study = bayesian_search_forecaster(
forecaster=forecaster,
y=data['target'],
cv=cv_fast,
search_space=search_space,
metric='mean_absolute_error',
n_trials=100,
return_best=True,
)
# Access the best trial with study.best_trial
Common Mistakes
- Not setting
return_best=True: The forecaster is not updated with the best parameters unless this is True. - Too few trials in Bayesian search: Start with at least 20-50 trials for meaningful exploration.
- Using TimeSeriesFold for initial tuning: Use
OneStepAheadFoldfirst for fast screening, then validate the top candidates withTimeSeriesFold. - Forgetting to include lags in search space: For Bayesian search, lags can be included in
search_space()— this is often the most impactful parameter.