Skip to contents

Fine-tuning

Source: vignettes/fine-tuning.Rmd
fine-tuning.Rmd
library(nixtlar)
#> Error in get(paste0(generic, ".", class), envir = get_method_env()) : 
#>   object 'type_sum.accel' not found

1. Introduction

Foundation models, such as TimeGPT, are trained on large and diverse datasets, enabling them to generate predictions for data not seen during training. This process, where the model is used directly to solve a forecasting task, is known as zero-shot learning. It is also possible, however, to further train the model on a new dataset, starting from its pre-trained parameters. This process, called fine-tuning, aims to improve the accuracy of predictions by tailoring the model to the specific characteristics of the new data.

By default, TimeGPT employs a zero-shot approach for faster computation. However, several options are available for fine-tuning TimeGPT to your data. While fine-tuning increases the total execution time, it can improve forecast accuracy.

In this vignette, we explain how to fine-tune TimeGPT using the nixtlar package. It assumes you have already set up your API key. If you haven’t done this, please read the Get Started vignette first.

2. Fine-tuning parameters

The parameters that can be used for fine-tuning TimeGPT via nixtlar are:

  • finetune_steps (int): The number of training iterations on the input data. Forecasts are then produced using the updated model.

  • finetune_depth (int, from 1 to 5): A value controlling how many parameters of TimeGPT are fine-tuned on your dataset. When set to 1, only a few parameters are fine-tuned, while setting it to 5 fine-tunes all parameters. 1

  • finetune_loss (string, see options): The name of the loss function used during fine-tuning. Options available are:

    • default: A proprietary loss function robust to outliers
    • mae: Mean Absolute Error
    • mse: Mean Squared Error
    • rmse: Root Mean Squared Error
    • mape: Mean Absolute Percentage Error
    • smape: Symmetric Mean Absolute Percentage Error

Note that the finetune_depth and finetune_loss parameters will only work when finetune_steps > 0. Otherwise, these parameters will be ignored.

The fine-tuning parameters are available in the following nixtlar functions:

  • nixtla_client_forecast
  • nixtla_client_historic
  • nixtla_client_cross_validation

3. Example

In this section, we will show a simple example where fine-tuning improves the accuracy of the forecast. We will forecast the last 24 hours of the electricity consumption dataset included in nixtlar.

df <- nixtlar::electricity

train_df <- df |> 
  dplyr::group_by(unique_id) |> 
  dplyr::slice(1:(dplyr::n()-24)) 

test_df <- df |> 
  dplyr::group_by(unique_id) |> 
  dplyr::slice_tail(n = 24)

After splitting the data into training and test sets, we will generate a forecast using the zero-shot model of TimeGPT and another two using finetune_steps and finetune_depth.

fc_zeroshot <- nixtlar::nixtla_client_forecast(train_df, h = 24)
#> Frequency chosen: h
fc_finetune <- nixtlar::nixtla_client_forecast(train_df, h = 24, finetune_steps = 100)  
#> Frequency chosen: h
fc_finetune_depth <- nixtlar::nixtla_client_forecast(train_df, h = 24, finetune_steps = 100, finetune_depth = 2) 
#> Frequency chosen: h

Next, we will evaluate the accuracy of the predictions using the Mean Absolute Error (MAE), a widely used metric for evaluating forecasts. Note that we need to convert the timestamps in the test set to merge them with the forecasts. This is because the nixtla_client_forecast function returns timestamps as date objects for plotting with the nixtla_client_plot function.

test_df$ds <- lubridate::ymd_hms(test_df$ds)

compute_mae <- function(forecast, test){
  res <- merge(test, forecast, by = c("unique_id", "ds")) |> 
    dplyr::mutate(abs_error = abs(y-TimeGPT))
  
  mae <- round(mean(res$abs_error), 2)
  return(mae)
}

print(paste0("MAE zero-shot model: ", compute_mae(test_df, fc_zeroshot)))
#> [1] "MAE zero-shot model: 4.35"
print(paste0("MAE fine-tune model: ", compute_mae(test_df, fc_finetune)))
#> [1] "MAE fine-tune model: 4.02"
print(paste0("MAE fine-tune model with depth: ", compute_mae(test_df, fc_finetune_depth)))
#> [1] "MAE fine-tune model with depth: 3.85"

4. Final recommendations

Fine-tuning can involve a trial-and-error process. We recommend monitoring the performance of your model and adjusting the fine-tuning parameters as needed. Keep in mind that fine-tuning may lead to longer training times and can increase the risk of overfitting, so use it with caution.