Extending Horizon and Understanding Inference Step
Following the blog post, I want to understand the inference step better and how to extend the horizon of the forecast.
If we wanted the orange line to go beyond the blue now, what is the best way to do that? The model in the image above is from the blog post. It was trained with the following configuration:
from transformers import TimeSeriesTransformerConfig, TimeSeriesTransformerForPrediction
config = TimeSeriesTransformerConfig(
prediction_length=prediction_length, # 24 months
context_length=prediction_length*6, # context length
lags_sequence=lags_sequence,
num_time_features=len(time_features) + 1, # we'll add 2 time features ("month of year" and "age", see further)
num_static_categorical_features=1, # we have a single static categorical feature, namely time series ID
cardinality=[len(train_dataset)], # it has 366 possible values
embedding_dimension=[2], # the model will learn an embedding of size 2 for each of the 366 possible values
encoder_layers=4,
decoder_layers=4,
)
So the model is trained to predict 24 months into the future. At inference time, we use the .generate() method. Only past data is passed because this is a test set.
for batch in test_dataloader:
outputs = model.generate(
static_categorical_features=batch["static_categorical_features"].to(device),
static_real_features=batch["static_real_features"].to(device),
past_time_features=batch["past_time_features"].to(device),
past_values=batch["past_values"].to(device), # only past values, no future values
future_time_features=batch["future_time_features"].to(device),
past_observed_mask=batch["past_observed_mask"].to(device),
)
But these outputs will only be for historical data where we have the ground truth. How do we forecast beyond our training data? One thought is to recursively forecast by taking the outputs from the above model.generate(), and append that to the test data, then forecast again. Surely the extension will not be as good as the forecast where we have labels. But the end goal of a forecasting model is to provide predictions into the future (in this example, that would be projecting into the future by the prediction_length: 24 months). How do we achieve this in an efficient way given this modeling package on HuggingFace? Or is it easier to use gluonTS directly for these purposes?
so
@studygold
the batch["past_values"] is indeed from the test data but is the last context window before the very last prediction window (which is what the test-splitter does). We could have also used the very last context window from the training set (but the training dataloader has the random window splitter). The test splitter as mentioned will return the very last window before the prediction window since that is the way the test set was created.
[ ..... train data .... [context][predict] .... ]
[ ...... test data .......... [context]|[prediction window]]
|
L-> batch["past_values"] in generate
[ .... time features ......... |[time feat ]]
|
L-> batch["future_time_features"] in generate
Thus the model actually only uses the batch["future_time_features"] from the "test" data which are time features made from the unseen future date-times.
Thus to create predictions for which you have no ground truth, you will need to provide the generate function the very last context window after which you would like predictions, together with the appropriate batch["future_time_features"] for the future.
Let me know if that makes sense? To iterate, we are only using the unseen ground-truth data for plotting and evaluation metrics, the generate does not use it.
Thus to create predictions for which you have no ground truth, you will need to provide the generate function the very last context window after which you would like predictions, together with the appropriate batch["future_time_features"] for the future.
@kashif How do I do this? The two pieces of code to modify seem to be:
from gluonts.transform.sampler import InstanceSampler
from typing import Optional
def create_instance_splitter(config: PretrainedConfig, mode: str, train_sampler: Optional[InstanceSampler] = None,
validation_sampler: Optional[InstanceSampler] = None,) -> Transformation:
assert mode in ["train", "validation", "test"]
instance_sampler = {
"train": train_sampler or ExpectedNumInstanceSampler(
num_instances=1.0, min_future=config.prediction_length
),
"validation": validation_sampler or ValidationSplitSampler(
min_future=config.prediction_length
),
"test": TestSplitSampler(),
}[mode]
return InstanceSplitter(
target_field="values",
is_pad_field=FieldName.IS_PAD,
start_field=FieldName.START,
forecast_start_field=FieldName.FORECAST_START,
instance_sampler=instance_sampler,
past_length=config.context_length + max(config.lags_sequence),
future_length=config.prediction_length,
time_series_fields=[
"time_features",
"observed_mask",
],
)
and
def create_test_dataloader(
config: PretrainedConfig,
freq,
data,
batch_size: int,
**kwargs,
):
PREDICTION_INPUT_NAMES = [
"static_categorical_features",
"static_real_features",
"past_time_features",
"past_values",
"past_observed_mask",
"future_time_features",
]
transformation = create_transformation(freq, config)
transformed_data = transformation.apply(data, is_train=False)
# we create a Test Instance splitter which will sample the very last
# context window seen during training only for the encoder.
instance_sampler = create_instance_splitter(
config, "test"
) + SelectFields(PREDICTION_INPUT_NAMES)
# we apply the transformations in test mode
testing_instances = instance_sampler.apply(transformed_data, is_train=False)
# This returns a Dataloader which will go over the dataset once.
return DataLoader(IterableDataset(testing_instances), batch_size=batch_size, **kwargs)
Though the test loader may not be usable and we need to make an "out of sample data loader?" I am not sure how to do that though.
one quick way to test things out is to make a test-set by adding dummy values to the end of the target array and then the gluonts helpers etc as in blog post can be used to make the appropriate tensors for the generate function.
@kashif What do you mean by dummy values (I have no categorical data)? Do you mean extend the target array into the future with missing values? I think predicting out of sample is a standard desire for anyone making a forecast model. Could you provide a code example for how to do this? It seems like it should be supported. For example, if you fit a model in gluonTS, to predict out of sample you just need to:
# Train a DeepAR model on all data but the last 36 months
training_data, test_gen = split(dataset, offset=-36)
model = DeepAREstimator(
prediction_length=12,
freq="M",
trainer=Trainer(epochs=5)
).train(training_data)
Then predict out of sample like this:
future_pred = list(model.predict(dataset))
df["#Passengers"].plot(color="black", figsize=(15, 3))
for f in future_pred:
f.plot()
plt.show()
I was hoping there was an easy way to do it with this Transformer on Hugging face. Otherwise, wouldn't it be easier to use the Transformer on gluonTS?
@studygold
what I meant was to append an array of say [-1, ... -1] to the end of your time series target values, where the length of this array is your prediction length, and then use this resulting dataset as the test-set as per the blog. This will result in the generate function predicting the time points corresponding to the dummy values you appended. I can check if I can add this to the blog.
as mentioned, the test-splitter helper does that for you... that is what is happening with the actual test set as well... as shown in the crappy diagram above