chronos-forecasting

chronos-forecasting

Chronos: Pretrained (Language) Models for Probabilistic Time Series Forecasting

Stars: 1996

Visit
 screenshot

Chronos is a family of pretrained time series forecasting models based on language model architectures. A time series is transformed into a sequence of tokens via scaling and quantization, and a language model is trained on these tokens using the cross-entropy loss. Once trained, probabilistic forecasts are obtained by sampling multiple future trajectories given the historical context. Chronos models have been trained on a large corpus of publicly available time series data, as well as synthetic data generated using Gaussian processes.

README:

Chronos: Learning the Language of Time Series

preprint huggingface faq License: MIT

๐Ÿš€ News

  • 17 May 2024: ๐Ÿ› Fixed an off-by-one error in bin indices in the output_transform. This simple fix significantly improves the overall performance of Chronos. We will update the results in the next revision on ArXiv.
  • 10 May 2024: ๐Ÿš€ We added the code for pretraining and fine-tuning Chronos models. You can find it in this folder. We also added a script for generating synthetic time series data from Gaussian processes (KernelSynth; see Section 4.2 in the paper for details). Check out the usage examples.
  • 19 Apr 2024: ๐Ÿš€ Chronos is now supported on AutoGluon-TimeSeries, the powerful AutoML package for time series forecasting which enables model ensembles, cloud deployments, and much more. Get started with the tutorial.
  • 08 Apr 2024: ๐Ÿงช Experimental MLX inference support added. If you have an Apple Silicon Mac, you can now obtain significantly faster forecasts from Chronos compared to CPU inference. This provides an alternative way to exploit the GPU on your Apple Silicon Macs together with the "mps" support in PyTorch.
  • 25 Mar 2024: v1.1.0 released with inference optimizations and pipeline.embed to extract encoder embeddings from Chronos.
  • 13 Mar 2024: Chronos paper and inference code released.

โœจ Introduction

Chronos is a family of pretrained time series forecasting models based on language model architectures. A time series is transformed into a sequence of tokens via scaling and quantization, and a language model is trained on these tokens using the cross-entropy loss. Once trained, probabilistic forecasts are obtained by sampling multiple future trajectories given the historical context. Chronos models have been trained on a large corpus of publicly available time series data, as well as synthetic data generated using Gaussian processes.

For details on Chronos models, training data and procedures, and experimental results, please refer to the paper Chronos: Learning the Language of Time Series.


Fig. 1: High-level depiction of Chronos. (Left) The input time series is scaled and quantized to obtain a sequence of tokens. (Center) The tokens are fed into a language model which may either be an encoder-decoder or a decoder-only model. The model is trained using the cross-entropy loss. (Right) During inference, we autoregressively sample tokens from the model and map them back to numerical values. Multiple trajectories are sampled to obtain a predictive distribution.

Architecture

The models in this repository are based on the T5 architecture. The only difference is in the vocabulary size: Chronos-T5 models use 4096 different tokens, compared to 32128 of the original T5 models, resulting in fewer parameters.

Model Parameters Based on
chronos-t5-tiny 8M t5-efficient-tiny
chronos-t5-mini 20M t5-efficient-mini
chronos-t5-small 46M t5-efficient-small
chronos-t5-base 200M t5-efficient-base
chronos-t5-large 710M t5-efficient-large

Zero-Shot Results

The following figure showcases the remarkable zero-shot performance of Chronos models on 27 datasets against local models, task-specific models and other pretrained models. For details on the evaluation setup and other results, please refer to the paper.


Fig. 2: Performance of different models on Benchmark II, comprising 27 datasets not seen by Chronos models during training. This benchmark provides insights into the zero-shot performance of Chronos models against local statistical models, which fit parameters individually for each time series, task-specific models trained on each task, and pretrained models trained on a large corpus of time series. Pretrained Models (Other) indicates that some (or all) of the datasets in Benchmark II may have been in the training corpus of these models. The probabilistic (WQL) and point (MASE) forecasting metrics were normalized using the scores of the Seasonal Naive baseline and aggregated through a geometric mean to obtain the Agg. Relative WQL and MASE, respectively.

๐Ÿ“ˆ Usage

To perform inference with Chronos models, install this package by running:

pip install git+https://github.com/amazon-science/chronos-forecasting.git

[!TIP]
The recommended way of using Chronos for production use cases is through AutoGluon, which features ensembling with other statistical and machine learning models for time series forecasting as well as seamless deployments on AWS with SageMaker ๐Ÿง . Check out the AutoGluon Chronos tutorial.

Forecasting

A minimal example showing how to perform forecasting using Chronos models:

import pandas as pd  # requires: pip install pandas
import torch
from chronos import ChronosPipeline

pipeline = ChronosPipeline.from_pretrained(
    "amazon/chronos-t5-small",
    device_map="cuda",  # use "cpu" for CPU inference and "mps" for Apple Silicon
    torch_dtype=torch.bfloat16,
)

df = pd.read_csv("https://raw.githubusercontent.com/AileenNielsen/TimeSeriesAnalysisWithPython/master/data/AirPassengers.csv")

# context must be either a 1D tensor, a list of 1D tensors,
# or a left-padded 2D tensor with batch as the first dimension
# forecast shape: [num_series, num_samples, prediction_length]
forecast = pipeline.predict(
    context=torch.tensor(df["#Passengers"]),
    prediction_length=12,
    num_samples=20,
)

More options for pipeline.predict can be found with:

print(ChronosPipeline.predict.__doc__)

We can now visualize the forecast:

import matplotlib.pyplot as plt  # requires: pip install matplotlib
import numpy as np

forecast_index = range(len(df), len(df) + 12)
low, median, high = np.quantile(forecast[0].numpy(), [0.1, 0.5, 0.9], axis=0)

plt.figure(figsize=(8, 4))
plt.plot(df["#Passengers"], color="royalblue", label="historical data")
plt.plot(forecast_index, median, color="tomato", label="median forecast")
plt.fill_between(forecast_index, low, high, color="tomato", alpha=0.3, label="80% prediction interval")
plt.legend()
plt.grid()
plt.show()

Extracting Encoder Embeddings

A minimal example showing how to extract encoder embeddings from Chronos models:

import pandas as pd
import torch
from chronos import ChronosPipeline

pipeline = ChronosPipeline.from_pretrained(
    "amazon/chronos-t5-small",
    device_map="cuda",
    torch_dtype=torch.bfloat16,
)

df = pd.read_csv("https://raw.githubusercontent.com/AileenNielsen/TimeSeriesAnalysisWithPython/master/data/AirPassengers.csv")

# context must be either a 1D tensor, a list of 1D tensors,
# or a left-padded 2D tensor with batch as the first dimension
context = torch.tensor(df["#Passengers"])
embeddings, tokenizer_state = pipeline.embed(context)

Pretraining and fine-tuning

Scripts for pretraining and fine-tuning Chronos models can be found in this folder.

๐Ÿ”ฅ Coverage

๐Ÿ“ Citation

If you find Chronos models useful for your research, please consider citing the associated paper:

@article{ansari2024chronos,
  author  = {Ansari, Abdul Fatir and Stella, Lorenzo and Turkmen, Caner and Zhang, Xiyuan and Mercado, Pedro and Shen, Huibin and Shchur, Oleksandr and Rangapuram, Syama Syndar and Pineda Arango, Sebastian and Kapoor, Shubham and Zschiegner, Jasper and Maddix, Danielle C. and Wang, Hao and Mahoney, Michael W. and Torkkola, Kari and Gordon Wilson, Andrew and Bohlke-Schneider, Michael and Wang, Yuyang},
  title   = {Chronos: Learning the Language of Time Series},
  journal = {arXiv preprint arXiv:2403.07815},
  year    = {2024}
}

๐Ÿ›ก๏ธ Security

See CONTRIBUTING for more information.

๐Ÿ“ƒ License

This project is licensed under the Apache-2.0 License.

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for chronos-forecasting

Similar Open Source Tools

For similar tasks

For similar jobs