Robust LSTM

• Faculty of Economics and Business, KU Leuven, Belgium
  [christophe.croux@kuleuven.be]

• Department of Mathematics and Statistics, University of Helsinki, Helsinki, Finland
  [klaus.nordhausen@helsinki.fi

• Department of Mathematics and Statistics, University of Jyväskylä, Jyväskylä, Finland
  [mika.e.sipila@jyu.it,sara.l.taskinen@jyu.fi] ]

Abstract

Long Short-Term Memory (LSTM) models are special cases of recurrent neural networks. They have become a standard tool in the deep learning community for time series prediction. Standard LSTM models turn out not to be robust to outliers, despite the belief that deep neural networks can cope with highly non-linear and noisy patterns. In this paper we introduce a robust version of LSTM.

There are two quick fixes for the non-robustness of LSTM: (i) the scaling of the series can be done is a robust way, eg using median/MAD (ii) the least squares loss function may be replaced by a Huber Loss function. A further improvement is obtained by adding a cleaning step within an iterated version of LSTM, as in Gelper et al. [2010]. The latter is our proposed version of LSTM.

Using simulation experiments, we show that robust LSTM can cope with different types of outliers, including patches of outliers and level shifts. Finally, we investigate how robust LSTM models may be used for time series outlier detection, and how accurate such a detection method is.