Neurocomputing. 2019 Feb 28;331:281-288. doi: 10.1016/j.neucom.2018.11.066. Epub 2018 Nov 29.

Machine Learning on Sequential Data Using a Recurrent Weighted Average.

Neurocomputing

Jared Ostmeyer, Lindsay Cowell

PMID: 30799908 PMCID: PMC6380500 DOI: 10.1016/j.neucom.2018.11.066

Abstract

Recurrent Neural Networks (RNN) are a type of statistical model designed to handle sequential data. The model reads a sequence one symbol at a time. Each symbol is processed based on information collected from the previous symbols. With existing RNN architectures, each symbol is processed using only information from the previous processing step. To overcome this limitation, we propose a new kind of RNN model that computes a recurrent weighted average (RWA) over every past processing step. Because the RWA can be computed as a running average, the computational overhead scales like that of any other RNN architecture. The approach essentially reformulates the

Keywords: Attention Mechanism; Recurrent Neural Network; Sequences

References

1. Neural Comput. 2000 Oct;12(10):2451-71 - PubMed
2. IEEE Trans Neural Netw. 1994;5(2):157-66 - PubMed
3. Neural Comput. 1997 Nov 15;9(8):1735-80 - PubMed

Publication Types

• Journal Article

Grant support

• R01 AI097403/AI/NIAID NIH HHS/United States