Display options
Cite
Sum J, Leung CS, Ho K. Convergence analyses on on-line weight noise injection-based training algorithms for MLPs. IEEE Trans Neural Netw Learn Syst. 2012;23(11):1827-40doi: 10.1109/TNNLS.2012.2210243.
Sum, J., Leung, C. S., & Ho, K. (2012). Convergence analyses on on-line weight noise injection-based training algorithms for MLPs. IEEE transactions on neural networks and learning systems, 23(11), 1827-40. https://doi.org/10.1109/TNNLS.2012.2210243
Sum, John, et al. "Convergence analyses on on-line weight noise injection-based training algorithms for MLPs." IEEE transactions on neural networks and learning systems vol. 23,11 (2012): 1827-40. doi: https://doi.org/10.1109/TNNLS.2012.2210243
Sum J, Leung CS, Ho K. Convergence analyses on on-line weight noise injection-based training algorithms for MLPs. IEEE Trans Neural Netw Learn Syst. 2012 Nov;23(11):1827-40. doi: 10.1109/TNNLS.2012.2210243. PMID: 24808076.
Copy Download .nbib Format: NLM
Share it on

IEEE Trans Neural Netw Learn Syst. 2012 Nov;23(11):1827-40. doi: 10.1109/TNNLS.2012.2210243.

Convergence analyses on on-line weight noise injection-based training algorithms for MLPs.

IEEE transactions on neural networks and learning systems

John Sum, Chi-Sing Leung, Kevin Ho

PMID: 24808076 DOI: 10.1109/TNNLS.2012.2210243

Abstract

Injecting weight noise during training is a simple technique that has been proposed for almost two decades. However, little is known about its convergence behavior. This paper studies the convergence of two weight noise injection-based training algorithms, multiplicative weight noise injection with weight decay and additive weight noise injection with weight decay. We consider that they are applied to multilayer perceptrons either with linear or sigmoid output nodes. Let w(t) be the weight vector, let V(w) be the corresponding objective function of the training algorithm, let α >; 0 be the weight decay constant, and let μ(t) be the step size. We show that if μ(t)→ 0, then with probability one E[||w(t)||2(2)] is bound and lim(t) → ∞ ||w(t)||2 exists. Based on these two properties, we show that if μ(t)→ 0, Σtμ(t)=∞, and Σtμ(t)(2) <; ∞, then with probability one these algorithms converge. Moreover, w(t) converges with probability one to a point where ∇wV(w)=0.

Publication Types