Weight noise injection-based MLPs with group lasso penalty: Asymptotic convergence and application to node pruning

Article Type

Research Article

Publication Title

IEEE Transactions on Cybernetics

Abstract

The application and theoretical analysis of fault tolerant learning are very important for neural networks. Our objective here is to realize fault tolerant sparse multilayer perceptron (MLP) networks. The stochastic gradient descent method has been employed to perform online learning for MLPs. For weight noise injection-based network models, it is a common strategy to add a weight decay regularizer while constructing the objective function for learning. However, this l2-norm penalty does not generate sparse optimal solutions. In this paper, a group lasso penalty term is used as a regularizer, where a group is defined by the set of weights connected to a node from nodes in the preceding layer. Group lasso penalty enables us to prune redundant hidden nodes. Due to its nondifferentiability at the origin, a smooth approximation of the group lasso penalty is developed. Then, a rigorous proof for the asymptotic convergence of the learning algorithm is provided. Finally, some simulations have been performed to verify the sparseness of the network and the theoretical results.

First Page

4346

Last Page

4364

DOI

10.1109/TCYB.2018.2864142

Publication Date

12-1-2019

Link to Full Text

Share

COinS