Generalization Error Analysis of Neural Networks with Gradient Based Regularization

Li, Lingfeng1; Tai, Xue-Cheng1; Yang, Jiang2,3

2022-10-01

10.4208/cicp.OA-2021-0211

Communications in Computational Physics   Impact Factor & Quartile

1815-2406

1991-7120

In this work, we study gradient-based regularization methods for neural networks. We mainly focus on two regularization methods: the total variation and the Tikhonov regularization. Adding the regularization term to the training loss is equiv-alent to using neural networks to solve some variational problems, mostly in high di-mensions in practical applications. We introduce a general framework to analyze the error between neural network solutions and true solutions to variational problems. The error consists of three parts: the approximation errors of neural networks, the quadrature errors of numerical integration, and the optimization error. We also apply the proposed framework to two-layer networks to derive a priori error estimate when the true solution belongs to the so-called Barron space. Moreover, we conduct some numerical experiments to show that neural networks can solve corresponding varia-tional problems sufficiently well. The networks with gradient-based regularization are much more robust in image applications.

Machine learning regularization generalization error image classification

SCI

English

Others

National Science Foundation of China and Hong Kong RGC Joint Research Scheme (NSFC/RGC )[2019B030301001] ; Guangdong Provincial Key Laboratory of Computational Science[NSFC-11871264] ; National Science Foundation of China[RCJC20210609103819018] ; [11961160718]

Physics

Physics, Mathematical

WOS:000882773300004

GLOBAL SCIENCE PRESS

Web of Science

1.Hong Kong Ctr Cerebrocardiovasc Hlth Engn, Shatin, 19W, Hong Kong Sci Pk, Hong Kong, Peoples R China
2.Southern Univ Sci & Technol, Dept Math, Shenzhen, Peoples R China
3.Southern Univ Sci & Technol, SUSTech Int Ctr Math, Shenzhen, Peoples R China

Li, Lingfeng,Tai, Xue-Cheng,Yang, Jiang. Generalization Error Analysis of Neural Networks with Gradient Based Regularization[J]. Communications in Computational Physics,2022,32(4).

Li, Lingfeng,Tai, Xue-Cheng,&Yang, Jiang.(2022).Generalization Error Analysis of Neural Networks with Gradient Based Regularization.Communications in Computational Physics,32(4).

Li, Lingfeng,et al."Generalization Error Analysis of Neural Networks with Gradient Based Regularization".Communications in Computational Physics 32.4(2022).