Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training

Liu, Risheng1,2,3; Liu, Xuan1,2; Zeng, Shangzhi4; Zhang, Jin5,6; Zhang, Yixuan5

2022

38th International Conference on Machine Learning (ICML)

2640-3498

INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162

JUL 17-23, 2022

null,Baltimore,MD

1269 LAW ST, SAN DIEGO, CA, UNITED STATES

JMLR-JOURNAL MACHINE LEARNING RESEARCH

Recently, Optimization-Derived Learning (ODL) has attracted attention from learning and vision areas, which designs learning models from the perspective of optimization. However, previous ODL approaches regard the training and hyper-training procedures as two separated stages, meaning that the hyper-training variables have to be fixed during the training process, and thus it is also impossible to simultaneously obtain the convergence of training and hyper-training variables. In this work, we design a Generalized Krasnoselskii-Mann (GKM) scheme based on fixed-point iterations as our fundamental ODL module, which unifies existing ODL methods as special cases. Under the GKM scheme, a Bilevel Meta Optimization (BMO) algorithmic framework is constructed to solve the optimal training and hypertraining variables together. We rigorously prove the essential joint convergence of the fixed-point iteration for training and the process of optimizing hyper-parameters for hyper-training, both on the approximation quality, and on the stationary analysis. Experiments demonstrate the efficiency of BMO with competitive performance on sparse coding and real-world applications such as image deconvolution and rain streak removal.

Corresponding

English

CPCI-S

National Natural Science Foundation of China["61922019","61733002","62027826","11971220"] ; National Key R&D Program of China[2020YFB1313503] ; major key project of PCL[PCL2021A12] ; Shenzhen Science and Technology Program[RCYX20200714114700072] ; Guangdong Basic and Applied Basic Research Foundation[2022B1515020082]

Computer Science

Computer Science, Artificial Intelligence

WOS:000900064903044

Web of Science

1.Dalian Univ Technol, DUT RU Int Sch Informat Sci & Engn, Dalian, Liaoning, Peoples R China
2.Key Lab Ubiquitous Network & Serv Software Liaoni, Dalian, Liaoning, Peoples R China
3.Peng Cheng Lab, Shenzhen, Guangdong, Peoples R China
4.Univ Victoria, Dept Math & Stat, Victoria, BC, Canada
5.Southern Univ Sci & Technol, SUSTech Int Ctr Math, Dept Math, Shenzhen, Guangdong, Peoples R China
6.Natl Ctr Appl Math Shenzhen, Shenzhen, Guangdong, Peoples R China

Liu, Risheng,Liu, Xuan,Zeng, Shangzhi,et al. Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training[C]. 1269 LAW ST, SAN DIEGO, CA, UNITED STATES:JMLR-JOURNAL MACHINE LEARNING RESEARCH,2022.