Variable Selection for Distributed Sparse Regression Under Memory Constraints

Wang, Haofeng1,2; Jiang, Xuejun2; Zhou, Min3; Jiang, Jiancheng4

2023-02-01

10.1007/s40304-022-00291-w

Communications in Mathematics and Statistics   Impact Factor & Quartile

2194-6701

2194-671X

This paper studies variable selection using the penalized likelihood method for distributed sparse regression with large sample size n under a limited memory constraint. This is a much needed research problem to be solved in the big data era. A naive divide-and-conquer method solving this problem is to split the whole data into N parts and run each part on one of N machines, aggregate the results from all machines via averaging, and finally obtain the selected variables. However, it tends to select more noise variables, and the false discovery rate may not be well controlled. We improve it by a special designed weighted average in aggregation. Although the alternating direction method of multiplier can be used to deal with massive data in the literature, our proposed method reduces the computational burden a lot and performs better by mean square error in most cases. Theoretically, we establish asymptotic properties of the resulting estimators for the likelihood models with a diverging number of parameters. Under some regularity conditions, we establish oracle properties in the sense that our distributed estimator shares the same asymptotic efficiency as the estimator based on the full sample. Computationally, a distributed penalized likelihood algorithm is proposed to refine the results in the context of general likelihoods. Furthermore, the proposed method is evaluated by simulations and a real example.

Variable selection Distributed sparse regression Memory constraints Distributed penalized likelihood algorithm

SCI

English

Corresponding

NSFC[11871263] ; NSF grant of Guangdong Province of China[2017A030313012]

Mathematics

Mathematics

WOS:000921784400001

SPRINGER HEIDELBERG

Web of Science

1.Harbin Inst Technol, Dept Math, Harbin, Peoples R China
2.Southern Univ Sci & Technol, Dept Stat & Data Sci, Shenzhen, Peoples R China
3.Hong Kong Baptist Univ United Int Coll, Beijing Normal Univ, Zhuhai, Peoples R China
4.Univ North Carolina Charlotte, Dept Math & Stat, Charlotte, NC USA

Wang, Haofeng,Jiang, Xuejun,Zhou, Min,et al. Variable Selection for Distributed Sparse Regression Under Memory Constraints[J]. Communications in Mathematics and Statistics,2023.

Wang, Haofeng,Jiang, Xuejun,Zhou, Min,&Jiang, Jiancheng.(2023).Variable Selection for Distributed Sparse Regression Under Memory Constraints.Communications in Mathematics and Statistics.

Wang, Haofeng,et al."Variable Selection for Distributed Sparse Regression Under Memory Constraints".Communications in Mathematics and Statistics (2023).