On the validity of retrospective predictive performance evaluation procedures in just-in-time software defect prediction

Song, Liyan1,2; Minku, Leandro L.3; Yao, Xin1,2

2023-09-01

10.1007/s10664-023-10341-8

EMPIRICAL SOFTWARE ENGINEERING   Impact Factor & Quartile

1382-3256

1573-7616

Just-In-Time Software Defect Prediction (JIT-SDP) is concerned with predicting whether software changes are defect-inducing or clean. It operates in scenarios where labels of software changes arrive over time with delay, which in part corresponds to the time we wait to label software changes as clean (waiting time). However, clean labels decided based on waiting time may be different from the true labels of software changes, i.e., there may be label noise. This typically overlooked issue has recently been shown to affect the validity of continuous performance evaluation procedures used to monitor the predictive performance of JIT-SDP models during the software development process. It is still unknown whether this issue could potentially also affect evaluation procedures that rely on retrospective collection of software changes such as those adopted in JIT-SDP research studies, affecting the validity of the conclusions of a large body of existing work. We conduct the first investigation of the extent with which the choice of waiting time and its corresponding label noise would affect the validity of retrospective performance evaluation procedures. Based on 13 GitHub projects, we found that the choice of waiting time did not have a significant impact on the validity and that even small waiting times resulted in high validity. Therefore, (1) the estimated predictive performances in JIT-SDP studies are likely reliable in view of different waiting times, and (2) future studies can make use of not only larger (5k+ software changes), but also smaller (1k software changes) projects for evaluating performance of JIT-SDP models.

Just-in-time software defect prediction Performance evaluation procedures Verification latency Online learning Concept drift Label noise

SCI

English

First ; Corresponding

National Natural Science Foundation of China (NSFC)[62002148] ; Guangdong Provincial Key Laboratory[62250710682] ; Program for Guangdong Introducing Innovative and Enterpreneurial Teams[2020B121201001] ; null[2017ZT07X386]

Computer Science

Computer Science, Software Engineering

WOS:001069479400003

SPRINGER

COMPUTER SCIENCE

Web of Science

1.Southern Univ Sci & Technol, Res Inst Trustworthy Autonomous Syst, Shenzhen, Peoples R China
2.Southern Univ Sci & Technol, Dept Comp Sci & Engn, Guangdong Prov Key Lab Brain inspired Intelligent, Shenzhen, Peoples R China
3.Univ Birmingham, Sch Comp Sci, Birmingham B15 2TT, England

Song, Liyan,Minku, Leandro L.,Yao, Xin. On the validity of retrospective predictive performance evaluation procedures in just-in-time software defect prediction[J]. EMPIRICAL SOFTWARE ENGINEERING,2023,28(5).

Song, Liyan,Minku, Leandro L.,&Yao, Xin.(2023).On the validity of retrospective predictive performance evaluation procedures in just-in-time software defect prediction.EMPIRICAL SOFTWARE ENGINEERING,28(5).

Song, Liyan,et al."On the validity of retrospective predictive performance evaluation procedures in just-in-time software defect prediction".EMPIRICAL SOFTWARE ENGINEERING 28.5(2023).