ROG: A High Performance and Robust Distributed Training System for Robotic IoT

Guan，Xiuxian1; Sun，Zekai1,5; Deng，Shengliang1; Chen，Xusheng1; Zhao，Shixiong1; Zhang，Zongyuan1; Duan，Tianyang1; Wang，Yuexuan1; Wu，Chenshu1; Cui，Yong2; Zhang，Libo3; Wu，Yanjun3; Wang，Rui4; Cui，Heming1,5

10.1109/MICRO56248.2022.00032

2022

55th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)

1072-4451

978-1-6654-7428-3

Proceedings of the Annual International Symposium on Microarchitecture, MICRO

2022-October

336-353

1-5 Oct. 2022

Chicago, IL, USA

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Critical robotic tasks such as rescue and disaster response are more prevalently leveraging ML (Machine Learning) models deployed on a team of wireless robots, on which data parallel (DP) training over Internet of Things of these robots (robotic IoT) can harness the distributed hardware resources to adapt their models to changing environments as soon as possible. Unfortunately, due to the need for DP synchronization across all robots, the instability in wireless networks (i.e., fluctuating bandwidth due to occlusion and varying communication distance) often leads to severe stall of robots, which affects the training accuracy within a tight time budget and wastes energy stalling. Existing methods to cope with the instability of datacenter networks are incapable of handling such straggler effect. That is because they are conducting model-granulated transmission scheduling, which is much more coarse-grained than the granularity of transient network instability in real-world robotic IoT networks, making a previously reached schedule mismatch with the varying bandwidth during transmission. We present ROG, the first ROw-Granulated distributed training system optimized for ML training over unstable wireless networks. ROG confines the granularity of transmission and synchronization to each row of a layer's parameters and schedules the transmission of each row adaptively to the fluctuating bandwidth. In this way the ML training process can update partial and the most important gradients of a stale robot to avoid triggering stalls, while provably guaranteeing convergence. The evaluation shows that, given the same training time, ROG achieved about 4.9%6.5% training accuracy gain compared with the baselines and saved 20.4%50.7% of the energy to achieve the same training accuracy.

distributed training energy efficient robust training throughput wireless networks

Others

English

EI ; CPCI-S

HK RGC GRF["17202318","17207117"] ; HK ITF[GHP/169/20SZ] ; NSFC[62132009]

Computer Science ; Engineering

Computer Science, Hardware & Architecture ; Engineering, Electrical & Electronic

WOS:000886530600020

20224613109619

Bandwidth ; Budget control ; Energy efficiency ; Internet of things ; Robots ; Wave transmission

Energy Conservation:525.2 ; Information Theory and Signal Processing:716.1 ; Radio Systems and Equipment:716.3 ; Data Communication, Equipment and Techniques:722.3 ; Computer Software, Data Handling and Applications:723 ; Robotics:731.5

2-s2.0-85141723019

Scopus

1.The University of Hong Kong,Department of Computer Science,Hong Kong,Hong Kong
2.Tsinghua University,Beijing,China
3.Institute of Software,Chinese Academy of Sciences,Beijing,China
4.Eee,Southern University of Science and Technology,China
5.Pujiang Lab,Shanghai,China

Guan，Xiuxian,Sun，Zekai,Deng，Shengliang,et al. ROG: A High Performance and Robust Distributed Training System for Robotic IoT[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2022:336-353.