Gradient Normalization Makes Heterogeneous Distributed Learning as Easy as Homogenous Distributed Learning
报告人:孙涛
报告地点:腾讯会议ID: 8898648458 密码: 115119
报告时间:2024年09月06日星期五14:00-15:00
邀请人:徐东坡
报告摘要:
This talk proposes a general gradient normalization technique that can be readily applied to numerous popular distributed learning algorithms such as compressed distributed stochastic gradient descent, federated averaging, and asynchronous distributed stochastic gradient descent. Through rigorous analysis, we prove gradient normalization removes the adverse influence of data heterogeneity on all these algorithms and enables them to achieve linear speedup rates without assuming any bounded data heterogeneity. Numerical experiments are conducted to validate all theoretical findings.
主讲人简介:
孙涛,国防科技大学计算机学院副教授,主要研究兴趣包括随机优化及其并行分布。在CCF A类、IEEE Trans和中科院一区期刊上以第一或通讯作者身份发表论文30余篇。主持国家自然科学基金面上、青年项目、国防科工局项目两项。担任CCF AI专委会执行委员、期刊《Computational Intelligence》编委,并在ICML、NeurIPS、AAAI等会议担任程序委员或领域主席。曾获湖南省杰青、中国科协青托、国防科大拔尖人才计划等支持、获国防科大青年科技创新奖和ACM中国新星奖-长沙分会。博士论文获得CCF优博提名奖以及湖南省优博士奖。