Research on Solutions to Non-IID and Weight Dispersion

Haosen Jiang, Yuting Lan, Yihan Wang

2024

Abstract

Federated learning is an emerging basic technology of artificial intelligence. The design goal is to carry out high-efficiency machine learning among multi-participants or multi-computing nodes under the premise of ensuring information security during big data exchange, protecting terminal data and personal data privacy, and ensuring legal compliance. At the same time, federated learning also faces many challenges, such as the heterogeneity of data, that is, the problem of the non-independent and identically distributed (Non-IID), and the problem of weight dispersion. After a comprehensive review of the literature and experiments, the following conclusions are reached: For Non-IID, the SCAFFOLD algorithm uses a control variable c to correct the training direction, which is also updated when the client and server are updated. For the weight dispersion problem, this paper takes the Model-contrastive Federated Learning (MOON) algorithm as an example to analyze that the reason for the problem is that only the weight distribution of the output layer is considered, while the similarity measurement of model parameters on other layers is ignored. Based on this conclusion, this study gives suggestions for improvement and prospects for the future: Non-IID caused by distributed databases needs to reconsider the federated learning model and algorithm, and selective sampling according to the data distribution type of clients may improve the performance and stability of the federated learning system. Federated learning algorithms such as MOON, which have weight dispersion problems, can reduce the impact by removing negative sample pairs, or increase the loss of weight similarity.

Paper Citation

in Harvard Style

Jiang H., Lan Y. and Wang Y. (2024). Research on Solutions to Non-IID and Weight Dispersion. In Proceedings of the 1st International Conference on Data Science and Engineering - Volume 1: ICDSE; ISBN 978-989-758-690-3, SciTePress, pages 148-153. DOI: 10.5220/0012832600004547

in Bibtex Style

@conference{icdse24,
author={Haosen Jiang and Yuting Lan and Yihan Wang},
title={Research on Solutions to Non-IID and Weight Dispersion},
booktitle={Proceedings of the 1st International Conference on Data Science and Engineering - Volume 1: ICDSE},
year={2024},
pages={148-153},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012832600004547},
isbn={978-989-758-690-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 1st International Conference on Data Science and Engineering - Volume 1: ICDSE
TI - Research on Solutions to Non-IID and Weight Dispersion
SN - 978-989-758-690-3
AU - Jiang H.
AU - Lan Y.
AU - Wang Y.
PY - 2024
SP - 148
EP - 153
DO - 10.5220/0012832600004547
PB - SciTePress