Non-IID federated learning with Mixed-Data Calibration
- 1 Laurel Springs School
- 2 Johns Hopkins University
* Author to whom correspondence should be addressed.
Abstract
Federated learning (FL) is a privacy-preserving and collaborative machine learning approach for decentralized data across multiple clients. However, the presence of non-independent and non-identically distributed (non-IID) data among clients poses challenges to the performance of the global model. To address this, we propose Mixed Data Calibration (MIDAC). MIDAC mixes M data points to neutralize sensitive information in each individual data point and uses the mixed data to calibrate the global model on the server in a privacy-preserving way. MIDAC improves global model accuracy with low computational overhead while preserving data privacy. Our experiments on CIFAR-10 and BloodMNIST datasets validate the effectiveness of MIDAC in improving the accuracy of federated learning models under non-IID data distributions.
Keywords
Machine Learning, Federated Learning, Non-IID, Data Privacy, Global Model Calibration
[1]. Mahesh, Batta. (2019). Machine Learning Algorithms -A Review. 10.21275/ART20203995.
[2]. Biljana L. Risteska Stojkoska, Kire V. Trivodaliev. 2017. A review of Internet of Things for smart home: Challenges and solutions. (2017). Journal of Cleaner Production, Volume 140, Part 3.
[3]. Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B. Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M. Paixão, Filipe Mutz, Lucas de Paula Veronese, Thiago Oliveira-Santos, Alberto F. De Souza. 2021. Self-driving cars: A survey. (2021). Expert Systems with Applications, Volume 165.
[4]. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). (2016).
[5]. H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2023. Communication-Efficient Learning of Deep Networks from Decentralized Data. (2023). arXiv: 1602.05629 [cs.LG].
[6]. Peter Kairouz et al. 2021. Advances and Open Problems in Federated Learning. (2021). arXiv: 1912.04977 [cs.LG].
[7]. Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. 2021. Federated Learning on Non-IID Data Silos: An Experimental Study. (2021). arXiv: 2102.02079 [cs.LG].
[8]. Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated Optimization in Heterogeneous Networks. (2020). arXiv: 1812.06127 [cs.LG].
[9]. Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. 2020. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. (2020). arXiv: 2007.07481 [cs.LG].
[10]. Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. 2021. FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. (2021). arXiv: 2102.07623 [cs.LG].
[11]. Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. 2021. No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data. (2021). arXiv: 2106.05001 [cs.LG].
[12]. Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. 2020. The Non-IID Data Quagmire of Decentralized Machine Learning. (2020). arXiv: 1910.00189 [cs.LG].
[13]. Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. 2021. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. (2021). arXiv: 1910.06378 [cs.LG].
[14]. Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. (2009).
[15]. Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. 2023. MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification. (2023). arXiv: 2110.14795 [cs.CV].
[16]. Seungjin Choi, Andrzej Cichocki, Hyung-Min Park, and Soo-Young Lee. 2005. “Blind source separation and independent component analysis: A review.” Neural Information Processing-Letters and Reviews, 6, 1, 1–57.
[17]. Junlin Han, Weihao Li, Pengfei Fang, Chunyi Sun, Jie Hong, Mohammad Ali Armin, Lars Petersson, and Hongdong Li. 2022. Blind Image Decomposition. (2022). arXiv: 2108.11364 [cs.CV].
[18]. N. Bouguila, D. Ziou, and J. Vaillancourt. 2004. “Unsupervised learning of a finite mixture model based on the Dirichlet distribution and its application.” IEEE Transactions on Image Processing, 13, 11, 1533–1543. doi: 10.1109/TIP.2004.834664.
[19]. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. (2015). arXiv: 1409.1556 [cs.CV].
[20]. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. (2015). arXiv: 1512.03385 [cs.CV].
Cite this article
Zhang,X.;Shen,Y. (2024). Non-IID federated learning with Mixed-Data Calibration. Applied and Computational Engineering,45,168-178.
Data availability
The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.
Disclaimer/Publisher's Note
The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
About volume
Volume title: Proceedings of the 4th International Conference on Signal Processing and Machine Learning
© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and
conditions of the Creative Commons Attribution (CC BY) license. Authors who
publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons
Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this
series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published
version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial
publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and
during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See
Open access policy for details).