FedACQ: adaptive clustering quantization of model parameters in federated learning

• Date: 28 November 2023
• Pages: 88-110
• DOI: https://doi.org/10.1108/IJWIS-08-2023-0128
• Published date: 28 November 2023
• Subject Matter: Information & knowledge management,Information & communications technology,Information systems,Library & information science,Information behaviour & retrieval,Metadata,Internet
• Author: Tingting Tian,Hongjian Shi,Ruhui Ma,Yuan Liu

FedACQ: adaptive clustering

quantization of model parameters

in federated learning

Tingting Tian

College of Artif‌icial Intelligence and Computer, Jiangnan University, Wuxi, China

and Laboratory of Media Design and Software Technology, Jiangnan University,

Wuxi, China

Hongjian Shi

School of Electronic Information and Electrical Engineering,

Shanghai Jiao Tong University, Shanghai, China

Ruhui Ma

School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong

University, Shanghai, China and Shanghai Key Laboratory of Scalable Computing

and Systems, Shanghai Jiao Tong University, Shanghai, China, and

Yuan Liu

College of Artif‌icial Intelligence and Computer, Jiangnan University, Wuxi, China and

Laboratory of Media Design and Software Technology, Jiangnan University, Wuxi, China

Abstract

Purpose –For privacy protection, federated learning based on data separation allows machine learning

models to be trained on remotedevices or in isolated data devices. However, due to the limited resources such

as bandwidth and power of localdevices, communication in federated learning can be much slowerthan in

local computing. This study aims to improve communication eff‌iciency by reducing the number of

communicationrounds and the size of information transmittedin each round.

Design/methodology/approach –This paper allows each user node to perform multiple local trainings, then

upload th e local mod el parame ters to a c entral se rver. The central server updates the global model parameters by

weighted averaging the parameter information. Based on this aggregation, user nodes f‌irst cluster the parameter

information to be uploaded and thenreplace each value with the mean value of its cluster. Considering the asymmetry of the

federated learning framework, adaptively select the optimal number of clusters required to compress the model information.

Findings –While maintaining the loss convergence rate similar to that of federated averaging, the test

accuracydid not decrease signif‌icantly.

Originality/value –By compressing uplink traff‌ic, the work can improve communication eff‌iciency on

dynamic networkswith limited resources.

Keywords Federated learning, Communication, Average aggregation, Clustering quantization,

Self-adaptation

Paper type Research paper

This research was funded by the National Natural Science Foundation of China und er grant number

61972182.

Since submission of this article, the following author has updated his af‌f‌iliation: Hongjian Shi is at the

Shanghai Key Laboratory of Scalable Computing and Systems, Shanghai Jiao Tong University, Shangha i,

China.

IJWIS

20,1

88

Received9 August 2023

Revised20 October 2023

Accepted22 October 2023

InternationalJournal of Web

InformationSystems

Vol.20 No. 1, 2024

pp. 88-110

© Emerald Publishing Limited

1744-0084

DOI 10.1108/IJWIS-08-2023-0128

The current issue and full text archive of this journal is available on Emerald Insight at:

https://www.emerald.com/insight/1744-0084.htm

1. Introduction

Artif‌icial intelligence (AI), as a driver of the new round of sci-tech revolut ion and industrial

transformation, has moved from laboratory research to industrial practice on a large scale. It

benef‌its from the continuous accumulationof data, breakthroughs in algorithms and the constant

increase in computing power. While bringing tremendous opportunities for economic

development and social progress, AI also contains risks and challenges, such as privacy leakage

duetodatamisuse(

Tripathy et al.,2022;Yoo et al.,2022), biased intelligent decision-making due

to data discrimination and application risks due to algorithm security (Cai et al.,2023).

AI systems rely on large amounts of data. According to content and scenarios, the data

involved in AI applications can be divided into three categories: original data generat ed by

users and identity data, data ref‌lecting the appearance of the users’behavior collected through

users’daily life behaviors, network records and App records and characteristic index data

obtained from algorithms. These data bring immeasurable business value to enterprises and

eff‌icient and convenient services to people, but they may have the potential to compromise

sensitive and private data in the process of f‌low (Guo et al.,2021;Ha and Dang, 2022 ). In this

regard, the academic community has conducted several targeted studies. Qian et al. (2022)

divide mainstream privacy protection technology into data separation (Kairouz et al.,2021;

McMahan et al.,2017), data interfer ence (Abadi et al.,2016;Phanet al.,2017), secure multiparty

computing (Hesamifard et al.,2017;Mohassel and Zhang, 2017), hardware enhancement

(Schuster et al.,2015)a nd access mode hiding (Stefanov and Shi, 2013). Each type of technology

focuses on solving the privacy issues faced at different stages and relies on dissimilar solutions.

Federated learning (FL) (Kairouz et al.,2021) is an item of data separation technology, of

which the core idea is to train AI models without collecting user data to protect private

information. Specif‌ically, FL f‌irst broadcasts models to user nodes, after which each user

node trains a local model using private data and uploads model updatesto a central server.

The central server then aggregatesthe received information and sends it back to user nodes.

Finally, each user node usesthe aggregated information to update the model parameters. FL

updates global model parameters jointly through collaborative training among local nodes.

User nodes continuously interact with the central server for model updates during the

learning cycle of the system, resulting in a high communication overhead. Considering the

large number of user nodes in the whole FL system (Liu et al., 2022), the communication

burden on the network will be huge.

To reduce the total data to be transmitted for interacting information between the central

server and the user nodes, existing research has focused on two main aspects: one is to reduce

the frequency of communication, and the other is to reduce the amount of data to be

transmitted in each round of communication. Unlike traditional federated stochastic gradient

descent (SGD), which only performs a single batch of gradient computation on randomly

selected user nodes in each communication round, the federated average (McMahan et al., 2017)

aggregation approach allows each user node to perform local training more than once based on

its computing power and then uploads its local model parameter information to a central

server. In addition to optimizing the communication frequency between the central server and

the user node, it is also feasible to compress the model parameters transmitted in

communication rounds. Communication in FL is divided into two stages: uplink and downlink.

The path that sends model parameters from the central server to the user node is downlink, and

the inverse path is uplink. Since the global average mobile upload bandwidth is less than half

of the download bandwidth (Kone

cný et al., 2016), we studies the uplink traff‌ic’scompr ession.

The main model compression methods include knowledge distillation (KD) (Fang et al.,

2022), network pruning (Whitaker and Whitley, 2022) and model quantization (Alistarh

et al.,2017;Zhou et al., 2017). KD uses small-scale student models to learn knowledge from

FedACQ

89