Federated Sequential DecisionMaking: Bayesian Optimization,Reinforcement Learning, andBeyond

Federated learning (FL) in its classic form involves the collaborative training of supervised

learning models (e.g., neural networks) among multiple agents/clients. However, in

addition to supervised learning, many other machine learning tasks which are inherently

sequential decision-making problems, such as Bayesian optimization (BO) and reinforcement

learning (RL), also find important applications in the federated setting. For example,

the crucial problem of hyperparameter tuning of neural networks in the federated setting

calls for algorithms for federated BO; collaborative clinical treatment recommendation

among multiple hospitals is a natural application for federated RL. However, the extension

of these classic sequential decision-making algorithms into the federated setting is faced

with immense challenges. Firstly, these algorithms (e.g., BO and RL) have to be adapted

to satisfy the core principles of FL. For example, consistent with the requirement of FL,

the raw data (e.g., the history of observations in BO and the trajectories in RL) of every

agent can never be shared with the other agents. Next, it is challenging to preserve the

rigorous theoretical guarantees of these classic sequential decision-making algorithms

(e.g., the sub-linear regret upper bound of classic BO algorithms and the sample complexity

of classic policy gradient algorithms for RL) and at the same time consistently

improve their empirical performances by leveraging the federation of multiple agents.

In this regard, a number of recent works have tackled these challenges and hence introduced

federated versions of classic sequential decision-making algorithms (e.g., federated

BO and federated RL algorithms) which satisfy the core principles of FL and are both

theoretically grounded and practically effective. In light of these recent advances, this

chapter discusses federated sequential decision-making problems with a focus on recent

representative works on federated BO and federated RL, and describes open problems

and potential future directions in these areas.