In recent years, federated learning has garnered widespread attention due to its ability to train machine learning models across distributed devices while ensuring data privacy. However, little attention has been paid to user needs and personalization in this context, particularly in the field of driver action recognition. In this presentation, I will introduce the challenges of federated learning in driver action recognition and the rationale behind the poor performance of non-user-centric frameworks. Additionally, I will present our efforts to advance the development of personalized driver action recognition, which include personalized and peer-to-peer federated learning frameworks. Through this presentation, my objective is to highlight the potential of personalized driver action recognition and advocate for the deployment of federated learning in practical connected and automated vehicles.