Given the recent advancements in deep learning (DL), and computing capabilities, there has been significant progress in the development of systems analyzing facial behavior, evolving from images under controlled laboratory conditions, to more challenging real-world scenarios, including video-based recognition. However, DL models typically require training with large-scale datasets to provide a high level of performance. The collection and annotation of such datasets is a costly undertaking that relies on domain experts. Moreover, the annotation process is highly vulnerable to the ambiguity of expressions or Action Units (AUs) due to the bias induced by experts. This paper provides a comprehensive review of state-of-the-art weakly supervised learning (WSL) methods for facial behavior analysis based on images and videos. First, we introduce a taxonomy of relevant WSL scenarios, and then methods proposed for classification applications (using weak categorical/discrete labels), and for regression applications (using weak ordinal/dimensional labels). For both types of recognition applications (classification and regression), we provide a systematic review of state-of-the-art machine learning (ML) models for expression and AU recognition, and intensity estimation in each WSL scenario, discussing their strengths and limitations. An overview of widely used experimental methodologies (public datasets and protocols) is also provided for the performance evaluation of these state-of-the-art models. Finally, our critical analysis of the methods, along with their experimental results, provides insight into the key challenges of WSL in different scenarios, motivating future research to leverage weakly labeled data for real-world facial behavior analysis problems. Our review indicates that WSL methods may provide a cost-effective approach to train robust ML/DL models for facial behavior analysis in real-world scenarios. A website including constantlyupdated survey is provided at https://github.com/praveena2j/ awesome-Weakly-Supervised-Facial-Behavior-Analysis.
