Facial expression recognition (FER) plays an important role in human-computer interaction, psychology, and various applications. It requires significantly large annotated data with practical variations. Existing FER datasets in the uncontrolled environment are small and with a limited number of expressions. In addition, they are lacking with facial ambiguity and crosscultural diversity. To address this need, we have proposed a more comprehensive FER dataset (called FER20E) and an emotion theory, exploring a remarkable expansion to include 20 discrete facial expressions. FER20E contains more than 100,000 facial images with existing eight and twelve unexplored emotions. It takes into account cultural, environmental, and demographic variations like illumination, pose, rotation, scale, age, gender, and occlusion. To reduce the manual intervention for data annotation, we have additionally designed an automatic annotation tool by utilizing a lightweight deep convolutional neural network (DCNN). To evaluate its efficacy, various state-of-the-art models are applied to it. This research not only advances FER but also has broader implications, including enhancing security measures, improving health monitoring, and potentially aiding in the early diagnosis of mental health conditions, thereby contributing to overall well-being and mental health awareness. The code and dataset will be available at https://github.com/akstheme/FER20E.

Suspiciousness detection is crucial for anticipating potential security threats and facilitating timely intervention and risk mitigation. Enhanced by deep learning, vision-based systems can play a significant role in this area. This paper presents a computationally efficient vision-based suspiciousness estimation system cascading detection, classification, and analysis modules. It incorporates factors like suspicious objects, facial expressions, and abnormal body language. A suspicious object detector (SOD) is designed to precisely locate the objects invariant to the size, scale, rotation, translation, and occlusion. A deep convolutional neural network is implemented for body language and facial expression recognition with the image and landmark features as its input. To estimate the final suspiciousness, an algorithm (USE-riskometer) is proposed in this work by considering all the detected and analyzed factors. All the modules are separately trained with the corresponding object detection and facial expression datasets. In addition to this, a novel dataset for suspiciousness estimation is proposed in this work. This dataset includes various suspicious elements, such as weapons, fire, crowd presence, facial expressions, and body language in an uncontrolled environment. Each module, along with the dataset, is evaluated against state-of-the-art methods, demonstrating robustness. This work represents a significant advancement in preemptive security measures, leveraging advanced technologies for improved recognition of suspicious activities.