In this paper, we propose a novel session-based continuous authentication scheme using photoplethysmography (PPG). The supporting model is developed in  two phases: first, we utilize deep autoencoders to extract features from PPG signals, and later we adopt  Local Outlier Factor (LOF) to authenticate the user. LOF is trained only on legitimate user data, making it practical for real-world scenarios. In detail, the model is trained on session-based data and generates user signatures at the beginning of each session with a short buffer duration. Despite its simplicity, our solution achieves staggering performance: an F1 score of 91.3% and 91.0% on the CapnoBase and BIMDC benchmarking datasets, respectively. Compared to a state-of-the-art baseline model, the proposed solution reduces by  2.3%  the EER on the BIMDC dataset while attaining  a 6%+ increase in the F1 metric with respect to  the CapnoBase dataset. Additionally, the proposed solution has a faster runtime, requiring approximately 0.5 seconds to process 18,000 beats compared to the competing model’s 3.6 seconds, resulting in an  85% reduction in runtime.