Diabetic Retinopathy (DR) and Glaucoma (GL) are critical eye diseases that can lead to vision loss if not identified early. Manual diagnosis using fundus images is time-consuming and error-prone, highlighting the need for automated methods for efficient and accurate classification. Deep learning models improve diagnostic performance but still face challenges such as class imbalance and similarities between disease stages, which complicate early detection.To address these challenges, we propose an Enriched Feature-based Multi-Class (EFMC) deep learning classifier for detecting DR and GL from fundus images. The EFMC classifier operates in three stages: extracting domain-specific features through unsupervised learning, enriching these features with multi-scale filters, and classifying them using multiple CNN classifiers with a hybrid loss function. This approach effectively tackles class similarity and imbalance. Experimental results on benchmark datasets show that the EFMC system outperforms state-of-the-art models, achieving 98.75% accuracy and a 98% F1-score.