Distinguished from most hyperspectral Anomaly Detection(AD) methods based on trainable parameter networks, the recently proposed method called AETNet eliminates the need for parameter adjustments or retraining on new test scenes by training an anomaly enhancement network on background data with false anomalies. In this letter, we achieve this by proposing a novel training and inference framework that enhances the network's background spectral feature extraction capability without any data augmentation. During training on background data, the complete network is trained using the reverse distillation framework with a spectral feature alignment mechanism to improve the network's background feature expressiveness. For inference, a pruned network is applied, composed solely of components most relevant to expressing features in the spectral dimension. This effectively reduces redundant information, enhancing both inference efficiency and anomaly detection accuracy. Experimental results demonstrate that our method outperforms state-of-theart methods on the HAD100 dataset, striking an optimal balance between detection accuracy and inference speed. Our code is available at https://github.com/cristianoKaKa/FERD.