In this paper, we propose an anomalous sound detection (ASD) method that feeds audio signals through a feature extractor trained with a discriminative model and then through an anomaly detector trained with a generative model. The proposed method uses outlier exposure and Mixup to focus the feature extractor on differences between normal and other sounds in the embedding space, improving ASD results. Evaluation of the proposed method on the DCASE2020 Task 2 dataset showed a 4.7 % improvement in AUC and a 6.5 % improvement in detection performance stability compared to the conventional methods. Moreover, previous ASD studies have rarely investigated the possibility of improving detection per?formance through anomalous data during model training or model performance when this anomalous data is contaminated with normal data, even though the proportion of anomalous data used for training is an important factor. The proposed method effectively utilizes small amounts of anomalous data during training, outperforming conventional methods. It also achieves higher performance even when this anomalous training data is contaminated with normal data. Experimental results show that the proposed method improves ASD performance by 10.4 % when anomalous data is utilized during training, and 8.9 % when this anomalous data is contaminated compared to conventional methods. The source code for Serial-OE is available at https://github.com/ibkuroyagi/Serial-OE.