Deep learning has achieved remarkable results in modulation classification under two assumptions: a large amount of labeled class-balanced data is available, and test data shares the same distribution as data used for training. However, due to dynamic wireless fading channels and hardware imperfections, it is implausible that these assumptions hold in practice. This paper proposes Model-based Data Augmentation for Deep learning-based Modulation Classification (MDA-DMC) to build a high-quality dataset from a small amount of labeled seed data. We develop two novel augmentation methods to combat channel and hardware impairments, along with two widely used augmentation methods, adding Gaussian noise and rotation. We are the first to investigate the relationship between augmentation methods and impairments due to channel and hardware imperfections. We show that MDA- DMC can successfully combat the hardware imperfections for the same channel model as the channel model of seed data. Furthermore, with a few AWGN seed data, MDA-DMC achieves an accuracy gain of up to 17.20% in fading channels. By adding a few fading seed data, MDA-DMC improves accuracy by an additional 10.78% in fading channels.

Automatic Modulation Classification (AMC) receives significant interest in the context of current and future wireless communication systems. Deep learning emerged as a powerful AMC tool, as it allows for the joint learning of discriminative features, and signal classification. However, the optimization of Deep Neural Network (DNN) architectures for AMC is a manual and time-consuming process that requires profound domain knowledge and much effort. Moreover, most proposed solutions focus mainly on classification accuracy, while optimization of network complexity is neglected. In this paper, we propose a novel bi-objective memetic algorithm, BO-NSMA, to search optimal DNN architectures for AMC to maximize classification accuracy and minimize network complexity. We show that BO-NSMA, with a small initial population of six individuals and only ten generations, finds a DNN architecture that outperforms all human-crafted State-of-the-Art (SoA) models. BO-NSMA discovered the first low-complexity Convolutional Neural Network (CNN)-based model, which achieves slightly better performance than costly Recurrent Neural Network (RNN)-based approaches, allowing a 2.8-fold reduction in network complexity with 0.7% performance improvement. Compared to its counterparts from Network Architecture Search (NAS), BO-NSMA finds the best architecture, which achieves up to 18.24% accuracy gain and up to a 78.71-fold reduction in network complexity.