This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. A major revision is done, compared to the previous version. Abstract: Neural networks (NNs) have been applied to predict complex ship motions for stable ship maneuvering and operations at sea, but the coupling effects of different DoFs are still not effectively modeled in black-box NNs. To this end, we propose a novel deep NN framework that explicitly models the coupling effects through factorization machines, and significantly improves the prediction accuracy without introducing much extra complexity. The proposed framework is also lightweight and frequency-aware, which serves real-time and high-frequency prediction with high accuracy. We experimentally demonstrated that the framework is not only friendly to real-time prediction and online learning, but also interpretable and with great potential to be complemented by common machine learning tricks and go deeper.