The detection and analysis of gravitational waves (GW) has opened new windows on the universe in recent years. Our team has developed the GWSkyNet and GWSkyNet-Multi models, which are designed to operate with the limited data publicly available from the LIGO-Virgo collaboration. We employ a combination of neural networks and explainable AI techniques to predict the nature of the detected signals. The XAI part helps us understand the strengths and biases of our models.