This paper presents a novel method for identification of the sub-system parameters of a Wiener-Hammerstein nonlinear (WHNL) system that is used for modeling RF Power Amplifier characteristics. The proposed method first isolates the overall linear system from the memoryless nonlinearity by exploiting the Bussgang Decomposition method. Then, DFT analysis is used for the estimation of the inner linear system. Finally, the outer linear system parameters are updated based on the inner system estimation. The estimated systems are then used to model the target system for an in-band-full-duplex (IBFD) scenario. Performance of self-interferene cancellation (SIC) has been evaluated under the existence of Signal-of-Interest (SoI). Error vector magnitude (EVM) metric of the SoI is used to compare with a half-duplex (HD) receiver under various inner linear system parameters. SIC performance has been examined with respect to the changing power levels of the SoI and self-interference signal for various delay and gain values of a practical two-tap inner linear system. The benefit of modeling the inner linear system has been revealed by comparing the SIC performance with Hammerstein nonlinear model. The performance has also been compared to well known black box models such as Generalized Memory Polynomial (GMP) and Artificial Neural Networks (ANN).