This paper introduces a systematic design method for decoupling elements, which can significantly improve the isolation between two co-located antennas, e.g. between transmit and receive antennas of an in-band full-duplex system. The design method applies the theory of characteristic modes for controlling the phase and amplitude of the scattered fields of the decoupling element, in order to optimally cancel the original incident fields which couple to the receiving antenna. We describe concisely the effects that characteristic angle, modal near-field, and modal excitation of the decoupling element have on the antenna isolation. For validating the proposed method, a planar wavetrap is designed and the isolation improvement verified with full-wave simulations. When we use the proposed method to optimize a wavetrap that is placed between two co-located patch antennas, we obtain an improvement of the isolation between the antennas by 33 dB at the centre frequency of their operational frequency band, and at least 12-dB improvement across the whole 142-MHz operational bandwidth of the two antennas. As a benchmark, the wavetrap is replaced by an absorber occupying 10 times the volume of the wavetrap. The absorber gives only 6 dB of isolation improvement, substantiating the effectiveness of the proposed wavetrap method.