Using computer simulations, we analyze the bit-error rate (BER) characteristics of a high-order QAM optical transmission system repeated by two-mode phase-sensitive optical parametric amplifiers (PSAs). In this system, an idler, which is a phase-conjugated twin of a signal, propagate through the link together with the signal. One advantage of this system is that the noise figure of two-mode PSAs used as repeaters can approach 0 dB. The other advantage is that two-mode PSAs cancel out self-phase modulation of the signal at each repeater through the interaction between the transmitted signal and idler. Owing to these advantages, the BER characteristics of 16, 64, and 256QAM signals are significantly improved compared with the system using erbium-doped fiber amplifiers and phase-conjugation twin waves, provided that dispersion-shifted fibers are employed for transmission.