This paper proposes a Bellman Deviation algorithm for the detection of man-in-the-middle (MITM) attacks occurring when an agent controls a Markov Decision Process (MDP) system using  model-free reinforcement learning. We show an intuitive necessary and sufficient “informational advantage” condition  for  the proposed algorithm to guarantee the detection of attacks  with high probability, while  also avoiding false alarms.