In this paper, an optimal tracking guidance method for Reusable Launch Vehicles (RLV) in reentry phase is proposed based on improved Adaptive Dynamic Programming (ADP) with experience reply (ER). The trajectory-tracking problem is transformed into a nonlinear affine optimization problem and solved by an infinite horizon optimal control scheme. An actor-critic ADP with novel network weight tuning algorithms is developed. By introducing the experience replay technique, the persistence of the excitation requirement can be assessed while updating the critic neural network and therefore the generalization performance is improved for the ADP based controller. Through Lyapunov stability theory, it is proved that the proposed control scheme can guarantee all states of the RLV system to be uniformly ultimately bounded. Simulations of both simple analytical nonlinear system and RLV guidance system under model uncertainty are conducted. In both cases better performances in terms of smoothness and accuracy are achieved compared with noise expansion, demonstrating the effectiveness and advantages of the proposed method for the optimal trajectory-tracking guidance of RLV.