This paper proposes a method for modeling the dynamic motion of the fabric using the oriented bounding box (OBB) and applies it to the collision avoidance between the fabric and the obstacle. The dynamic motion of the fabric cannot be modeled accurately because the fabric motion and deformation are affected by many hard-to-estimate factors. In the proposed method, the fabric state is represented by the OBB instead of a detailed fabric model. The fabric motion is represented by a time series of OBBs and modeled by the proposed OBB-LSTM. Using the OBB-LSTM, a simple dynamic collision avoidance method is designed to plan a robot endpoint trajectory that connects the start point and the goal point without collision between the fabric attached to the robot end-effector and the obstacle. The effectiveness of the proposed method is demonstrated in collision avoidance experiments under different obstacle settings using a real robot system.