In this paper, we propose a novel method to improve the shape sensing accuracy of FBG for catheter by fusing FBG-based sensed shape with sparse fluoroscopic images. The main advantage of the new proposed method compared to other methods are the limited number in fluoroscopic image used during procedure while it still maintains high precision real-time 3D visualization of the catheter. To demonstrate the performance of the proposed method 2D and 3D dynamic experiments were carried out and they shows promising results. For a catheter with an embedded fiber length of 170 mm, the proposed approach can reconstruct the 3D shape with a median root mean square error of 0.54 mm were seen in the 3D experiments compared to the traditional approach of using FBG alone of 0.86 mm.