Pan-tilt-zoom (PTZ) cameras, which dynamically adjust their field of view (FOV), are pervasive in large-scale scenes, such as train stations, squares, and airports. In real scenarios, PTZ cameras are required to quickly make decisions informed about where to direct its focus through contextual cues from the surrounding environment. To achieve this goal, some researches project camera videos into three-dimensional (3D) models or panoramas and allow operators to perceive spatial relationships. However, these works face several challenges in terms of real-time processing, localization accuracy, and realistic reference. To address this problem, we propose a visual expansion and real-time calibration for PTZ cameras assisted by panoramic models. We attempt to meet the demand for real-time processing with a motion estimation model for a PTZ camera, to improve calibration performance of PTZ images with only two feature point pairs, and to provide a realistic environmental context through a panoramic model. We verify our methods on both public and our self-built test scene. It can be seen from the experimental results that our method can exhibit impressive accuracy and efficiency.