Background With the rapid development of artificial intelligence technology, highly intelligent and unmanned factories have become an important trend. In the complex environments of smart factories, the long-term tracking and inspection of specified targets such as operators and special goods, as well as comprehensive visual recognition and decision-making capabilities throughout the whole production process, are critical components of automated unmanned factories. It is inevitable that issues such as target occlusion and disappearance will occur, thus exacerbating the long-term tracking challenge. Currently, there are no long-term tracking studies specifically addressing occlusions in these environments. Methods We first construct three new benchmark datasets in the complex workshop environment of a smart factory (referred to as SF-Complex3 data), which include challenging conditions such as complete occlusion and partial occlusion of targets. Next, utilising a brain memory inspired approach, we determine uncertainty estimation parameters: confidence, peak-to-sidelobe ratio (PSR), and average peak to correlation energy (APCE), to derive a continual learning based adaptive model update method. Additionally, we design a lightweight target detection model to automatically detect and locate targets in the initial frame and during re-detection. Finally, we integrate the algorithm with ground mobile robots and UAV-based imaging and processing equipment to build a new visual detection and tracking framework, termed SFC-RT (Smart Factory Complex Tracking and Identification). Results We conducted extensive tests on the benchmark UAV20L and SF-Complex3 datasets. Compared to state-of-the-art tracking algorithms, our proposed algorithm demonstrates an average performance improvement of 6% when facing key challenging attributes. Moreover, it can smoothly run on embedded platforms, including mobile robots and UAVs, at a real-time speed of 36.4 fps. Conclusions The proposed SFC-RT framework is shown to efficiently and accurately address the challenges of target loss and occlusion for long-term tracking within complex smart factory environments. It meets the requirements of real-time performance, robustness, and lightweight design.