Augmented reality-based brain-computer interface (AR-BCI) system is one of the important ways to promote BCI technology outside of the laboratory due to its portability and mobility, but its performance in real-world scenarios has not been fully studied. In current study, we first investigated the effect of ambient brightness on AR-BCI performance. 5 different light intensities were set as experimental conditions to simulate typical brightness in real scenes, while the same steady-state visual evoked potentials (SSVEP) stimulus was displayed in the AR glass. The data analysis results showed that SSVEP can be evoked under all 5 light intensities, but the response intensity became weaker when the brightness increased. The recognition accuracies of SSVEP were negatively correlated to light intensity, the highest accuracies were 89.35% with FBCCA and 83.33% with CCA under 0 lux light intensity, while they decreased to 62.53% and 49.24% under 1200 lux. In addition, we designed a new SSVEP recognition algorithm, named ensemble online adaptive CCA (eOACCA), to solve the accuracy loss problem in high ambient brightness. The main strategy of eOACCA is to provide initial filters for high-intensity data by iteratively learning low-light-intensity SSVEP data. The experimental results showed that the eOACCA algorithm had significant advantages under higher light intensities (>600 lux). Compared with FBCCA, the accuracies of eOACCA under 1200 lux was increased by 12.83%. In conclusion, current study contributed to the in-depth understanding of the performance variations of AR-BCI under different lighting conditions, and was helpful to promote the AR-BCI application in complex lighting environments.