Understanding community restoration state and the corresponding assembly mechanism is helpful to assess the restoration measures and predict community dynamics. We collected plots by shrub cover (low, medium and high) in three Caragana mircophylla shrub-encroached grasslands which were fenced since 1979, 1983, and 2003 (fencing duration) in the northern China, to explore the effect of fencing duration and shrub cover on the community restoration by vegetation investigation and phylogenetic approach. There were significant differences in community composition among different fencing duration or shrub cover treatments. Species richness in the site of fencing since 1979 or in the plots of high shrub cover was relatively higher than that in any other sites or plots. By phylogenetic analysis, functional traits were phylogenetically convergent. Based on the standardized effect sizes of mean pairwise distance (SESMPD) ranged from -1.96 to 1.96 in six out of nine plots, which suggested that stochastic processes dominated community assembly. SESMPD were lower than -1.96 in the rest three plots which indicated that competitive exclusion drove community assembly. These results indicated that the increase of fencing duration or shrub cover could enhance competitive exclusion. The present findings highlighted the importance of shrub in influencing the community composition and community assembly, supporting that shrub-encroached grassland is another stable state in the semi-arid northern China. Therefore, it is essential to distinct shrub-encroached grassland from degraded grasslands when formulating relevant conservation and management measures in the semi-arid regions.