The spatial pattern and community assembly of soil microbial taxa have notable meanings for biodiversity shaping and maintaining mechanisms. Important for ecosystem functionality and equilibrium, the microbial rare taxa may have distinct spatial patterns and community assembly mechanisms from those of abundant taxa. However, such information is lacking, especially for rare fungal taxa at large scales. Here, we investigated distance-decay patterns and underlying assembly mechanisms for abundant and rare fungal taxa in 129 soil samples collected across 4,000 km in Chinese Northern grasslands, based on high-throughput sequencing data. A total of 208 abundant OTUs (relative abundance > 0.1%) and 5,779 rare OTUs (relative abundance < 0.01%) were obtained, accounting for 2.73% and 75.85% of entire OTUs, respectively. Community similarities of abundant and rare fungal taxa both declined significantly (P < 0.001) with the increasing geographic distance, following the distance-decay relationship. However, the turnover rate for rare fungal taxa (0.0024 per 100 km) was almost half of that for the abundant taxa (0.0054 per 100 km) based on the binary Bray-Curtis distance. Such lower turnover for rare fungal taxa was likely due to their community assembly mechanism dominated by stochastic processes, which were less influenced by environmental variance along the geographic distance. In contrast, the assembly of abundant fungal taxa was dominated by deterministic factors including soil variables and plant traits, which changed greatly along the geographic distance. Consistently, the rare fungal taxa were less sensitive to environmental changes spatially since their turnover rate by the environmental distance was much lower than that of abundant taxa (0.0027 Vs 0.0099). In summary, our findings revealed that rare fungal taxa shaped mainly by stochastic processes had lower spatial turnover than abundant fungal taxa shaped by deterministic processes, which enhanced our understanding of the biogeography for rare microbial taxa.