Background: Impulsivity is closely related to the tendency to engage in risky behaviors. Previous research identified macrostructural brain alterations in individuals exhibiting impulsive behaviors. Understanding the microstructural brain changes linked to impulsivity can elucidate its underlying mechanisms and guide effective treatment strategies. In this study, we employed diffusion magnetic resonance imaging (DMRI) connectometry to investigate white matter tracts associated with impulsivity while considering potential sex differences. Methods: We enrolled 218 healthy participants from the Leipzig Study for Mind-Body-Emotion Interactions (LEMON) database. Correlations between DMRI-derived white matter changes and impulsivity were assessed using scores from the UPPS Impulsive Behavior Scale’s four subscales (lack of perseverance (PE), lack of premeditation (PM), sensation seeking (SS), and negative urgency (NU)). Results: Our findings revealed negative correlations between quantitative anisotropy (QA) values in bilateral cerebellum, middle cerebellar peduncle (MCP), and the severity of PE and PM across the cohort. Additionally, QA values within MCP, corpus callosum (CC) body, and forceps major exhibited negative correlations with SS. Conversely, QA values in forceps minor were positively correlated with PM, and QA values in both the forceps minor and bilateral cingulum showed positive correlations with SS. Remarkably, the observed correlations between UPPS subscale scores and QA value alterations within white matter tracts varied between males and females. Conclusions: Impulsivity is correlated with discernible alterations in white matter integrity across diverse tracts, including CC, cerebellum, and cingulum. Moreover, males and females show distinct patterns of correlations between white matter integrity and impulsivity.