It is currently unclear whether early life stress (ELS) affects males and females differently. However, a growing body of work has shown that sex moderates responses to stress and injury, with important insights into sex-specific mechanisms provided by work in rodents. Unfortunately, most of the ELS studies in rodents were conducted only in males, a bias that is particularly notable in translational work that has used human imaging. Here we examine the effects of unpredictable postnatal stress (UPS), a mouse model of complex ELS, using high resolution diffusion magnetic resonance imaging. We show that UPS induces several neuroanatomical alterations that were seen in both sexes and resemble those reported in humans. In contrast, exposure to UPS induced fronto-limbic hyper-connectivity in males, but either no change or hypoconnectivity in females. Moderated-mediation analysis found that these sex-specific changes are likely to alter contextual freezing behavior in males but not in females.