Biological invasions have caused the loss of freshwater biodiversity worldwide. The interplay between adaptive responses and demographic characteristics is expected to be important for the resilience of populations to biological invasions, but the interaction between these factors is poorly understood. The native freshwater gastropod Amnicola limosa is distributed along spatial variation in impact from an invasive molluscivorous fish (Neogobius melanostomus), as well as calcium concentrations, limiting the distribution of this invader (refuges). We investigated the potential for genetic adaptation of A. limosa to the invasive predator and the low calcium habitats. We conducted pooled whole-genome sequencing of twelve gastropod populations from the Upper St. Lawrence River, complemented with a laboratory reciprocal transplant of wild F0 A. limosa to measure survival and fecundity in treatments of water calcium concentration (low/high) and round goby cue (present/absent). We quantified gene flow between the habitat types to test how population structure might interact with adaptation. We found that low calcium, uninvaded habitats could act as refugia for the gastropods from the invasive fish and provide migrants to declining invaded gastropod populations through strong gene flow (i.e., demographic rescue), which also maintained genetic diversity (i.e., genetic rescue). However, we also detected signatures of divergent selection between habitat types and evidence of low fitness of individuals from refuge populations in both habitat types. This suggests that migrants from refuges could introduce maladapted alleles to recipient populations in high calcium, invaded habitats, thereby reducing fitness via outbreeding depression and producing conflict between demographic, genetic, and evolutionary rescue.