The impact of climate warming on biodiversity loss is exacerbated not only by changes in mean but also by spatio-temporal variability in temperature. Access to refugia can mitigate the impact of thermal fluctuations amongst species. The effectiveness of refugia during periods of adverse warming scenarios, i.e., seasonal fluctuations, hotter-than-average summers, and warmer-than-average winters remains largely unexplored. Here, we study a bio-energetic consumer-resource model and identify the mixed success of refugia in maintaining species persistence and stability, depending on the amplitude of fluctuations, diverse warming scenarios, and species body size. Whilst refugia withhold otherwise inevitable extinction at high amplitude fluctuations in all the warming scenarios, at lower amplitudes, they may not provide similar benefits. This arises due to non-monotone thermal responses of their foraging efforts and monotonically increasing metabolic requirements. The qualitative difference among thermal responses leads to more energy losses rather than gains at low amplitudes. We find that refugia are most beneficial during hotter summers and least during typical seasonal fluctuations. Our results also suggest that refugia can be more favourable to species in temperate and Mediterranean regions, unlike those inhabiting tropical regions. We also consider an extreme heat wave event and observe that small-bodied species can counteract their negative effects by seeking refuge at low amplitudes. Overall, our work hints at selective adaptation to refugia - conditioned on the aggregated effect of thermal conditions of the local habitat and species body size - as a mechanism for biodiversity maintenance.