Ample evidence has indicated shifts in distribution of fish populations in response to environmental stress. However, most studies focused at the whole population scale. This neglects the spatial dynamics between groups of different body size (body size groups), that fundamentally shapes the spatial structure of a population. Here, we explored the mechanisms that modulate spatial dynamics of body size groups, and applied our analyses to three North Sea fish populations which experienced severe declines in biomass from 1977 to 2019: Atlantic cod (Gadus morhua), haddock (Melanogrammus aeglefinus), and whiting (Merlangius merlangius). All three populations exhibited strong declines in the overlapped area between body size groups in winter over 43 years, yet their mechanisms differed. These declines were either due to (1) different magnitudes of contraction of the distribution area of body size groups; and/or (2) different speeds and directions of spatial shift among various body size groups, both increasing spatial segregation within populations. These patterns were either associated with ocean warming, and/or declining population biomass, and such associations often varied according to distinct body size groups. Our analytical approach provides a powerful tool for identifying vulnerable populations under environmental stress and can be generalized to study a variety of size/age structured populations at various ecosystem types.