Semantic change detection (SCD) represents a challenging task in the interpretation of remote sensing images (RSIs), with the goal of identifying change regions and extracting semantic information from bi-temporal RSIs simultaneously. The recent integration of deep neural networks leveraging multi-task learning has shown promise in enhancing SCD performance. However, there is still a challenge in improving SCD performance, specifically in designing a fine-grained network structure that can handle the two subtasks of change region localization and semantic information recognition in parallel. In this context, a novel multi-task Siamese network, termed EGMS-Net, is proposed to boost the performance of SCD, which consists of three core components. Firstly, a coarse-to-fine multi-task Siamese network is constructed to obtain semantic information and change information at multiple levels. Secondly, an adaptive change information enhancement method based on spatialspectral collaborative attention mechanism is proposed, which can assist the accurate localization of change regions without significantly increasing the model parameters. Thirdly, a change information guidance module is developed to strengthen the interaction between multi-task branches and reduce the difficulty of network training. Experiments on two benchmark datasets demonstrate that the proposed EGMS-Net outperforms existing state-of-the-art methods in the SCD community. The code of this work will be available at https://github.com/IceStreams/EGMS-Net.