Glaciers play a critical role in our society, impacting everything from our access to clean water to the tourism industry. Their accelerated melt represents a key indicator of the changing climate, highlighting the need for efficient monitoring techniques. The traditional way of assessing glacier area change is by rebuilding glacier inventories. This often relies on manual correction of semi-automated outputs from satellite imagery, which is time-consuming and susceptible to human biases. However, recent advancements in Deep Learning have enabled significant progress towards fully automatic glacier mapping. In this work, we propose DL4GAM: a multi-modal Deep Learning-based framework for Glacier Area Monitoring, available open-source. It includes uncertainty quantification through ensemble learning and a procedure to identify the imagery with the best mapping conditions independently for each glacier. DL4GAM is trained and evaluated on the European Alps, a region for which experts estimated a retreat rate of around 1.3% over 2003-2015. We use DL4GAM to investigate the glacier evolution from 2015 to 2023 using Sentinel-2 imagery and elevation (change) maps. By employing geographic cross-validation, our models, based on U-Net ensembles, demonstrate strong generalization capabilities, accurately predicting the glacier areas. We then apply them on 2023 data and estimate the area change both at glacier level. After filtering the noisy predictions, we provide annual area change rates over 2015-2023 for about 1000 glaciers, covering around 82% of the region. After extrapolating these values to the entire region, we estimate a retreat of -1.90 {plus minus} 0.71% per year.