This study proposes an innovative approach to address the Capacitated Vehicle Routing Problem with Time Windows (CVRPTW) by integrating Reinforcement Learning (RL) into Evolutionary Algorithms (EAs), forming the Reinforcement Learning-assisted EA (RL-EA). While traditional EAs struggle with scalability and convergence speed, RL offers promise in dynamic decision-making. By combining the strengths of both paradigms, RL-EAs aim to enhance the state-of-the-art solutions for CVRPTW. The proposed approach seeks to optimise fleet routes while adhering to capacity and time constraints, crucial for logistics and transportation sectors. This study contributes to the advancement of optimisation techniques in complex transport scenarios, offering potential improvements in efficiency and cost effectiveness.