Abstract: An automated and lightweight method to accurately segment optical coherence tomography (OCT) images can bring a plethora of benefits, such as the production of objective diagnostic indicators at a fast rate and the implementation in imaging devices with ease. Due to the unique imaging principle, OCT images differ from natural images as they feature layered structures stretching along the image width, instead of completely closed regions. Conventional convolutional neural networks designed for natural images are usually sub-optimal for segmenting OCT images. Therefore, it is imperative to develop a segmentation network with a strong awareness of the structural features in OCT images for more efficient predictions. In this work, we introduce a novel lightweight deformable OCT segmentation network (LiDeOCTNet) to enable a flexible and scalable feature receptive field for an accurate segmentation of the irregular structures in OCT images. When compared with the classic UNet, LiDeOCTNet achieved better performance in segmenting both retinal and endoscopic OCT images. In comparison to the state-of-the-art networks, LiDeOCTNet offered competitive results with a far more lightweight network. The simplistic design of our network may lead to a feasible OCT-aware framework to achieve reliable segmentation of OCT images in real time.