Surface defect detection is an essential topic in the industrial inspection field. Many methods based on machine vision (MV) have been applied. However, it is still a challenging task due to the complexity of defects, including low-contrast, small objects, and irregular geometric boundaries. To deal with these problems, this paper proposes a novel multi-level feature fusion network (MFNet) with  the multi-branch structure for surface defect detection. Firstly, we extract low- and high-level features via the encoder based on ResNet34. Secondly, an improved atrous spatial pyramid pooling module is adapted to expand the receptive field of low-level features. Then, the decoder adopts a multi-branch structure to fuse multi-level features for details, and a global attention module is introduced to strengthen the effectiveness of feature fusion and detection accuracy. Finally, the optimal result from multiple outputs can be obtained by multi-branch. Extensive experiments are carried out on three representative defect datasets: CrackForest, Kolektor, and RSDDs. The quantitatively contrastive experiments prove that our method enjoys a better semantic segmentation performance in industrial defect detection, outperforming four excellent semantic segmentation networks (CrackForest: Accuracy-98.54%, F1-Score-74.71%, IoU-60.22%; Kolektor: Accuracy-99.82%, F1-Score-83.09%, IoU-71.38%; Type-Ⅰ RSDDs: Accuracy-99.79\%, F1-Score-85.09%, IoU-74.84%).