In this correspondence, an efficient YOLOv3-based approach for DoA estimation named YOLO-DoA is proposed, which is implemented as a detection task to spatially separated angular boxes. DoAs of sources with confidence scores are directly predicted from the spectrum proxy with YOLO-DoA and an end-to-end estimation is realized. By combining squeeze and-excitation operation, cross stage partial connections, and an improved loss for angular box regression, the performance of YOLO-DoA is significantly enhanced.