Point cloud registration is a vital task in 3D perception, with several different applications in robotics. Recent advancements have introduced neural-based techniques that promise enhanced accuracy and robustness. In this paper, we thoroughly evaluate well-known neural-based point cloud registration methods using the Point Clouds Registration Benchmark, which was developed to cover a large variety of use cases. Our evaluation focuses on the performance of these techniques when applied to real-complex data, which presents a more challenging and realistic scenario than the simpler experiments typically conducted by the original authors. With few exceptions, the results reveal a significant performance gap, with most neural-based methods performing poorly on real-complex data. However, amidst the underwhelming results, 3DSmoothNet stands out as an exception, demonstrating excellent performance across the majority of benchmark sequences. Remarkably, it outperforms a state-of-the-art feature extractor, namely FPFH, showcasing its effectiveness in capturing informative and discriminative features from point clouds. Given these results, we assert that while neural-based techniques could have advantages over conventional methods, further research is necessary to fully unlock their practical utility. We advocate for more extensive studies employing unbiased and realistic testing procedures, akin to the rigorous evaluation framework employed in this work. This paper represents a crucial step towards establishing a more robust literature in the field of point cloud registration. By exposing the limitations and advantages of existing neural-based methods, we aim to inspire future research and drive the development of more effective techniques for real-world applications.