Clustering is a fundamental tool of scientific analysis, ubiquitous in disciplines from biology and chemistry to astronomy and pattern recognition. We propose a novel clustering algorithm based on the natural idea that a cluster and its nearest neighbor with higher mass should be merged into one cluster, unless they both have relatively large masses and the distance between them is also relatively large. The find of mass and distance peaks reveals the mergers that don’t conform to the rule and should be removed. The algorithm is parameter-free and harnesses this idea to recognize any cluster and find the proper number of clusters and noise autonomously. Experiments on numerous synthetic and real-world data sets show the enormous versatility of the proposed algorithm that remarkably outperforms the best compared algorithm. Additionally, we also compare it with latest state-of-the-art deep clustering algorithms on several challenging image data sets. The proposed algorithm without any deep representation achieves better or close performance than deep clustering algorithms on image clustering.Our codes are available: https://github.com/JieYangBruce/TorqueClustering .