This study explores the application of machine learning in financial fraud detection and compares the performance of four models: logistic regression, support vector machine, random forest, and XGBoost. By analyzing and processing transaction data from US financial institutions, it is found that the XGBoost model performs well in processing unbalanced data sets and can maintain high precision at high recall, thereby effectively identifying fraudulent behavior and reducing false positives. Experimental results show that the XGBoost model outperforms other models in all indicators, especially in accuracy, precision, recall, and F1 value. Feature importance analysis verifies its sensitivity to key features and helps optimize feature engineering. Through the analysis of the precision-recall curve, the performance of the XGBoost model is more stable at different thresholds, which is of great significance for practical applications in financial fraud detection. The study emphasizes the key role of accurate and fast fraud detection systems in reducing losses in financial institutions and improving customer satisfaction and trust. The results of this study provide effective technical means for financial institutions and promote technological progress and innovation in the entire financial industry in preventing financial fraud.