The binding of proteins and ligands is critically important in drug design. However, accurately and efficiently predicting binding affinity still remains a formidable challenge. Recent research has established that deep learning-based computational methods provide a more straightforward and effective solution for quantifying binding affinity compared to traditional experimental approaches. In this study, we introduce a novel sequence-based deep learning architecture, termed ALSRI-Net (Attention-based Long-Short-Range Interaction Network). ALSRI-Net employs self-attention and cross-attention mechanisms to distinctly capture long-range and short-range interaction features between proteins and ligands. By integrating these interaction features, ALSRI-Net delivers more precise and robust predictions. The performance of ALSRI-Net was rigorously evaluated using the PDBbind v2016 core set and PDBbind v2013 core set, which serve as benchmarks, demonstrating the superior performance of ALSRI-Net. Additionally, we conducted dimensionality reduction and visualization of the extracted long-range and short-range interaction features, as well as their integrated features, providing insights into the respective contributions of these interactions and elucidating why the integration of these features leads to enhanced predictive accuracy. Our code and models are publicly available at https://github.com/Funiverse/ ALSRI-Net.