The pre-ejection period (PEP) is a widely used cardiac parameter in psychophysiological research that reflects the duration between the onset of ventricular depolarization and the opening of the aortic valve and is an established marker of sympathetic nervous system activity. While many algorithms for automated PEP extraction from electrocardiography (ECG) and impedance cardiography (ICG) signals have been proposed in the literature, they have not been systematically compared against each other. This lack of standardized algorithm comparisons is due to the absence of open-source algorithms and annotated datasets for evaluating PEP extraction algorithms. To address this issue, we introduce PEPbench, an open-source Python package with different Q-peak and B-point detection algorithms from the literature that can be combined to form comprehensive PEP extraction pipelines, and a standardized framework for evaluating PEP extraction algorithms. We use PEPbench to systematically compare 90 different algorithm combinations. All combinations are evaluated on datasets from two different studies with manually annotated Q-peaks and B-points, which we make publicly available as the first datasets with reference PEP annotations. Our results show that the algorithms can differ vastly in their performance and that the B-point detection algorithms introduce a considerable amount of error. Thus, we suggest that automated PEP extraction algorithms should be used with caution on a beat-to-beat level as their error rates are still relatively high. This highlights the need for open and reproducible benchmarking frameworks for PEP extraction algorithms to improve the quality of research findings in the field of psychophysiology. With PEPbench, we aim to take a first step towards this goal and encourage other researchers to engage in the evaluation of PEP extraction algorithms by contributing algorithms, data, and annotations. Ultimately, we hope to establish a community-driven platform, fostering innovation and collaboration in the field of psychophysiology and beyond.