Listening effort can be defined as a measure of cognitive resources used by listeners to perform a listening task. Several methods have been proposed to assess listening effort, but the reliability of these methods has not yet been thoroughly established, which is necessary before using them in research or clinical settings. This study included 32 participants who performed speech-in-noise tasks in two sessions (separated approx. 1 week apart) by listening to Sentences and Word lists presented at different signal-to-noise ratios (-9, -6, -3, and 0 dB). We assessed the test-retest reliability of the self-reported measure of listening effort and frontal midline theta (Fmθ) power, which has been proposed as a neural correlate of listening effort. The reliability of the percentage of correct words was also examined. Relative and absolute reliability was evaluated using intraclass correlation coefficients (ICC) and Bland-Altman analysis, respectively. The standard error of measurement (SEM) and the smallest detectable change (SDC) were also assessed. Overall, the reliability analysis revealed an acceptable between-session variability for the correct words and effort rating. However, Fmθ power showed high variability, which brings into question its use as a reliable correlate of listening effort.