Long-term changes in health-related quality of life (HrQoL) after SARS-CoV-2 infection are common, but the causes and consequences of these changes are not well understood. HrQoL was assessed using the European Quality-of-Life-5-Dimensions-5-Level-Version (EQ-5D-5L) in 2,618 participants (56% female; aged 18–88 years) approximately 9 months (baseline) and 26 months (follow-up) after their first infection. The results were compared to 40 different demographic and clinical variables. A clinically important improvement in HrQoL was noted during the observation period. At baseline, the number of remaining symptoms from the infection (RS), fatigue (FACIT-Fatigue Scale), depressive symptoms (PHQ-8), muscle pain, age, and anxiety (GAD-7) explained 43% of the variance in HrQoL. At follow-up, fatigue, RS, depressive symptoms, anxiety, and muscle pain explained 50% of the variance. Although changes in fatigue, depressive symptoms, anxiety, and RS were associated with a change in HrQoL, the predictive value of the variables was virtually zero. The study suggests that HrQoL improves statistically significantly during the observation period. However, the extent of recovery is difficult to predict from concurrent changes in demographic and clinical variables. Besides a high RS number, fatigue is the main predictor of poor HrQoL after infection, followed by the presence of depressive and anxiety symptoms.