People with HIV now have increased longevity; however, their health-related quality of life (HRQoL) still lags significantly compared to people without HIV. Perceived stress negatively impacts HRQoL, whereas psychosocial resources are linked to better HRQoL. This longitudinal analysis aims to explore the buffering role of psychosocial resources on the relationship between HRQoL and perceived stress. Participants (N = 240) included 142 persons with HIV (PwH) and 98 without HIV, M(SD) = 50.9(8.1) years. Multilevel models over four study years examined longitudinal relationships between HRQoL (outcome) and perceived stress (predictor) and potential moderation by psychosocial resources (personal mastery, social support, and resilience) by HIV serostatus. Among PwH only, personal mastery (p = 0.001), social support (p = 0.015), and resilience (p = 0.029) were associated with an attenuated effect of perceived stress (less negative slopes) for physical HRQoL over time. Bolstering personal mastery, social support, and resilience may have relevance for improving physical well-being among PwH.