Adults with Down syndrome have an increased risk of aging-related physical and mental health conditions and experience them at an earlier age than the general population. There is a need to investigate modifiable lifestyle factors that may reduce risk for these conditions. The present study investigated the associations between physical activity (i.e., sedentary behavior and moderate-to-vigorous activity) assessed via accelerometer across 7 days and caregiver-reported physical and mental health of 66 nondemented middle-aged adults with Down syndrome aged 25–55 years (52% female). Regression analyses indicated that more time spent in moderate intensity physical activity was associated with less risk of sleep apnea (β = −0.031, p = 0.004) and endocrine/metabolic conditions (β = −0.046, p = 0.009), and lower total number of physical health conditions (β = −0.110, p = 0.016) and anxiety disorders (β = −0.021, p = 0.049) after controlling for relevant sociodemographics. After also adjusting for body-mass-index (BMI), the association between time spent in moderate intensity physical activity and sleep apnea (β = −0.035, p = 0.002), endocrine/metabolic conditions (β = −0.033, p = 0.045) and total physical health (β = −0.091, p = 0.026) remained significant. Unexpectedly, time spent in sedentary behavior was negatively associated with musculoskeletal conditions (β = −0.017, p = 0.044). Findings indicate important associations between physical activity in everyday life and the physical and mental health of adults with Down syndrome. Social policies and interventions aimed at reducing time spent sitting around (i.e., sedentary behavior) and encouraging moderate-to-vigorous activity may be a low-burden and low-cost mechanism for fostering healthy physical and mental aging in the Down syndrome population.