Objectives: To investigate the relationships between physical performance, quality of life (QoL), and social participation among elderly men and women with a history of osteoporosis-related fractures. Method: The study was a population-based cross-sectional study including 155 participants aged 60 to 93 years from the Swedish longitudinal investigation, “Good Aging in Skåne.” The participants had suffered fracture of the vertebrae, hip, pelvis, or ankle. Physical performance was expressed as walking speed (WS) and timed get-up-and-go (TUG). QoL was measured as using the Health Related Quality of Life (HRQoL) and Life Satisfaction (LS) scales. Social participation was defined as taking part in social, cultural, and leisure activities. Results: Lower WS and/or TUG were associated with lower HRQoL, lower LS, and a reduction in social participation, after adjustment for confounding factors. Discussion: Measurements of WS and TUG could be used to determine QoL and social participation in elderly people having sustained fractures.