Despite abundant literature on personality and stigma, the role of disability type in this relationship has remained unaddressed. In the current study, we examined whether the relationship between observers’ openness to experience and agreeableness on the one hand, and social distance on the other, was moderated by the target person’s type of disability (psychiatric vs. physical). One hundred thirty-nine participants were randomly assigned to complete a social distance questionnaire referring to a vignette of a person in three conditions (physical disability, psychiatric disability, and control). A main effect of openness on social distance was found. Additionally, we found an interaction effect of agreeableness and the type of disability. Namely, the relationship between agreeableness and social distance was significant only in the physical disability condition but not in the other two conditions. To conclude, the current study emphasizes the role of personality traits in social distance toward individuals with disabilities.