The use of face masks and coverings has been a central component of efforts to mitigate the impact of the COVID-19 pandemic and has been legally mandated in some countries. Most academic studies to date, however, have focussed primarily on its effectiveness in reducing SARS-CoV-2 transmission, largely neglecting the social dimensions of mask mandates. In this narrative interview-based study, we consider experiences of face masks, with a particular focus on groups considered to be at a potential disadvantage from compulsory masking. Drawing on 40 telephone, video-call and e-mail interviews, we highlight the impact of inconsistent communication and the notion of mask wearing as an act of altruism on participants’ experiences. In particular, we show how intolerance towards individuals who did not wear masks could result in stigma and exclusion, regardless of the legitimacy of their reasons. We suggest that more is needed to mitigate the ‘dark side’ of discourses of collective effort and altruism at a time of societal stress and fracture, and to account for the needs and interests of groups for whom compulsory masking may result in further marginalisation.