The link between gender nonconformity and psychopathology may be due in part to negative childhood experiences resulting from other people’s reactions to gender nonconformity. The aim of this study was to test whether recalled perceived levels of parental and peer acceptance of childhood gender nonconforming behaviors and play mediate the relationship of childhood gender nonconformity with depression and social anxiety in adulthood. We also tested whether this relationship was moderated by sexual orientation and, among gay men, whether internalized homophobia was an additional mediator. All variables were measured in a large sample of male participants using self-report (n = 449 gay men, age: M = 27.8 years, SD = 6.69; and n = 296 heterosexual men, age: M = 27.4 years, SD = 6.57) in Poland. Gay men reported more childhood gender nonconformity than heterosexual men. The relationship between gender nonconformity and depressive symptoms as well as social anxiety symptoms was significant in both gay and heterosexual men. Among gay men, this relationship was partially mediated by peer but not parental acceptance of the measured aspects of gender nonconformity and internalized homophobia. Among heterosexual men, recalled perceived parental acceptance of gender nonconformity partially mediated the relationship between gender nonconformity and depressive and social anxiety symptoms. Our findings were partially in line with those found in Western European and North American samples. Although the two groups differed in their recalled perceived gender nonconformity, they did not differ in their depression or social anxiety scores. Nevertheless, childhood gender nonconformity may be an indirect risk associated with mental health symptoms, irrespective of sexual orientation. Its higher prevalence among nonheterosexual individuals makes it a particular risk for this group.