Alexithymia has been associated with substance use, but the magnitude of the association has not been evaluated and sub-group differences, if any, are unknown. The aim of this meta-analysis is to systematically review the association between alexithymia and substance use (alcohol or illicit drugs). We identified studies through a systematic review of PubMed and Web of Science and obtained a total of 52 publications using the Toronto Alexithymia Scale-20 scale. Random effects meta-analysis was used to evaluate the overall and sub-group associations. Of the studies, 50 were cross-sectional and two longitudinal. Alexithymia was associated with any substance use (Cohen’s d = 0.62, 95% confidence interval [CI] 0.49–0.76), with little difference between estimates for use of alcohol or illicit drugs. A stronger association was observed for the alexithymia dimension “Difficulty in Identifying Feelings” (d = 0.64, 95% CI = 0.47–0.81) and “Difficulty in Describing Feelings” (d = 0.44, 95% CI = 0.32–0.55) than for “Externally Oriented Thinking” (d = 0.19, 95% CI = 0.09–0.28). The association was stronger in studies with clinical patient populations (d = 0.83, 95% CI = 0.62–1.05) than in those investigating general or student populations, and in studies with a majority of male rather than female participants. These findings suggest a strong overall association between alexithymia and substance use and a very strong association among clinical patient populations. The association may be stronger with the emotion-related dimensions than with the cognition-related dimension of alexithymia. As nearly all the studies were cross-sectional, more longitudinal studies are needed.