Scene Construction Theory suggests similar neural mechanisms for visual imagery and autobiographical memory, supporting the seeming scientific consensus that a loss of visual imagery affects autobiographical memory. Based on the Dual Coding Theory and the Reverse Hierarchy Model, we also assumed influences of visual imagery on recent visual memory and even verbal memory, although little evidence has been provided so far. Thus, in a sample of 67 congenital aphantasics (= persons without mental imagery) and 32 demographically matched controls, it was investigated whether deficits in visual imagery are associated with deficits in visual as well as verbal short-term and long-term memory. The memory tasks were theoretically selected based on task difficulty, retrieval condition, and subcategories of stimuli, as previous null findings were attributed to insensitive tasks that were solvable by aphantasics by means of non-visual alternative strategies. Significant group differences were found in all memory components, with aphantasics performing worse than non-aphantasics. Therefore, evidence was obtained for the influence of visual imagery on all memory components beyond autobiographical memory.