Skilled reading is thought to rely on well-specified lexical representations that compete during visual word recognition. The establishment of these lexical representations is assumed to be driven by phonology. To test the role of phonology, we examined the prime lexicality effect (PLE), the index of lexical competition in signing deaf (N = 28) and hearing (N = 28) adult readers of Hungarian matched in age and education. We found no PLE for deaf readers even when reading skills were controlled for. Surprisingly, the hearing controls also showed reduced PLE; however, the effect was modulated by reading skill. More skilled hearing readers showed PLE, while more skilled deaf readers did not. These results suggest that phonology contributes to lexical competition; however, high-quality lexical representations are not necessarily built through phonology in deaf readers.