Why do many people underestimate economic and racial inequality and maintain that theirs is a meritocratic society? Existing work suggests that people are rationalizing, misinformed, or misled. This article proposes an additional explanation: Inequality itself makes economic and racial disparities difficult to understand. In unequal societies, individuals establish their networks at formative institutions patterned by class and race. As a result, they unwittingly condition on key causal pathways when making descriptive and causal inferences about inequality. We use a simple agent-based model to show that, under circumstances typical to highly stratified societies, individuals will underestimate the extent of economic and racial inequality, downplay the importance of inherited advantages, and overestimate the relative importance of individual ability. Moreover, we show that they will both underestimate the extent of racial discrimination and overestimate its relative importance. Because segregated social worlds bias inference in these ways, all individuals (rich and poor) have principled reasons to favor less redistribution than they would if their social worlds were more integrated.