This article provides an ethnographic account of how urban refugees and legal protection officers from the United Nations High Commissioner for Refugees in Kampala, Uganda compete to define the present by way of a struggle to manage the past. Taking the bonds between the visible and invisible cityscapes of Kampala as a focal point for the inquiry, the article juxtaposes the practices involved in recording official history with the scattered memories circulating in Kampala’s urban refugee community. The article shows how urban refugee governance is produced through bureaucratic records of the past, regulatory practices, and the politics of exclusion. It reconstructs refugees’ experiences of rejection and mistreatment as physical mappings of Kampala, in which the creation and closure of urban spaces give meaning to the idea of “protection space” and urban refugeehood. The ambition is to begin to develop a critique of urban refugee management by outlining a “shadowgraphy” of Kampala from the perspective of the urban displaced.