The refugee crisis of 2015 became a major issue of both national and pan-European debate. Behavioral reactions among natives in the form of support for radical-right parties or leaving neighborhoods following influxes of non-Westerners are well documented, but a detailed account of how asylum seekers contribute to these dynamics remains elusive. In this paper, I study how asylum centers and refugees choosing their own residences prompt each of these two behavioral outcomes using register data for the whole of Sweden (2013–2018). The analyses show a divergence depending on the particular type of refugee exposure experienced and the specific behavior under analysis. Only increased radical-right support is observed following the establishment of a new asylum center, whereas greater native out-mobility is found following refugees self-selecting into native-based areas.