The characteristics of residential environments that may affect health are posited to contribute to social and race/ethnic inequities in health through the differential allocation of health-promoting resources and health-harming conditions that stem from macro-level processes that systematically sort large groups of the American population into different neighborhoods. However, few studies have examined how exposure to these neighborhood conditions is inequitably experienced by individuals over adulthood. Longitudinal studies are well positioned to contribute to our understanding of the accumulation of neighborhood (dis)advantages and their impact on health throughout the life course but must first overcome the challenge of measuring differences in neighborhood context across time and between population groups. Using a longitudinal cohort of Black and white U.S. adults followed over a 25-year period with linked census tract data, we computed a Neighborhood Vulnerability Index (NVI) that combines multiple indicators of neighborhood risk into one composite score and tested the assumption of configural, metric, and scalar invariance both longitudinally and between race/gender groups. Using confirmatory factor analysis, we identified a two-dimensional model for neighborhood vulnerability and computed an index that demonstrated multiple levels of race, gender, and race-by-gender invariance (χ2 1000.48, df 15, RMSEA 0.07, SRMR 0.02, CFI 0.98, TLI 0.97, AIC 751,272). Inequities in exposure to neighborhood vulnerability between Black and white men and women at the neighborhood level has important implications for understanding the root of social, health, and economic disparities that have persisted in the U.S. over the past several decades.