The International Health Regulation—State Party Annual Reporting (IHR-SPAR) and the Global Health Security Index (GHSI)) have been developed to aid in strengthening national capacities for pandemic preparedness. We examine the relationship between country-level rankings on these two indices, along with two additional indices (the Universal Health Coverage Service Coverage Index and World Bank Worldwide Governance Indicator (n = 195)) and compared them to the country-level reported COVID-19 cases and deaths (Johns Hopkins University (JHU) COVID-19 Dashboard) through 17 June 2020. Ordinary least squares regression models were used to compare weekly reported COVID-19 case and death rates per million in the first 12 weeks of the pandemic between countries classified as low, middle, and high ranking on each index, while controlling for country socio-demographic information. Countries with higher GHSI and IHR-SPAR index scores experienced fewer reported COVID-19 cases and deaths, but only for the first 8 weeks after the country’s first case. For the GHSI, this association was further limited to countries with populations below 69.4 million. For both the GHSI and IHR-SPAR, countries with a higher sub-index score in human resources for pandemic preparedness reported fewer COVID-19 cases and deaths in the first 8 weeks after the country’s first reported case. The UHC-SCI and WGI country-level rankings were not associated with COVID-19 outcomes. The associations between GHSI and IHR-SPAR scores and COVID-19 outcomes observed in this study demonstrate that these two indices, although imperfect, may have value, especially in countries with a population under 69.4 million people for the GHSI. Preparedness indices may have value; however, they should continue to be evaluated as policymakers seek to better prepare for future global public health crises.