Research indicates that mental health problems increased across the globe after the outbreak of the COVID-19 pandemic. However, there is a need for research examining specific risk factors for mental health problems, while accounting for symptoms before the pandemic. This study examined risk factors for depression and anxiety symptoms among Israeli adults following the first wave of the COVID-19 pandemic in Israel, above and beyond depression and anxiety symptoms reported 3 years before the pandemic. We performed a two-wave 3-year longitudinal study (W1 July–September 2017; W2 May–June 2020). The final sample included 578 participants who completed anxiety and depression self-report questionnaires at both waves. The W2 assessment additionally included being considered high-risk for COVID-19, and measures regarding loneliness, perceived stress, and COVID-19 worries. Both anxiety and depression symptoms were significantly higher at W2 during the pandemic. Worries related to COVID-19, perceived stress, loneliness, and prior mental health symptoms predicted depression and anxiety in W2. Additionally, being younger was associated with W2 anxiety. The current study highlights risk factors for psychological distress in light of the COVID-19 pandemic. Attention of clinicians and policy makers should be given to the important role of loneliness when screening and treating people during this pandemic.