Anxiety is known to dysregulate the salience, default mode, and central executive networks of the human brain, yet this phenomenon has not been fully explored across the STEM learning experience, where anxiety can impact negatively academic performance. Here, we evaluated anxiety and large-scale brain connectivity in 101 undergraduate physics students. We found sex differences in STEM-related and clinical anxiety, with longitudinal increases in science anxiety observed for both female and male students. Sex-specific relationships between STEM anxiety and brain connectivity emerged, with male students exhibiting distinct inter-network connectivity for STEM and clinical anxiety and female students demonstrating no significant within-sex correlations. Anxiety was negatively correlated with academic performance in sex-specific ways at both pre- and post-instruction. Moreover, math anxiety in male students mediated the relation between default mode-salience connectivity and course grade. Together, these results reveal complex sex differences in the neural mechanisms driving how anxiety is related to STEM learning.