Following up on a recent analysis by Psaltis et al (2020 Phys. Rev. Lett. 125 141104), we show that the observed shadow size of M87* can be used to unambiguously and robustly constrain the black hole geometry in the vicinity of the circular photon orbit. Constraints on the post-Newtonian weak-field expansion of the black hole's metric are instead more subtle to obtain and interpret, as they rely on combining the shadow-size measurement with suitable theoretical priors. We provide examples showing that post-Newtonian constraints resulting from shadow-size measurements should be handled with extreme care. We also discuss the similarities and complementarity between the EHT shadow measurements and black-hole gravitational quasi-normal modes.