I address the empirical properties of the popular robust LM tests of Anselin et al. (1996) for the specification of spatial models when employed in a scenario characterized by unobserved common factors with idiosyncratic loadings. I describe the small-sample behavior by way of simulation, without deriving any analytical results. I build upon the analysis in Millo (2025), extending it from homogeneous time effects to common factors with heterogeneous loadings, a very common setting, e.g., in empirical macroeconometrics. As in the former paper, I document severe distortions in the empirical size and power of the spatial tests when omitting the common factors. Then, I evaluate the strategy of controlling for the heterogeneity by augmentation, including simple (TFE) or interactive fixed effects (IFE) in the test specification. Unlike the homogeneous cases, I find that the correction to the test power may come at a non-negligible cost in terms of size distortion: for some combinations of sample sizes, in particular for short panels, IFE-corrected tests can be severely over-rejecting. This is traced back to a well-known incidental parameter problem. TFE-corrected tests can instead suffer from low power. Nevertheless, either form of augmentation is preferable to ignoring time effects when potentially present.