Spiral antiferromagnets beyond the spin-wave approximation: Frustrated XY and Heisenberg models on the honeycomb lattice

We examine the stability of classical states with a generic incommensurate spiral order against quantum fluctuations. Specifically, we focus on the frustrated spin-1/2XY and Heisenberg models on the honeycomb lattice with nearest-neighbor J(1) and next-nearest-neighbor J(2) antiferromagnetic couplings. Our variational approach is based on the Jastrow wave functions, which include quantum correlations on top of classical spin waves. We perform a systematic optimization of wave vectors and Jastrow pseudopotentials within this class of variational states and find that quantum fluctuations favor collinear states over generic coplanar spirals. The Neel state with Q = (0,0) extends its stability well beyond the classical value J(2)/J(1) = 1/6. Most importantly, the collinear states with Q = (0,2 pi/root 3) (and the two symmetry-related states) are found to be stable in a large regime with intermediate frustration, while at the classical level they are limited to the point J(2)/J(1) = 0.5. For large frustration, the 120. state is stabilized for finite values of J(2)/J(1) in both models.