Drought events are expected to become increasingly frequent, with potentially severe aftermaths on agriculture through direct and indirect effects on crops. It is thus necessary to understand how management practices can mitigate the impacts of droughts on yields, harmful organisms and ecosystem service providers in different soil contexts. Soil disturbance reduction is often suggested as one such practice. In this study, we investigated the effects of drought (50% reduction in natural precipitation), tillage regime (conservation vs. conventional tillage), and the pivotal soil nutrient phosphorous on crop yield, as well as on the control of weeds, pests, and pathogens. We set our manipulative experiment in 18 arable fields in Northern Italy, and drought conditions were simulated with rainout shelters. Drought had a negative effect on yields and increased the biomass and species richness of weeds. Conservation tillage had lower crop disease incidence but higher weed biomass than conventional tillage. Drought and conventional tillage both reduced the number of synergies between the different ecosystem services indicators. Soil phosphorus increased weed biomass, but decreased disease incidence in soybean. Arthropod pests and predators were not significantly affected by any of the tested variables. Against the predictions, the effects of conservation tillage on drought mitigation and ecosystem services were mixed, indicating that complex combinations of multiple interventions will be required to reduce the negative effects of drought, weeds and pests under a changing climate.