We describe a search for gravitational waves from compact binaries with at least one component with mass 0.2–0.1 solar masses and mass ratio q ≥ 0.1 in Advanced Laser Interferometer Gravitational-Wave Observatory (LIGO) and Advanced Virgo data collected between 2019 November 1, 15:00 UTC and 2020 March 27, 17:00 UTC. No signals were detected. The most significant candidate has a false alarm rate of 0.2 yr^-1. We estimate the sensitivity of our search over the entirety of Advanced LIGO’s and Advanced Virgo’s third observing run, and present the most stringent limits to date on the merger rate of binary black holes with at least one subsolar-mass component. We use the upper limits to constrain two fiducial scenarios that could produce subsolar-mass black holes: primordial black holes (PBH) and a model of dissipative dark matter. The PBH model uses recent prescriptions for the merger rate of PBH binaries that include a rate suppression factor to effectively account for PBH early binary disruptions. If the PBHs are monochromatically distributed, we can exclude a dark matter fraction in PBHs fPBH ≥ 0.6 (at 90 per cent confidence) in the probed subsolar-mass range. However, if we allow for broad PBH mass distributions, we are unable to rule out fPBH = 1. For the dissipative model, where the dark matter has chemistry that allows a small fraction to cool and collapse into black holes, we find an upper bound fDBH < 10^−5 on the fraction of atomic dark matter collapsed into black holes.
