This paper presents isolated photon-hadron correlations using pp and p-Pb data collected by the ALICE detector at the LHC. For photons with |η| < 0.67 and 12 < pT < 40 GeV/c, the associated yield of charged particles in the range |η| < 0.80 and 0.5 < pT < 10 GeV/c is presented. These momenta are much lower than previous measurements at the LHC. No significant difference between pp and p-Pb is observed, with PYTHIA 8.2 describing both data sets within uncertainties. This measurement constrains nuclear effects on the parton fragmentation in p-Pb collisions, and provides a benchmark for future studies of Pb-Pb collisions.