We study the static and dynamical properties of a harmonically confined Rouse polymer coupled to a fluctuating correlated medium, which affect each other reciprocally during their stochastic evolution. The medium is modeled by a scalar Gaussian field which can feature modes with slow relaxation and long-range spatial correlations. We show that these modes affect the long-time behavior of the average position of the center of mass of the polymer, which, after a displacement, turns out to relax algebraically towards its equilibrium value. This is a manifestation of the non-Markovian nature of the effective evolution of the position of the center of mass, once the degrees of freedom of the medium have been integrated out. In contrast, we show that the coupling to the medium speeds up the relaxation of higher Rouse modes. We further characterize the typical size of the polymer as a function of its polymerization degree and of the correlation length of the medium, particularly when the system is driven out of equilibrium via the application of a constant external driving force. Finally, we study the response of a linear polymer to a tensile force acting on its terminal monomers.