Machine learning models are famously vulnerable to adversarial attacks: small adhoc perturbations of the data that can catastrophically alter the model predictions.
While a large literature has studied the case of test-time attacks on pre-trained
models, the important case of attacks in an online learning setting has received
little attention so far. In this work, we use a control-theoretical perspective to study
the scenario where an attacker may perturb data labels to manipulate the learning
dynamics of an online learner. We perform a theoretical analysis of the problem
in a teacher-student setup, considering different attack strategies, and obtaining
analytical results for the steady state of simple linear learners. These results enable
us to prove that a discontinuous transition in the learner’s accuracy occurs when
the attack strength exceeds a critical threshold. We then study empirically attacks
on learners with complex architectures using real data, confirming the insights of
our theoretical analysis. Our findings show that greedy attacks can be extremely
efficient, especially when data stream in small batches
