In this paper we propose a real-time algorithm for detecting and tracking moving objects in a video sequence. Based on the on-line boosting framework, our algorithm is able to detect an object as a member of a class, e.g. pedestrian, then a specific model for each instance of the class can be built on-line allowing at the same time robust tracking and recognition of the particular instance as it leaves and re-enters the scene. Promising experimental results have been performed on standard video sequences.