Joystick-based teleoperation is a dominant method for remotely controlling various types of robots, such as excavators, cranes, and space telerobotics. Our ultimate goal is to create effective methods for training and assessing human operators of joystick-controlled robots. Towards that goal, an extensive study consisting of a total of 38 experimental subjects on both simulated as well as a physical robot, using either no feedback or auditory feedback, has been performed. In this paper, we present the complete experimental setup and we report only on the 18 experimental subjects teleoperating the simulated robot. Multiple observables were recorded, including not only joystick and robot angles and timings, but also subjective measures of difficulty, personality and usability data, and automated analysis of facial expressions and blink rate of the subjects. Our initial results indicate that: First, that the subjective difficulty of teleoperation with auditory feedback has smaller variance as compared to teleoperation without feedback. Second, that the subjective difficulty of a task is linearly related with the logarithm of task completion time. Third, we introduce two important indicators of operator performance, namely the Average Velocity of Robot Joints (AVRJ), and the Correct-to-Wrong-Joystick Direction Ratio (CWJR), and we show how these relate to accumulated user experience and with task time. We conclude with a forward-looking discussion including future steps.
