Intuitive Multi-Modal Programming in a Virtual Environment

 

Programming of industrial robots is a difficult and time-consuming process. It is the main barrier of introducing automation into small and median size enterprises[Z. Pan et al., 2010]. This project aims to improve an existing offline programming tool by enabling a more visual and intuitive approach to programming. The goal is to enable people without software engineering experience to control robots. Current programming software(ABB RobotStudio is a prime example) are primarily text based. While the visualization of the robot cell is offered, the primary user interaction is through typing text in a programming language. A theoretical understanding of user interaction with the software is needed to guide the development process towards a more intuitive programming environment. Human-computer interaction can be seen as a series of information exchanges between the user and the computer. The user has a mental model of how the software should behave. Based on their current understanding the user interacts with the software. The computer receives the input, modifies its internal state and produces appropriate feedback. The user modifies their understanding of the software based on the feedback. The information exchange is continued until the user achieves their goal. This framework – human-computer interaction as a communication process – will guide this project's development. Several insights can be drawn from this framework. First, the way the user interacts with the programming environment should be as close to their understanding of robot programming as possible. The user should interact with a virtual robot, instead of interacting with textual abstractions. The way-points that robot follows, the skills it will perform, as well as the program flow should all be visualized in 3D space. Modification of robot's program should be done by direct manipulation of virtual entities in 3D space, instead of text manipulation. The second insight is that to make the communication process short, the communication should be made as expressive as possible. The user interaction should be multi-modal. The usage of natural interfaces, such as Leap Motion, Oculus Rift and haptic devices, will be explored. This will allow the user to more easily interact with a virtual environment. To conclude, the goal of this work is to create a multi-modal virtual environment in which programming of industrial robots is intuitive and simple.

 

[Z. Pan et al., 2010] Pan, Zengxi; Polden, Joseph; Larkin, Nathan; Duin, Stephen Van; Norrish, John, “Recent Progress on Programming Methods for Industrial Robots”, Robotics (ISR), 2010 41st International Symposium on and 2010 6th German Conference on Robotics (ROBOTIK), p. 1, 2010