The purpose of the interface test is to determine how the elements incorporated into the HCI interface effect an operator's ability to control multiple robots effectively. Specifically, testing is examining how an increase in the number of robots affects operator effectiveness.
As a user is controlling multiple robots, their mental workload increases as they switch their attention between different robots and tasks. Focused attention on the activities of a particular robots lessens awareness of the overall scenario.
The interface being tested attempts to mitigate this issue by both filtering the available information and giving constant access to an overview of the scene. It is hypothesized that the following results will be found from this set of user tests:
The test consists of:
Each participant will use the system individually. The test tasks should take approximately 42 minutes per participant. Questions on the interface will be answered during the introduction and training sessions. The target users for this system will have received some degree of training and are expected to be familiar with the interface before completing tasks. During the trials, questions will not be answered to simulate a deployment environment.
Twenty participants will be chosen. The participants will have the following characteristics:
Participants will potentially receive culinary compensation for their involvement with the experiment.
Requirements for Facility — The site for the interface test will meet the following criteria:
Independent Variable — These are the only variables changed each session:
Moderator Variables — We will categorize the participants according to the following moderator variables:
Dependent Variables — Quantitative performance measures for the validation testing will include the following:
Equipment — Equipment needed for the validation test includes:
Materials — Materials needed for the test include:
Sources of Data — The sources of data for data collection are:
Data Topics — The data collection topics will include:
Observers — Up to two observers will be permitted during each session.
Purpose — The purpose of data analysis is to discover how situational awareness and operator efficiency vary with number of robots controlled HCI.
Activities — Data analysis activities are:
The following products of the system test will be delivered:
Schedule for data collection — The schedule for conducting the data collection is shown below:
Training Session — A single robot is required to collect two samples and locate a bomb. The tester will guide the user through:
The user is first given an introduction to the experiment. The script for that interaction is:
"We are conducting a study on the scalability of our interface for robotic response to disaster situations. We invited you to participate in this study to help us refine the interface and identify potential usability improvements. We want to get your reactions and opinions by having you act as an operator responding to some simulated scenarios.
"Your personal identity will not be associated with any results from this study; a number with no connection to you will identify your results. No one will know which comments you made about the system. The purpose of the study is not to evaluate your performance, but to evaluate how well the system allows you to command multiple robots.
"First, we need some simple background information about you and your familiarity with computers."
[Complete background questionnaire]
"Today you are playing the part of a first-responder to a terrorist event. A bomb has exploded and you are a part of the first responders to the scene. You are responsible for collecting air and soil samples, and searching for additional explosive devices. You will have three scenarios to respond to: one for training where you can ask questions and two where you are operating on your own. After each of the trial scenarios, you will do a brief questionnaire about the scene and your awareness of the activities of the robots. At the end of the trials there will be another questionnaire about different parts of the interface and if you found them useful. There will also be a period for you to make any additional comments that you would like.
"The tasks will have a ten minute time limit. If you are unable to complete the task within the time limit, it is not a problem. The point is to evaluate the interface and if it did not allow you to control the robots well enough to complete the task, then the goal is to figure out how to make it better.
"If you are still willing to participate in this study, please sign this consent form."
[Complete consent form]
"For the training scenario you have to take two samples and search an area for a bomb. If your robots gets within two feet of the bomb, its bomb sensor will alert you.
"The trial scenarios will be the same except you will be taking three samples and searching for two bombs. The first time you will be using two robots and the second time you will have four.
"Do you have any question before getting started?"
The training task serves to familiarize the user with HCI. The script for this task is as follows:
"On the screen you can currently see the city block you are responsible for searching. Your robot is the little purple box. The arrow shows you the direction the robot is currently facing. To move your robot around you can right click on the map and create waypoints. Your robot will then drive to that position.
"In the same menu where you can create waypoints, you can also tell your robot to collect different types of samples. Tell the robot to go collect a dirt sample in the upper left intersection.
"Along the right side of the screen there is a status indicator for your robot to let you know its position and the status of its battery.
"At the bottom of the screen you can see information about the currently defined tasks.
"The interface has another mode that allows you to control your robot more directly. If you click the 'Manipulation Interface' button at the bottom of the screen or if you double click on a robot, you will switch to the video feed for that robot. From this screen you can use the mouse or the keyboard to control the robot's movements.
"You can return to the other view by clicking the 'Orchestration Interface' button. There is additional information available in this view that you will need in completing your tasks. Various overlays of the city can be shown by clicking the buttons below the map. The first overlay that you will need is the 'Sample Areas' view. This shows areas of probable contamination by the chemical agent that was released by the bomb. The response team needs samples from those areas and it is your job to get them.
"To practice this, have your robot take a sample from the upper area. You will need to define waypoints to control your robot's path and keep it from driving into buildings. If you get stuck the 'stall indicator' next to the battery indicator will turn red. If your robot stalls you will need to switch to the manipulation interface and help get it unstuck.
"Now that you're done with that, let's look at another overlay. The next overlay is 'Bomb Targets.' It shows areas of the map where there are likely to be additional bombs. You should send your robot to search these areas. Remember that it is often possible to see a bomb in the camera before the robot will sense it.
"Now that you've found the bomb, do you have any questions?"
"There are now two bombs and you need to take three samples. You have two robots to use. Go."
"There are now two bombs and you need to take three samples. You have four robots to use. Go."