User testing to obtain consensus for discovering the terrorist

Jul 31st, 2010 | By | Category: Featured Articles, News

The adoption of Visual Analytics methodologies in security applications is an approach that could lead to interesting results. Usually, the data that has to be analyzed finds in a graphical representation its preferred nature, such as spatial or temporal relationships. Due to the nature of these applications, it is very important that key-details are made easy to identify.

The simulation of  natural or human-made disaster is a topic that has gained attention, especially after the 9/11 events. Johnson argues that the CHI communities should try to gain valuable insights from those events and try to address the unexpected challenges in hope of improving the effectiveness of emergency personnel and escape plans [2]. The study of escape plans is also an interesting direction for research: Andrienko et al [3] describes an automatic scheduling algorithm generator able to generate evacuation plans, subsequently analyzed by human operators in order to judge their feasibility. Kim et al [4], developed a tool dedicated to first responders, for the analysis and representation of sensor network data. Their system displays on a mobile device, both in 2D and in 3D, a graphical representation of the situation of people whereabouts in a building, complete with their supposed health level.

building map highlighting the position of the detonation

Figure 1. Map of the building after the bombing. The bomb is supposed to be placed in the red area

This article presents a visualization tool that graphically displays the movement of 82 employees of a government building during an evacuation of the building in which they were working, caused by an explosion of a bomb. Thirteen casual users were asked to identify potential suspects and observe what happened. Each employee wore a badge equipped with an RFID that enabled the tracking of his/her movements. Data about the employees’ movements during the time of the incident was available, and for each time instant the actual X and Y coordinate on the floor map of each employee. The real name of the employees was associated to the RFID tag number. The leading idea for the experiment was that by observing the movements of the employees, the observer could have some insights on whether or not they would be implied in the bombing.
We let a group of 13 users of Computer Science background, test our tool by experimenting with the interface. Each users was briefly instructed on how to work with the two visualization modes we provided with the tool and what the purpose of the experiment was. We then left each user alone and free to interact with the interface.
The option of looking to the movements of people in a specific time span.

Figure 2. The movement traces of people are displayed for a specific time span.

The users were able to play, pause, resume, rewind, display frame by frame an animation of each employee’s movement during the time recorded. Also a trace of the employees’ movements were available on demand (see Figure 2). Users can customize this visualization by choosing the start and end point (in terms of keyframes) of their path. In this way it is possible to show which persons moved prior to the detonation of the bomb, or which person actually passes through the area of the explosion. This features were used by only 2 of 13 users.

All of the users (except one) repeatedly watched the animation and quickly concluded where the place of the detonation was, due to the fact that most of the employees that were present in that area, after the supposed explosion of the bomb, “stopped moving” (as most of the interviewed said). This observation made them more interested in the events that occurred in the red area marked in Figure 1. By looking more closely at that part of the animation they were able to identify a number of potential suspects and witnesses to the detonation.

a person can be highlighted in order to be easily followed in a crowd

Figure 3. A useful feature is the ability of highlighting a person.

A feature that was used often was the ability to “highlight” a person (see Figure 3) and follow his/her movement through the animation (since each person is rendered as a green dot, it would have been easy to lose sight of them otherwise).
Each test usually lasted between 10 and 15 minutes. After this time span (which was not enforced), the users usually reported to us that they had “concluded their observations”, so that we could begin the interview. Every one of them, (except one) agreed that the explosion must have occurred after frame 370, which probably marks the time of detonation or the time when the alarm is activated. In fact soon after that frame, everyone starts fleeing towards their nearest exit, leaving some of the employees of the northeastern quadrant fixed on the spot (identified as probable casualties by the interviewees).

We then proceeded to ask them the same questions that are reported on the answer form. The results of this testing shows that 76,92% of the interviewed agree that Number 21 is the most probable suspect of having detonated the bomb. It is interesting that so high a percentage of the interviewed agrees on his guiltiness. We asked the reason and all the interviewed users said that “he moves from his room to a room with two occupants where he either throws the device inside the room or activates it before exiting his room; subsequently, he flees and hides on a corner wall before escaping the facility”. Some of the other users expressed their concern towards the behavior of Number 13 and 59. In fact, they suspected that number 59 tries to escape the building, then changes direction and returns inside towards another exit. Number 13 follows 59 and at some point 59 “stops moving”. Those users that noticed this suspicious behavior thought that 59 must have been a witness to the event and that s/he was killed by 13. They were not able to explain why because 59 does not seem to be very close to the zone of detonation, but since more than one person noticed it, it could become relevant. Detailed results are shown in Table 1.

Table 1. Results of the interview

When asked to determine which ones amongst the employees could be witnesses to the event, the interviewed assumed that those who could be connected by an uninterrupted straight line to the suspect could have seen something. Since data about each person’s orientation was not available, by assuming that Nr. 21 is the most probable suspect, a witnesses is supposed to be everyone that has him in his/her “line of sight”, at some point during the animation. We observed, though, that the persons that we interviewed did not manage to identify all the potential witnesses (all those that “see” Nr. 21). We think that this is probably due to the fact that they spent more time on finding persons with a suspect behavior, rather than checking which of the other persons could have seen him.

Wen we began this work all we had was a list of coordinates, employees ID and timeframes. Five main question were raised:
1) Where was the device set off?
2) Identify potential suspects and/or witnesses to the event.
3) Identify any suspects and/or witnesses who managed to escape the building.
4) Identify any casualties.
5) Describe the evacuation.
People we interviewed were able to provide answers. The answers provided were very close to the ground truth in the data. This experiment shows that with the right tool everybody can do Visual Analytic activities and solve even complex problems. A possibility to reduce errors may consist in asking more users. The definition of how many user to ask is an open question. More information can be found in the VAST 2008 Challenge IVU entry.
Contact details:
Paolo Buono buonoatdidotunibadotit  (buonoatdidotunibadotit)  
Adalberto L. Simeone simeoneatdidotunibadotit  (simeoneatdidotunibadotit)  
University of Bari, Italy
References

[1] VAST 2008 Challenges:
http://www.cs.umd.edu/hcil/VASTchallenge08/tasks.html (Last Retrieved on June the 28th, 2010)

[2] C.W. Johnson, “Applying the lessons of the attack on the world trade center, 11th September 2001, to the design and use of interactive evacuation simulations,” Proceedings of the SIGCHI conference on Human factors in computing systems, Portland, Oregon, USA: ACM, 2005, 651-660;

[3] G. Andrienko, N. Andrienko, U. Bartling, “Visual analytics approach to user-controlled evacuation scheduling,” Information Visualization, vol. 7, 2008, 89-103.

[4] S. Y. Kim, Y. Jang,  A. Mellema, D. S. Ebert, T. Collinss, “Visual Analytics on Mobile Devices for Emergency Response”, Visual Analytics Science and Technology, VAST 2007, 35-42.

Tags: , ,