CA2500573A1 - Advances in nspace - system and method for information analysis - Google Patents

Abstract

Recent progress has focused on the "Sandbox" to support both ad-hoc and more formal analytical tasks. The Sandbox is a flexible and expressive thinking environment, It is a space literally focused on 'Human Interaction with Information'. This paper reviews new Oculus Sandbox capabilities and the rationale underlying them including observations of analysis work from Glass Box data. Capabilities for the Sandbox include put-this-there cognitive, automatic analysis templates, gestures for the fluid expression of thought and scalability mechanisms to support large analysis tasks.

Description

Appiicat:on numbe:'nume:Q dv ~emande:
Fi_rures: ~~ ,1 Pa~=es:
hl~-~p Unscannable items recei~red with this application -(Request original documents in File Prep. Section on the 10th Floor) Doc~,.~ments re4us aver cette demande ne pouvant titre balayes ( Commande: les documents ori~inau.Y Bans la section de preparation des dossiea au i(iieme eta~ej Background of Invention Analyst work is not sequential, and moves back and forth, from one stage to another, across multiple tasks at a moment's notice. There is a need for an integrated approach for supporting analysts." [Wright, Kapler, 2003]] An integrated work environment should provide a common visual vocabulary for analytic work, creating a mixed-initiative environment for the whole analysis workflow and a workspace ready for collaboration. It also needs to be a test bench into which new technologies can be integrated. Primarily, it is the cognitive space where the analyst will see, and interact with, more information, more quickly, with more comprehension.
The large number of potential context switches between tools/mediums constitute another kind of friction or overhead observed. Based on the observed process of building large shoeboxes in Word, (i.e. pasting sequentially usually in a single Word document all bits that appear relevant to the whole task), one could infer that evidence marshalling is particularly difficult. It must be hard to get the big picture by looking at pages and pages of text. The analyst probably relies heavily on memory to connect the dots.
Not unrelated to this is the lack of observable use of various analytical methods (e.g.
Page 1 of 36 inference networks, ACH, models, etc.). Considering the relative short term tasks and the difficulty of building say inference networks or any kind of analytical charts in Word, it is not particularly surprising. In the end, this seems to indicate that analysis content (hypothesizing, fleshing out hypotheses with evidence proving and refuting them, corroborating, evaluating the diagnosticity of the evidence, and assessing which hypotheses are most likely, etc.) is something that happens mostly in the analyst's head.
Many researchers [Heuer] have warned about the possible consequences of doing analysis primarily in one's head. The analyst is more prone to human cognitive biases and it may not be as easy to collaborate and to communicate about the analytical process and how the conclusions were reached than if analytical methods were used and so made the thinking explicit. The well known analysis "bathtub" curve [Rose, 1996], showing that most of the analyst time is spent in IR and Report Creation and almost relatively no time doing Analysis, was replicated.
References Bodnar, J.W., Warning Analysis for the Information Aye: Rethinking the Intelligence Process, Joint Military Intelligence College, December, 2003.
Heuer, Richard, Ps c~gy of Intelligence Analysis, Center for the Study of Intelligence, 1999.
Hughes, F. and D. Schum, Discovery-Proof Choice, The Art and Science of the Process of Intelligence Analysis - Preparing for the Future of Intelligence Anal,~is, Joint Military Intelligence College, 2003.
Moore, David, L. Krizan, E. Moore, Evaluatin Ig ntelli~ence: A Competency-Based Model, 2004.
Rose, Russ, Chair P 1000 Committee, P 1000 Report, 1996.
Wright, William and Kapler, Thomas, "Speaking with Analysts - Observations of Current Practices with Massive Data", submitted for consideration to the Journal of Intelligence Community Research and Development (JICRD), 2004.
Caid, W. and Pu Oing, System and Method of Context Vector Generation and Retrieval, United States patent 5,619,709, 1997 Jonker, D., W.Wright, D.Schroh, P.Proulx and Cort, Information Triage with TRIST, submitted for consideration to Conference on Intelligence Analysis 2005 Page 2 of 36 Brief Description of the Drawings A better understanding of these and other embodiments of the present invention can be obtained with reference to the following drawings and detailed description of the preferred embodiments, in which:
Figure 1 is a block diagram of a data processing system for a visualization tool;
Figure 2. TRIST and the Sandbox;
Figure 13. Assertions with Evidence Gates;
Figure 14. Powerful Finger Gesture Adds Emphasis;
Figure 15. Creating Space with Y " Editor Gestures;
Figure 16. Thumbnail Navigator in Lower Left of Sandbox;
Figure 17. nSpace Technical Architecture;
Figure 18 shows further details of the data processing system of Figure l;
Figure 19 shows an example layout of the visual interface of the data processing system;
Figure 20 shows visualisation tool modules;
Figure 21 shows an example representation for the Sandbox tool of Figure 2;
and Detailed Description of Embodiments of the Invention Data processing system for Tool Implementation The following detailed description of the embodiments of the present invention does not limit the implementation of the invention to any particular computer programming language. The present invention may be implemented in any computer programming language provided that the OS (Operating System) provides the facilities that may support the requirements of the present invention. A preferred embodiment is Page 3 of 36 implemented in the Java computer programming language (or other computer programming languages in conjunction with C/C++). Any limitations presented would be a result of a particular type of operating system, computer programming language, or data processing system and would not be a limitation of the present invention.
The Sandbox tool 12 is a flexible and expressive thinking environment that supports both ad-hoc and more formal analytical tasks. Its function is to support analysts as the orchestrators and directors of the analytic process, and is literally focused on 'Human Interaction with Information'. The Sandbox provides a fluid and flexible medium of analysis and expression. It integrates the acts of building, controlling and testing hypotheses, resulting in a natural thinking process. This is optimized by improving efficiency in interaction and control through reducing friction in the interface.
The Sandbox supports evidence marshalling as a first step to supporting working with the observations. Analysts can define layouts to reflect thought processes and assumptions, viewing multiple dimensions of the problem through the use of views of connections.
These linked, multi-dimensional views speed scanning of information by an order of magnitude.
Capabilities for the Sandbox include such as but not limited to:
put-this-there cognition;
automatic analysis templates;
gestures for the fluid expression of thought; and scalability mechanisms to support large analysis tasks.
The Sandbox is part of an integrated cognitive space where analysts will see and interact with massive amounts of information, more quickly, with more comprehension.
This is known as "nSpace" and is the combination of the mufti-dimensional linked views found in TRIST, "The Rapid Information Scanning Tool" [Jonker et al, 2005], see figure

2. The term shoebox (e.g. container) is used to describe the information that has been collected by an analyst. This information comes from several sources and incorporates various types of media. The process of building and organizing a shoebox is known as evidence marshalling. This task is difficult due to the massive amounts of information Page 4 of 36 involved, making it difficult to understand the big picture by looking at reams of evidence. Many analysts perform their evidence marshalling in Microsoft Word, which is not intended for such a purpose. The analyst is forced to rely heavily on personal memory to connect the concepts embedded in separate information pieces of the information set considered.
Currently, many analysts end up printing their shoebox so they can spread out the pages and mark them up with highlighter ink. This method is not only impractical for large amounts of information, but also completely avoids various existing analytical methods (e.g. inference networks, ACH, models, etc.). Considering the relatively short term tasks studied and the difficulty of building say inference networks or other analytical charts in Word, it is not particularly surprising. In the end, this seems to indicate that analysis (hypothesizing, fleshing out hypotheses with evidence proving and refuting them, corroborating, evaluating the diagnostic value of the evidence, and assessing which hypotheses are most likely, etc.) is something that happens currently without the aid of GUI tools 12.
Many researchers [Heuer, 1999] have warned about the possible consequences of doing analysis primarily in one's head. The analyst is more prone to human cognitive biases, and causing difficulty in collaborating and communicating about the analytical process. The thinking process is not made explicit, making it harder to explain how conclusions were reached than if formal analytical methods were used. An analysis system such as the tool 12 offers a flexible organization support for thoughts and information relevant to the analysis, encourage making analytical thinking explicit, facilitate the application of various analytical methods, scale to large amount of information and thinking, make it easier to keep more of that information and thinking available to working memory at once, and increase analytical performance by reducing friction points and overhead that steal time from analysis. Such a system should make a significant improvement by reducing the impacts of human cognitive biases, increasing Page 5 of 36 the volume and complexity of information used in the analysis, and facilitating communication and collaboration.
Referring to Figure 1, a visualization data processing system 100 includes a visualization tool 12 for processing a collection of information sources 14 (documents, webpages, images, text, ect...) for extracting information excerpts to present as a representation 18 of the information on a visual interface 202. The information sources 14 can be combined with selected user workflows 16 the tool 12, along with a generated information excerpt region (e.g. sandbox) to generate an interactive visual representation 18 on the visual interface (VI) 202 of the information excerpts. Management of the information sources 14 and workflows 16 are driven by user events 109 of a user (not shown) via a user interface 108 (see Figure 18) during interaction with the visual representation 18.
Referring to Figure 18, a data processing system 100 has the user interface for interacting with the tool 12, the user interface 108 being connected to a memory 102 via a BUS 106. The interface 108 is coupled to a processor 104 via the BUS
106, to interact with user events 109 to monitor or otherwise instruct the operation of the tool 12 via an operating system 110. The user interface 108 can include one or more user input devices such as but not limited to a QWERTY keyboard, a keypad, a trackwheel, a stylus, a mouse, and a microphone. The visual interface 202 is considered the user output device, such as but not limited to a computer screen display. If the screen is touch sensitive, then the display can also be used as the user input device as controlled by the processor 104. Further, it is recognized that the data processing system 100 can include a computer readable storage medium 46 coupled to the processor 104 for providing instructions to the processor 104 and/or the tool 12. The computer readable medium 46 can include hardware and/or software such as, by way of example only, magnetic disks, magnetic tape, optically readable medium such as CD/DVD ROMS, and memory cards.
In each case, the computer readable medium 46 may take the form of a small disk, floppy diskette, cassette, hard disk drive, solid-state memory card, or RAM provided in the Page 6 of 36 memory 102. It should be noted that the above listed example computer readable mediums 46 can be used either alone or in combination.
Referring again to Figure 18, the tool 12 interacts via link 116 with a VI
manager 112 (also known as a visualization renderer) of the system 100 for presenting the visual representation 18 on the visual interface 202. The tool 12 also interacts via link 118 with a data manager 114 of the system 100 to coordinate management of the information sources 14 and associated information excerpts from data files or tables 122 of the memory 102. It is recognized that the sources 14 and related information excerpts could be stored in the same or separate tables 122, as desired. The data manager 114 can receive requests for storing, retrieving, amending, or creating the sources 14 and excerpts via the tool 12 and/or directly via link 120 from the VI manager 112, as driven by the user events 109 and/or independent operation of the tool 12. The data manager manages the sources 14 and excerpts via link 123 with the tables 122 It is recognized that the tables 122 could also contain predefined or otherwise stored information excerpts already generated by the tool 12, as further described below. Accordingly, the tool 12 and managers 112, 114 coordinate the processing of sources 14, excerpts, retrieval and implementation of workflows 16 with user events 109, with respect to the content of the screen representation 18 displayed on the visual interface 202.
1 Introduction 1.1 nSpace Goals To support NIMD objectives, Oculus is inventing new interactive, information visualization techniques that are tightly coupled with massive data, software agents and the analyst's exploration task. Our main theme is that a break-through in finding novel intelligence is possible if all the components are combined together in a system of systems. We are working on a fluid, flexible medium of analysis and expression. We are expecting to achieve a measurable, order of magnitude, increase in productivity. Initial experiments with TRIST ("The Rapid Information Scanning Tool") have shown that analyst work product quality can be increased, in half the time, while reading double the Page 7 of 36 documents. We are now working at the information and cognition level, exploring new cognitive interactions and new information visualizations, and expect further productivity increases as we move to information-focused analysis.
1.2 nSpace: TRIST and the Sandbox We call this space "nSpace". As shown in Figure 2, nSpace is the combination of the mufti-dimensional linked views of information sources 14 found in TRIST
and the cognitive exploration and development mechanisms in the Sandbox, used to represent information excerpts and their interconnections to one another and the respective original sources 14. TRIST is an information retrieval system that supports searching, rapid scanning over thousands of search results of information sources 14 (such as but not limited to documents - both text and image - and/or information links - e.g. a webpage) in one display, browsing and extraction of information excerpts from the sources into the Sandbox. TRIST can include aids to query planning, a linked mufti-dimensional space for result characterization and correlation, and an integrated document/link viewer.
The Sandbox is a flexible and expressive thinking environment. It is a space literally focused on 'Human Interaction with Information', such that local collections of information excerpts with user attached links, associations, hypotheses, comments, annotations and other inter- and intra-excerpt connections are represented visually in the representation 18. For example, the analysis content of the information excerpts produced through use of the sandbox (e.g. assumptions, inferences, evidence tied to hypotheses, explanations) is such that the analysis content is visually linked to the information excerpts and may also be visually linked back to the information sources 14 of the respective information excerpts displayed in the sandbox of the representation 18.
Manipulation and organization of information excerpts is direct and very tactile. Intuitive iconic representations of entities, concepts and relations as information excerpts constitute a common, shared visual vocabulary. This is a visual thinking space for considering evidence in multiple frames, designed for user driven tactics in evidence marshalling and sense making like "people-places-organizations-events"
cognition or "put-this-there" cognition.
Page 8 of 36 1.3 Overview of Paper - Sandbox Progress We begin with a brief review of relevant observations from our earlier analyst cognitive task analysis (CTA). This review leads to performance objectives for the Sandbox which are explicit productivity impacts that need to be achieved. To complement the CTA, an analysis of Glass Box data for several analysts and tasks has recently been completed. This has yielded a first pass at a baseline of analysis performance, and the "cost structures" for that performance, that will help gauge the impact of the Sandbox. When Sandbox experimental results have been compiled, they can be compared with this baseline.
Four areas of technical progress have been achieved with new Sandbox capabilities. First, a variety of flexible means of expression allow "put-this-there"
cognition with minimum friction of interaction. The objective is to support speed-of thought analysis and visible thinking.
Second, analytical templates have been implemented using, for example, Fair-Isaac lightly-supervised document clustering. An analytical method is the use of process models/templates or organizational models. These models are templates that support analytical best practices and allow analytical knowledge and methods to be shared and re-used for various information excerpt sets contained in the sandbox.
Third, experimentation with gestures has begun for the fluid expression of thought, emphasis and actions. The "powerful finger" emphasizes key points.
Editor gestures Y ~ are used to create and remove Sandbox space. Evidence moved through gates with the "dragging hand" is automatically tagged as supporting or refuting assertions.
The fourth area of progress has been in scalability. Thumbnail navigation for the Sandbox allows quick directed movement within large Sandboxes. Thumbnail level of detail for groups minimizes a group while still preserving the semantic layout of information within the group. To improve performance, the Sandbox has been Page 9 of 36 completely re-implemented to allow for hardware acceleration as well as both 2D and 3D
forms of expression in one single workspace.
To illustrate and test Sandbox capabilities, a synthetic analysis task for nanotechnology assessment has been created. We conclude with a plan for an experiment with the Sandbox at NIST. The experiment will allow the development and testing of innovative metrics for cognitive performance as well as generate informal and formal feedback from users on Sandbox capabilities.
2 Cognitive Task Analysis for the Sandbox 2.1 Introduction Three types of CTA studies have been completed to date including structured interviews earlier in the program and then, more recently, Glass Box activity analysis and content analysis. The observations from all three CTA studies were used to refine the performance objectives for the Sandbox. They also provided an opportunity to consider potential task metrics. In addition, this type of analysis helps establish a baseline for subsequent experiments. A NIST experiment is planned at the beginning of 2005 which will focus on the new nSpace-Sandbox capabilities.
The following is a brief recap of observations from the earlier CTA analysis.
This is followed by more detailed observations derived programmatically as well as the higher level content analysis.
2.2 Speaking with Analysts Observations In the earlier broader CTA, structured interviews were conducted with fourteen analysts from several organizations who work with a variety of sources and on a range of short/long term, narrowlbroad focus subjects. From these interviews, it was clear that working with a local "knowledge base" or sandbox of pertinent information including references and annotations is a common daily practice. The following are some illustrative interview excerpts from the study.
Page 10 of 36 "Research is gathering bits. Analysis is what it might add up to." It can take two weeks to two months to bring together disparate raw data into a set of connections and patterns of activity, people and locations. The patterns can continue to be refined and developed.
I put the information in piles on the desk sorted by topics and arranged in time.
The topics are what I will write about. I make notes on anything and everything.
"Organization is half the battle. I use ppt to organize my thoughts. e.g. Five hypotheses and 100 pieces of evidence. Indexing, writing notes and reasoning.
I can spend 3 or 4 days compiling results."
The basic concept of analysis includes rolling up, drilling down and pivoting.
You rollup from lots of detail, clutter, maps. Show results in the same space.
You drill down to the needle, or by clicking through. Filtering is important: by geography, time, hierarchy. Pivoting is picking a thing that becomes the focus. Find out more on this element. Spawn a new search.
I've been eight years on this one region. I have a 300 page Word document that I
use as a log. It's very effective. I color code. Emergent issues are grey. XXX
items are yellow. Current questions are blue. I use the log to go back and search.
2.3 Glass Box Data Analysis Observations With a focus on the "analysis" task component, Glass Box data was processed to generate observations about the "costs" incurred by analysts, the "micro-products"
produced as well as friction points. Costs include, for example, time spent by tool and by activity. Micro-products include things like number of references, pastes and documents saved as well as number of characters/words typed. These costs and products can be observed programmatically in the data. Glass Box Release 211 was used to extract data for five and four tasks done respectively by analysts NNN and MMM. Additional statistics were also compiled for four analysts for 12 tasks.
Page 11 of 36 Figure 3 shows, 20 ~
0 .
-' .~~
~
~
s~ti;
~a ~

~ ~
for all analysts '8 ~.~ ~ ~~
and all ~ ~ k ~ r ~~ ;.~
~ i, ~
, ;

, ~
~ a~
~
'.
~r, 160 ;' a g ,~ ' . ~
~ ' fq w:

the average time ~ 1 a ~
tasks o ~~ ~
, , ~ , ~ ;.

, ,~ g ~

= 12O i ~ d;~~ a~'E, B
spent (i.e. cost) ~ ,o ~~, ~~"' on the o ~x~

, fa S ~
~ 34~ ~' ~ "
' m 8 0 t~E

~, Glass Box analysis~ ~~ Y~-, ~~B
~

D 6 ~~?r a, ~a.~ a r ~' f .m ~ s a ' ~
~
~'~

. ~ . ,~
This is a baseline ~ ~;, t , k t, u as ' s.

of how time is o o i spent _ a ~_;~ Iv~ g'Io J I ~
L

now using current ~ ~ ~ ~ I a o tools v ~ I
. ~
~
I

i ~
i w ~ ~
..

and rocesses. I
h i ~

l IE Word Other I

Roughly equal Figure 3. Cost Profile - Time. Average Task.
times are spent reviewing/analyzing contents of information sources through mediums/tools such as but not limited to Internet Explorer (IE) doing information retrieval, in Word doing both reference saving and analysis, and in the File Explorer organizing and navigating directories and files. (Any desktop or "Start" menu time is also counted as Explorer time.) The "Others" category includes information source analyzing/reviewing using such as Ref Manager, Adobe, Netscape and the Glass Box itself. At 38 hours per task on average, it is the most significant use of time. In a more complete analysis, time spent in each of these applications should be identified and allocated to information retrieval (IR), Analysis or other tasks.
t'lguIC 4. HIl3ly'Sl 1V1VIV - IlVlilJ lIl 1t. Qflu VV UIU 1'lKulG ~. t~maty~' mnvmn - muum m m: amu vv mu for Several Tasks. for Several Tasks.
Page 12 of 36 Figures 4 and 5 examine variability of time spent by analyst and by task. Task durations are noted. Figures 6 and 7 look at the same data but normalized. It is interesting to note that for shorter or longer duration tasks, the overall proportion of tool usage remains very similar.
figure 6. Analyst NNN rigure /. Analyst MMM
Time Distribution per Application. Time Distribution per Application.
Additional detail is possible to extract from the Glass Box data. An attempt was made to organize this detail into "costs" and "products" produced. Some results are shown in Tables I and 2. These results are more "mechanical" in the sense they are observations about actions done without considering quality. In Table 1, It is interesting to observe the extremely large number of context/medium switches. These are switches from one information medium to another. Switching from one file to another is not included in the count but represents even more context switching. It is also interesting to note the differences in cut-copy-pasting. Once information is pasted in to a document, it seems to stay where it was first placed. MS Word is a container of information but is not used to work with the information.
Table 2 shows that not much writing is done in the Log. A large number of unique documents and sources are found. However, only a few snippets make it into the actual report.
Scan Event 108 Snippets in Report from WordF J 5 Page 13 of3G

Scan Event 108 Sni 5 ets in Re ort from Word Read Event 9,915 Sni 12 ets in Re ort from Web Read Cache Event 1,168 # Uni 193 ue Source Documents ~.~. , a ~ ..
5 # Uni 65 ue Publishers Documents Referenced 20 Re ort Documents 19 ' Cut-Copy-Paste i_n 6 Total 8.2 a Doc. - All Documents Cut-Co y-Paste, across101 Re ort 3.6 Docs.

Context Switch 2,802 Lo 0.8 Table 1. C'.ost Profile - Mechanics. Table 2. Production Profile - Mechanics.
Average for all 'tasks. Average for all Tasks.
2.4 Analysis of Content Objects Observations of mechanical level activities help provide insight and a baseline profile of how work is done currently. Of additional interest, are higher level observations about "content" level objects. Observations, for example, about how many hypotheses were made, or how many elements of evidence were marshaled for assertions, or how many trains of thought were noted. These kinds of observations are possible but require human coding of content. Higher level observations about process and the content were made for Word documents for analyst NNN for tasks three and four.
The observations were extracted by watching the screenshot videos, from beginning to end. It was possible to observe where content objects came from, if they were typed or pasted in, where they were placed and how they were used.
Judgments were applied to categorize document content into assumptions, evidence, reference and other content objects using a framework of guidelines shown in Table 3. Objects were categorized based on where they best fit. They sometimes fit into multiple locations. For example, a pasted section of text that incorporates both a citation and a paragraph of information would allow for a paste activity in both the Reference object and Reference Short (<paragraph) object as the object contained both of the necessary components.
A distinction was made between a new object that was pasted or typed into a document and an object that was edited or had words changed in it. A new object always Page 14 of 36 counted as an activity in one of the categories while an edit of an object was not counted if it did not change the meaning or the reasoning of the object. The reason for this was to count objects once, not to count a change in the object unless there was a change to the meaning or the substance.
Type of Coan~t abject ~ ~u~ arid ~r~tp~on -_m~.'T_~.
._._~~___mm_ ._ ~._..
.__ ~m _____ _ "To Do" Ana_~lyst to do list, reminder, work plan_.~
Notes T~_,__ __..__.._ __._ - _, n Retrieval _f,~_~s;
I~~_ rm ~ ~~~~~
~T y -_.
~
~ ~ m _ a source Reference Citation bibliographic, url to ... . Whole or partial document. Coded short (<paragraph), Ref. Documentmedium (<page) _.__. ___..and lon~in length.,_' _._.__. ____ _-- _ __..._ . . __ __..._._. ......____. _ _'~

ObservationQuote, short, looks factual, based on a reference.

Abstract Summary, condensed~form of main o~'mts_-_ ___. _. , rv _.. ..
. _. ..__. _.... __._ __ .__ ~
Attstl~S
~~ecls _. _ ____ .__. _~.__ ~ ~_ -_. ' ~.
~. ~ _ ~ ~ ~ ~
__.___ Assumption An assumption had to literally be an assumption, explicitly identified as such by the analyst.

Hypothesis Supposition, inference, guess, assertion, hypothesis, judgement to which the analyst would bind emdence supporting or undermining it.

Evidence ; A fact, reference or an observation that the analyst tied to a supposition or an hypothesis to s~~ort or refute it. _ ..
_ ~ __ --Explanation~ Rationale, justification, explanation that binds the evidence to the hypothesis.

Analytical e.g. Analysis of competing hypotheses (ACH), models (e.g. process, org Method ' chart, etc.) or inference networks.

Other Passages~ Written text that did not fall into the other categories. Notes as short of Text <paragraph), medium (< page) and long m length.........

Table 3. Document Content Objects and Descriptions.
Observed content objects and related actions were noted in an activity log.
The type of content object and associated action (paste, write, delete, move, highlight) was logged in time sequence order. Time stamps were noted at document creation and saving.
Figures 8 and 9 show the activity log produced for Analyst NNN, Task 3. Along the left is a list of the object categories from Table 3. The top categories are IR
(Information Retrieval) objects, and the bottom are Analysis objects.
Activities were logged in sequence from left to right.
Page 15 of 36 r~ f ,. & 4Y is2 ., j = a j .. c. ':;

3 r I1 <.~ ~~a i ;s s ~ ,~ w ~ aw °' ~ " ~ ~ ~ m ~ s~ ~ wm "~"" ""r"" ~ ~
Aw~~ sew ' 1t. v A~~ v W ~ v~.t m ~ !m ~ r ~ m~ ~ ~ ~ ~
. v ~ ~ w v r CKwmf crw w ~r k»."
Figure 8. Content Activity Over Time Figure 9. Content Activity Over Time in Notes.doc (the Shoebox) in Bulletin.doc (the Report) Created: July 15 10:23 am Created: July 15 10:21 am Last Save: July 15 11:34 am Last Save: July 16 7:16 pm Object activities are color coded by type of activity as shown in the legend. Figures 8 and 9 are a trace of observed analytical activity for these documents.
Figures 8 and 9 show the overall workflow had linear and non-linear components. The analyst started with some analysis, followed by filling a shoebox and then wrote the report. The multiple context/medium switches recorded indicates that the analysts bounced back and forth between applications, but mainly between IE and Word while creating the shoebox and then mainly between the shoebox and the report.
During the report writing, the analyst only to referred to the shoebox, and very rarely added to it. The process of building shoeboxes in Word appears to have been linear.
Most of the information that was saved was pasted sequentially, in the order found, usually in a single Word document. Very minimal rearranging, highlighting or editing was done to the shoeboxes.
There seems to be no observable use of analytical methods such as ACH, models, or inference networks. The main analytical method being used seems to be evidence marshalling. Tying together of the pieces or entire contents of information sources 14 appears to have been done in the analyst's mind and/or as part of the report creation, or Page 16 of 36 was offline and unobserved. In fact, the analyst printed at least part of the shoebox to more easily refer to it while writing the report. The number of content objects were counted for each of the documents produced by Analyst NNN for Tasks 3 and 4.
Documents so processed were either used for a shoebox or for the final report.
Figures and 11 show the counts of content produced.
As well as a count of objects, we can also consider how much time was taken to produce that number of objects. It is possible to observe elapsed time on the tasks and the documents. Task 3 occurred July 15 and 16.
~ The notes document was started at 10:23 am on July 15. The last save was at 11:34 arn. This is an elapsed time of 71 minutes.
~ The r~~rt document was effectively started on July 16t" with a first activity at 5:04 pm and the last save was July 16 at 7:16 pm. This is an estimated elapsed time of 132 minutes.
Content Produced for EBOLA Bulletin To Do Rehsrence Short doc Med doc Long doc Fact based on Ref o, Abstract or Summary Assumption Hypothesis or Inf~srence ~o ShoeBo Evidence 1 Repo Jrt ° Explanation U
ACH
DAodels Inference Network Short text fragment Med text fragment Long text fragment Figure 10. Content Produced by A6 for Ebola Bulletin (Task 3).
Page 17 of3G
Count Content Produced for EBOLA Analysis To Do Reference Short doc Med doc Long doc Fact based on Ref o, Abstract or Summary Assumption ~I
Hypothesis or Inference o ShoeBo~, Evidence ~ Repo Jrt V Explanation ACH
Models Inference Ncawork Short text fragment Med text fragment Long text fragment Figure 11. Content Produced by A6 for Ebola Analysis (Task 4).
The Task 4 Ebola Analysis was a follow-on to the Task 3 Ebola Bulletin. The Task 3 notes document became the Task 4 notes document.
~ The notes document was opened Aug 9 at 12:37 pm, saved under another name Aug 10 at 1 I :14 am, with a final save at 11:21 am. The notes document (i.e.
the log) has an estimated elapsed time of 7 minutes.
~ The resort document was created Aug 9 at 12:33 pm with a last activity Aug 9 at 12:46prn,the a first activity Aug 10 at 10:02 am with a final save Aug 10 at 7:43 pm. The report document has an elapsed time of about 1 day.
Looking at Figures 10 and 11, the quantity of events where references were produced seems large relative to the other kinds of content objects.
References also appear to have been produced relatively quickly compared to the elapsed time spent in the report document. But it should be noted the elapsed times are just for the MS Word activities and do not include IE activities. It is interesting to see that hypothesis and evidence activities occurred in both shoebox and report documents.
Some ratios are possible to consider as potential metrics. For example, in terms of the proportion of information collected that ended up being used in the analysis, about one-third of the; shoebox seems to end up in the final report. Most of the analysis content Page 18 of 3G
Count of the information sources 14 produced (e.g. assumptions, inferences, evidence tied to hypotheses, explanations) appears almost exclusively in the final report. Any analysis content objects produced were used in the final report.
2.5 Conclusions These observations of Glass Box analysis work should be considered inconclusive and a trial run. Only two analysts and two tasks were included because only Glass Box release 211 was available. Now that GB Release 212 has been released, this kind of task analysis can be performed for a more representative number of analysts and tasks. GB
Release 212 contains nine tasks for analyst 4, eight tasks for analyst 5.
Another reason to consider these observations to be inconclusive and preliminary is that the GB
data task data appears to be inconsistent with task assignments and task products as published on Twiki. This needs to be investigated further. Nevertheless, a number of conclusions are possible.
Refernng to Figure 19, keeping track of sources 14 and their related queries is time consuming and thus costly. It does not seems to be the best use of the analyst core skills and time. nSpace could reduce this overhead considerably, since TRIST
query planning pane keeps track of the queries and their results. Any relevant information excerpts saved in the Sandbox keeps its link 200 to the document source 14 and queries it came from automatically. Accordingly, the retrieval history, as well as links of other "at first glance unrelated" excerpts to the shared sources 14 , of the information excerpts is associated with the respective excerpts for use by the user of the sandbox.
For example, refernng to Figure 19, the visual links 200 would show that the excerpt A came from query 202, while excerpt B came from a resultant document 204 of the same query 202, therefore showing the connectiveness of the two excerpts A,B other than by the user added analysis content 206 (e.g. annotations - etc. ...). It should be noted that visual elements of the link 200 would be generated by the tool 12 and shown to the user in the representation 18 for the sandbox, in order to represent the link of the excerpts A,B to the information sources (query 202 and document 204). The links 200 can be communicated visually to the user via the representation 18 using for example text, images, icons, or a Page 19 of 36 combination thereof. It is also recognised that the link 200 details could be communicated via audio capabilities of the UI 108 to the user (e.g. speaker) either when prompted by the user or on a periodic basis (for example when the particular excerpt is selected by the user). It is also recognised that the visual communication of the links 200 to the user can be "always on" or can be done on a selective basis by the system 100 with or without user interaction.
The large number of potential context switches between tools/mediums constitute another kind of friction or overhead observed that could be significantly reduced in nSpace since IR and analysis are performed in an integrated environment. Using nSpace should also reveal if the apparent linearity of parts of the workflow is artificially imposed by the current tools.
Based on the observed process of building large shoeboxes in Word, (i.e.
pasting sequentially usually in a single Word document all bits that appear relevant to the whole task), one could infer that evidence marshalling is particularly difficult. It must be hard to get the big picture by looking at pages and pages of text. The analyst probably relies heavily on memory to connect the dots. Not unrelated to this is the lack of observable use of various analytical methods (e.g. inference networks, ACH, models, etc.).
Considering the relative short term tasks and the difficulty of building say inference networks or any kind of analytical charts in Word, it is not particularly surprising. In the end, this seems to indicate that analysis content (hypothesizing, fleshing out hypotheses with evidence proving and refuting them, corroborating, evaluating the diagnosticity of the evidence, and assessing which hypotheses are most likely, etc.) is something that happens mostly in the analyst's head when not using the nSpace enabled tool 12. Many researchers [Heuer] have warned about the possible consequences of doing analysis primarily in one's head. The analyst is more prone to human cognitive biases and it may not be as easy to collaborate and to communicate about the analytical process and how the conclusions were reached than if analytical methods were used and so made the thinking explicit.
Page 20 of 36 The well known analysis "bathtub" curve [Rose, 1996], showing that most of the analyst time is spent in IR and Report Creation and almost relatively no time doing Analysis, was replicated.
3 Sandbox Performance Objectives The following discusses performance objectives for the Sandbox. These objectives are articulated for technology research purposes in order to guide invention and development. Performance objectives are also important for planning experiments and for developing performance metrics. Objectives represent hypotheses that can be tested.
The first objective for the Sandbox is to be a flexible place for visual thinking and working with evidence during analysis. Analysts need to be able to quickly and easily place, arrange, re-arrange, group, emphasize, highlight and compare information sets and individual information excerpts. Analysts should be able brainstorm in the Sandbox, generate hypotheses, marshal evidence, collaborate and report. It must be flexible enough to support many types of analysts and analytical styles. Making information and links between the information excerpts visible in the Sandbox can allow more information and more complexity to be managed in less time.
Another objective is for the Sandbox to support best analytical practices without imposing additional training requirements or cognitive and procedural strain.
In the Sandbox, analytical "blue prints" facilitate the application of best practices. Instantly flipping from one template to another, from one point-of view to another, can enhance comprehension, promote discovery and help counter biases. There are a wide variety of best analytical practices including organize-conceptualize-hypothesize, the application of scientific method and competing ideas as well as the clear and explicit representation of reasoning. Additional analytical methods now in use include [Heuer, 1999], [Bodnar, 2003], [Hughes, 2003]:
~ Generate hypotheses before searching for evidence;
Page 21 of 36 ~ Multi-dimensional analysis of process, organization, locations in time;
~ Inference networks;
~ Analysis of competing hypotheses;
~ Use multiple analytic strategies simultaneously for hypothesis generation and evidence marshalling;
~ The seven methods of evidence marshalling;
~ Situational logic. Constructing scenarios from evidence; and ~ Toulmin argumentation.
An objective for the Sandbox is to provide a measurable increase in performance.
Using the Sandbox, more analysis, higher quality analysis can be performed in less time.
As a first step, the aim will be to support the same quality of analysis and reporting with less time. This is a similar approach to what was shown with the TRIST
experiment where the same quality of information retrieval was provided in less time. The longer term objective is to provide a productivity increase over the whole analytical workflow with fewer steps, higher quality results, in less time.
Supporting a larger volume and complexity of analysis is another important objective. The Sandbox needs to be scaleable to massive data. Analysts already work with large amounts of complex data contained in the information sets and individual excerpts, and need to be able to work with even larger amounts. More evidence needs to be considered, more factors, more alternatives, all in a more collaborative setting.
Sandbox performance objectives also include being able to support multiple simultaneous tasks as well as providing an initial capability for collaboration. Because analysis is not sequential, and because analysts are engaged in multiple simultaneous tasks, it must be easy to switch contexts and easy to pickup contexts in the Sandbox.
Collaboration is a creative working mode and being able to operate the Sandbox with multiple analysts at the same time would allow shared perspectives and shared construction of lines of thought.
Page 22 of 36

4 Sandbox Prototype Capabilities 4.1 Put-This-There, Visible, Flexible Cognition Visualisation tool 12 The Sandbox is a space that supports visual thinking, providing alternatives to paper or Word for thinking activities such as hypothesizing, fleshing out hypotheses with evidence, corroborating, grouping, annotating and prioritizing. Explicit thinking helps ensure more rigorous thinking and thus can reduce the impact of some cognitive biases.
Visual thinking can increase an analyst's cognitive span, and also make it easier to communicate and facilitate collaboration. Seeing what is being considered will also show what is not being considered.
Direct manipulation and annotation are used to build and express meaning.
Analysts can write anywhere, group anything, place anything anywhere, add and edit entities (e.g. people, organizations, places), add and edit snippets and whole reference documents, images can be used and connections can be made on the visual interface (GUI) by direct interaction of the developer with the tool 12. Thinking can happen quickly and must not be frustrated by a burdensome medium difficult to operate. A
minimum of buttons and no dialog boxes are used to create, edit, place, arrange, re-arrange, group, emphasize, highlight and compare. The Sandbox allows the analyst to spend their time thinking, not interacting with the application.
The analyst can save any relevant information, including documents, snippets, images, tables, etc. simply by dragging them into the Sandbox visual area of the GUI, see Figures 2 and 19. References can be automatically saved. Time consuming manual tracking of sources can be minimised. Select the information in the Sandbox and its source can be highlighted in the TRIST application portion of the GUI, making it easier to check context and scan all the metadata associated with the source. This approach can also reduce the context switches that usually take place when collecting information and later going back its source.
Page 23 of 36 Referring to Figure 20, the interactive information analysis visualization tool 12 provides a space (e.g. the sandbox) in the visual representation 18 for visual thinking, for thinking activities such as hypothesizing, fleshing out hypotheses with evidence, corroborating, grouping, annotating and prioritizing. Explicit thinking can help ensure more rigorous thinking and thus reduce the impact of some cognitive biases.
Visual thinking can increase an analysts cognitive span, and can also make it easier to communicate and facilitate collaboration. Seeing what is being considered will also show what is not being considered.
Direct manipulation and annotation are used to build and express meaning.
Analysts can write anywhere, group anything, place anything anywhere, add and edit entities (e.g. people, organizations, places), add and edit information excerpts (e.g.
snippets and whole reference documents, images) can be used and connections can be made. Thinking happens quickly and must not be frustrated by a burdensome medium that requires too much cognitive overhead to operate the tool 12. The Sandbox lets the analyst quickly, and with a minimum of buttonology and little to no dialog boxes, create, edit, place, arrange, re-arrange, group, emphasize, highlight and compare the contained and displayed information excerpts and their connections to one another and to the information sources 14. The Sandbox is a thinking-centric place.
The analyst (user of the tool 12) can save any relevant information excerpt, including documents, snippets, images, tables, etc. simply by dragging them from the other areas of the visual representation 18 (e.g. TRIST - query results and/or documents) )into the Sandbox. References/links are automatically associated by an extraction module and therefore visually updated through a visualization renderer module to the representation 18, a snapshot of which can be saved in a storage of the system 100 if desired. Accordingly, a visual representation of the links of the excerpts to their original queries/document sources is done. Little to no time consuming manual tracking of sources 14 is required. The user selects the information excerpts) in the Sandbox and its source will be highlighted in to the user (for example in TRIST), making it easy to check Page 24 of 36 context and scan all the metadata associated with the original source 14 of the excerpt(s).
This approach can also eliminate all the context switches that usually take place when collecting information and later going back its source.
The Sandbox is a very tactile space with practically no visible interface, just the information and the thoughts. Direct manipulation increases the analyst's productivity by reducing the amount of work needed to perform any action. Various automatic layout mechanisms (e.g. lists, grids, circular arrangements) for the information excerpts when contained in the sandbox area are also available when there is no alternative, more meaningful layout apparent to the user. One of the strengths of nSpace is the flexibility it gives the analyst in pursuing the discovery of useful knowledge (in TRIST) and in following multiples lines of thoughts or trying multiple ways to marshal the evidence in the Sandbox, all in a very fluid manner.
The tool 12 also has a number of other modules, such as but not limited to a display manager for coordinating the interaction of associated modules with the user inputs 109 for rendering the representation 18 of the analytical content, links to sources 14, and the excerpts themselves to the sandbox area of the display 202. The tool 12 also has a connection/association module for guiding or otherwise enabling the user of the tool 12 to input analytical content for simultaneous display (selected or otherwise) with the excerpts of the sandbox area - through user events 109, a template module for providing a structured layout of the excerpts in the sandbox area, an association module for binding two or more excerpts and their related analytical content, a workflow module for coordinating a selected workflow of the tool user, and an emphasis module for altering the display of the excerpt content and form.
4.2 Analytical Templates-Models Module Process models (i.e. a manufacturing process) provide a structured framework to think about subjects of interest and events. A process model helps explain what is happening, why it is happening and what can be done to change it. Many analysts use and invent methods, models or templates to analyze situations. Models help share and harness Page 25 of 36 analytical knowledge and experience. In addition, different models help the analyst exercise a different point-of view to help think out of a standard mindset or pre-disposition. When an analyst has a process model in mind what they don't know jumps out at them. Gaps are evident.
The Sandbox allows the creation and use of analytical templates to support various concurrent ways of thinking about a problem by minimizing any additional cognitive strain. Sandbox helps to lower the cost of looking at evidence from multiple perspectives, trying many different models, which would presumably counter rigid mindsets biases while highlighting the strengths and deficiencies of all models considered.
The Sandbox can provide automatic evidence layout using "templates" of analytical frameworks. Normally, evidence (e.g. snippets, observations, documents) is organized in a specific way to support the analyst's put-this-there cognition, as shown below in (1.). Next the analyst can open a new template of, for example, an organization chart or a process model - see below in (2.), and the existing evidence is automatically placed in the new layout. Placement of evidence reflects similarity or closeness of fit as shown in Figure 12. Evidence that does not fit is placed outside and might prompt edits to the template/model. Placement inside the template could also reflect certainty of fit (e.g.
center of node vs. on the edge).
Document similarity using context vector clustering from Fair Isaac can be used to implement the templates [Caid, 1997]. The clustering uses lightly supervised methods and can be trained with keywords and sample documents. Subsequent "learning"
and improvement is possible with "experience". Document or snippet similarity is scored and the item is assigned to the closest cluster.
Page 26 of 36 With many such layouts placed side-by-side, the analyst can draw conclusions such as which model best fits the data. Using the linked selection mechanism across multiple models, the diagnosticity of the evidence can also be revealed.
Process models of, for instance, a drug smuggling process, provide a structured framework to think about subjects of interest and events. A process model helps analysts understand what is happening, why it is happening and what can be done about it.
Models provide significant benefits. They help share and harness analytical knowledge and experience. In addition, different models can help the analyst exercise a different point-of view to help think out of their standard box or pre-disposition. Finally, when an analyst has a process model in mind, or in the Sandbox, what they don't know jumps out at them. Gaps are evident.
Many analysts use and invent methods, models or templates to analyze ~ ~ ~ Lost merchandise certain type a , of tasks. These 'Analytical Blue Prints' capture their creator's knowledge,' o ~~~~ ~, ~y and most denies Fault analysts would benefit if this r~ c knowledge could be easily shared and re-used.~
One -to ~
Q

caveat of such templates is that they can 1. Exploration Workspace.
become a lens sometimes resulting~,_ ~~ ..a~_ ~_._v~ .._~.~
in -._~_ ~.._ dismissal of things that do not fit the model. ;.

_.
To address this, we create and Receme use analytical order templates in the Sandbox area a ~
to support -~

, ,~,f~
.e various concurrent ways of thinkin~abouti __ a Pad<age problem without adding any additional cognitive strain. We want to sh'~
lower the cost ~ecei've of looking at evidence from 2. Blank Manufacturing Process multiple Template.

perspectives, trying many different models, which would presumably counter rigid Page 27 of 36 mindsets biases while highlighting the strengths and deficiencies of all models j ' ~~ ~ ~=;
Receive considered. ads- ° ~"
,. ~ _w Manufachre z..___ _~._..
The Sandbox provides automatic evidence Package '~
layout using "templates" of analytical frameworks. Normally, evidence (e.g. _ 4 ~ -___ ~ __ snippets, observations, documents) is 3. Manufacturing Process Filled with Items from Exploration Space.
Figure 12. Using Templates.
tagged (e.g. through metadata) for selection into one or more template categories (predefined categaories) but is organized first in a specific way in the Sandbox area to support the Analyst's put-this-there cognition (i.e. analyst determined manual layout), as shown in Stepl of Figure 12. These tagged (i.e. associated with the excerpt) category can already be implicit from the source 14 or can be attached by the analyst through the extraction module when importing the excerpts into the sandbox area of the representation 18 (e.g. drag and drop). Next the analyst can open/select a new template of, for example, an organization chart or a process model, Step 2, and the existing evidence is automatically placed in the new layout according to the excerpt categories.
Placement of evidence reflects similarity or closeness of fit as shown in Step 3. Evidence that does not fit is placed outside and might prompt edits to the template/model.
Placement inside the template could also reflects certainty of fit (e.g.
center of node vs.
on the edge).
Document similarity using context vector clustering from Fair Isaac is used to implement the templates. Association grounded semantics, with probability distributions over context, builds and assesses similarity of meaning and provides improved linguistic synonymy and polysemy performance. In addition, the clustering uses lightly supervised methods and can be trained with keywords and sample documents. Subsequent "learning" and improvement is possible with "experience". Document similarity is scored and the document is assigned to the closest cluster if it scores at least the threshold Page 28 of 36 for that cluster. If not, the document is assigned to a "none of the above"
category.
Thresholds can be set interactively.
With many such layouts placed side-by-side in the representation 18, the analyst can draw conclusions such as which model best fits the data. Using the linked selection mechanism between the Sandbox and TRIST, the diagnosticity of the evidence can also be revealed.
4.3 Predefined Visual Association Module - Dragging-Hand Gesture for Evidence Gates The explicit representation of assertions and how evidence supports or refutes each assertion is another mechanism that has been implemented to support best practices in analytical methods. The assertions make explicit the points the analyst is trying to prove/disprove. In the Sandbox, an assertion is represented as a group that has visually represented 'Support and Refute Gates'. The act of binding evidence to an assertion is done fluidly by dragging the evidence to the assertion group through the appropriate gate.
A gate can be defined as a predefined visual association mechanism/construct for binding two or more distinct information excerpts (or portions thereof) though analytical content, such as but not limited to binding assertions with evidence. The gates can be along the right and left edges of the group box and can light up , for example, red or green (or other visual mechanism) when evidence is brought into the assertion. The green edge is lit up in Figure 13 to indicate that there is support rather than red for refute. The existence of the 'Refute Gate' should encourage the application of the scientific method, where the focus is on disproving hypotheses.
Further, nesting assertions is a way to create inference networks. At the top-left corner of each assertion, a graphical cumulative 'score' is displayed for 'at a glance' reminder of the level of evidence found so far. The cumulative score takes into account all tied evidence and nested assertions (linked assertions).
Page 29 of 36 4.4 Emphasis Module - Powerful-Finger and Other Gestures Level-of Detail (LOD) can be changed dynamically with the 'powerful-finger' mechanism. A simple gesture with a pointer (e.g. mouse) or touch screen command over a selected area of interest of the excerpt sandbox space can progressively add more details (e.g. seeing more text) and may scale the selected excerpt item (e.g.
an individual excerpt, portion thereof, or excerpt group) to add emphasis (size, position, colour, font style, bold, ect...) up to the desired effect. The revealed/emphasized details can either be persisted on the representation 18 (if for example that information is particularly relevant to current train of thoughts) or be temporary and fade back (if for example the details were quickly highlighted for emphasis for a colleague).
Using the Sandbox, the objective is to allow the analyst to deal with many more plausible scenarios, at once, than could have been done before. To support a higher volume of information and thoughts, in a 'put-this-there' environment, without hindering productivity, means addressing the mechanical cost of arranging items when the aim is simply to create or delete extra space for example. This is accomplished with a simple gesture encoding the command and its parameters. For example, the mode of selection of the excerpt (based on selection speed, direction, double/triple click, ect..) can be such that each mode has a predefined output format for emphasis of the selected item (excerpt and/or associated analytical content and links). One example could be that a single click does enlargement with no persistence while a double click does both enlargement, persistence and colour change. An amount of space in the sandbox area is created equivalent to the size of the gesture at the location and for example in the direction performed. The sandbox items can be animated to their new location and collision detection with gravity is used to help insure that all items will try to stay visible and to maintain the previous/selected order of the layout.
4.5 Scalability to Increase Volume of Thinking and Productivity The objective is to allow Sandbox content to scale to the massive without hindering the analyst's productivity. Each item in the Sandbox can be seen in the representation 18 at multiple levels of details (LOD) individually. At the lowest LOD, Page 30 of 36 intuitive iconic representations are compact and much easier to recognize and scan than text. Nothing needs to be hidden, the information elements is always kept, which should reduce cognitive load and help counter the recency bias. In the Sandbox, the user-created layout encodes meaning and context. LOD scaling with the powerful finger gesture maintains the semantics of the layouts. As shown in Figure 16, at all times, the 'big picture' (i.e. the full content of the Sandbox) can be visible in the Thumbnail Navigator at the lower left of the Sandbox. That window is used to navigate the Sandbox area.
Nothing needs to be hidden and the context of the information is easily accessible in TRIST. Changing LOD dynamically with the powerful finger gesture can maintain the semantics of the layouts. Thumbnail navigation for the Sandbox provides for quick directed movement within large Sandboxes. Thumbnail level of detail for groups minimizes a group while still preserving the semantic layout of information.
Items in the Sandbox can also be automatically arranged to "tidy up" information into lists and resize their parent groups to occupy the minimum amount of space. This process is animated, allowing the user to see where their information is being moved to.
4.6 Example Workflows - Module With a new tasking, the analyst might start brainstorming in the Sandbox, using prior and tacit knowledge to generate hypotheses or alternative scenarios, to jot down the key questions and maybe an analytical strategy or plan. Divergent thinking, crucial at this stage, is unrestricted in the Sandbox. Thoughts can flow freely, and be quickly organized to create more structure on the fly.
As relevant information is found in TRIST, it can be immediately integrated in the big picture, by either binding it to various alternatives or creating new ones to account for that evidence. The big picture in the Sandbox provides at a glance what needs to be proved or disproved, where evidence is lacking, and helps keep the collection process focused and efficient.
Page 31 of 3G

To reduce the cognitive biases associated with a particular mindset, exploring different perspectives is another key workflow. Trying to fit the collected evidence to different models might reveal gaps or different possible interpretations of the evidence.
the workflow module can either be prompted by the user or the module can prompt the user in coordination of workflow in analysis of the excerpts and associated analytical content.
4.7 Technical Architecture As shown in Figure 17, nSpace (TRIST and sandbox) uses a mufti-tier architecture for scalability and ease of deployment. Web Services standards are implemented to encapsulate the services in each tier. The client application accesses the Activity and Knowledge Base through the Application Services layer. Other background application processing, such as Search execution, can be offloaded to the Application Services layer to reduce load on the client. Resource Services provide wrappers around tools and databases that can be local or remote. Finally, if the Glass Box is present, it is used to log application usage and provide access to additional tools such as Ant Cafe Investigations. To improve performance, the Sandbox has been completely re-implemented to allow for hardware acceleration as well as both 2D and 3D forms of expression in one single workspace. Our visualization rendering platform supports a 100% Pure Java mode, or an optional acceleration mode that leverages OpenGL
hardware for 2D and 3D graphics. Sandbox can be a Java application that utilizes the Oculus.Java graphical library for visualizing the layout of information. The application employs a mufti-tier architecture for scalability and ease of deployment. Web Services standards can be implemented to encapsulate the services in each tier and to provide scalability, modularity and data processing functionality. An Activity and Knowledge Base is maintained and accessed through the Application Services layer. Other background application processing, such as Search execution, can be offloaded to the Application Services layer to reduce load on the client.
Page 32 of 36 New Capabilities in TRIST
While the recent focus has been on the Sandbox, a few additional capabilities have also been added to TRIST and are briefly recapped here.

5.1 Multiple Search Sources and Types and IE from Different Providers The nSpace environment has been designed for integrating and evaluating new science and technologies. It provides side-by-side comparison of alternative components and an integrated workspace to perform evaluations in a whole workflow context.
TRIST now provides access to additional search technologies and sources for obtaining the information excerpts. We started with 'Google Web Search, and now have added 'Google Image Search', 'NCBI PubMed Search', and the ability to search a corpus of documents compiled by Fair Isaac and a related database of all the entities contained in those documents.
5.2 Unsupervised Clustering (FI) Dimension The Fair Isaac Clustering Dimension uses unsupervised clustering (UC) in a region of the sandbox. Results dragged into that Dimension region are clustered automatically on the fly and categories representative of the clusters are created automatically. No prior knowledge of that category is necessary. UC can cluster independently of content, no ontology is needed (so for instance, documents could be in any foreign language, as long as corpus is also in that language). The analyst can reclassify all or a subset of the results on the fly, building quick understanding of content by quickly chunking it in many different ways. This functionality can also be used to isolate unusual results or specific results. Multiple dimensions of the problem are displayed and connected simultaneously.
5.3 Working at the Information Level with Entities In TRIST, people, organization and location entities are automatically extracted from documents, using content analysis technology from LCC and Fair Isaac. Now it is also possible to make "dimensions" for exploring and scanning entity relations.
Information level contact chaining exploration is possible. For example, the analyst Page 33 of 36 might be curious about a person being mentioned in a few relevant documents.
Dragging that person into a dimension might reveal connections to interesting organizations not necessarily mentioned in the set of relevant documents, rather than information excerpts that mention the same entity as connected to another information set of excerpts and associated information sources 14. The analyst may then follow her train of thoughts further and create a dimension for some of those organizations to quickly see who else work there and how they are related to the interesting person.

6 Experimental Plans for the Sandbox The first experiment with nSpace-Sandbox will be with the NIST surrogate analyst glass box environment with Naval Reservists as analysts. This experiment will provide an opportunity to assess an experimental protocol, gauge new types of metrics, gain feedback on Sandbox strengths and weaknesses and provide a target delivery milestone to focus software development. A similar methodology will be used as was followed for the earlier experiment with nSpace-TRIST.
Several planning discussions with NIST have been conducted. The Sandbox will be instrumented and analyst activity logged in the nSpace database as well as to the Glass Box data store. Additional activities required include the following:
~ Final planning with NIST incl. metrics and task scenario;
~ Draft brief Sandbox user documentation;
~ Deliver software to NIST to install and evaluate;
~ Revise software and documentation;
~ Deliver software for experiment;
~ Provide training for experiment participants;
~ Conduct experiment. Hot wash afterwards;
~ Analyze the results; and ~ Final report.
The plan is to conduct an experiment early in the new year. Some aspects of integrated information retrieval using TRIST may also need to be evaluated.
Page 34 of 36

7 Conclusions TRIST and the Sandbox are the first two components of nSpace, a fluid, flexible medium of analysis and expression. Early results show these tools provide a measurable significant increase in analyst productivity particularly when dealing with massive data.
Oculus will continue to evolve these tools as a framework for supporting the whole analytic workflow. Collaborations with analysts and other technology "partners" are essential for success.
7.1 Related Work Paper posted notes (Figure 22) are a common medium for analysis. They are inexpensive, easy to use, and flexible in terms of what can be captured on them and how they can be organized. Of course, an unusable critical mass of notes is reached fairly quickly, and content is difficult to scan, edit, re-arrange, update or transfer.
Concept Map software can be used to capture and organize thoughts with hierarchical linked text. In MindManager [MindJet] (figure 23), typing can be started anywhere in the thinking space. Thoughts can be placed anywhere but the dominant organization mechanism is a hierarchical tree with the root node centered in the middle of the space. Visual vocabulary and interactions are limited. Even for tree-like data structures, usability decreases as the volume of data increases.
Link analysis software, such as the Analyst's Notebook [i2] (figure 24) and Visual Links [Visual Analytics] (figure 25), has diagrammatic visual representations and is mainly used for visualizing connections (e.g. transactions, phone calls, 'is-related-to' relations, etc) between various types of entities. The icons and brief labels facilitate scanning. Over-simplification is possible as the full content and context of information is hidden. It seems better suited as a report tool than a thinking tool since it does not encourage various alternative thinking. As it is time consuming to layout the information, analysts may get attached to a particular layout and be reluctant to try Page 35 of36 different organization schemes that could provide different insights.
Furthermore, assumptions and evidence are not easily distinguishable.
rnNOrvmwrzeoz ~ 9aae~a zemuw.ee.rzwz I oo~u-Imo anlww~r~ T ~ 1~
7130 05:00 05:30 Oi:00 08 30 0790 07:30 ~ 00 09:00 f 000 10:30 t 1;00 . . t . , r , , n . . . . = n.~ . ~ . y,~, . . .. 1 '~~*11'I~~c,~!
~k*~'~_~tr~g~e~ ..,._,.
* ." " ~1KK CHART fOlt 5U5PECT OROAHQATION
seen W hotM 'h - rebpnwtw waeseen l8tbv30o21ao5 ø.
agrrwr h. rws~ h ~ m.r w no..n rwpane cd t a.rrwmm peNa.zuomat3 m »Mn ""
W a ro,~ zo07 o~:aa n~
_ _ romeaui9 Sf.lmwl of .,Ho vdbus Mown Afla pmnylOlrw trswA
pdta 9ebrte~i M

Figure 24 s - Analyst' Notebook i~ra~~~~~a!~ ~r ~s r-~ a......

Figure 25- Visual Links TOR LAW~5984338 v1 Page 36 of 36 Figure 22 - Posted Notes Figure 23 - MindManager

Claims