4.2 Grounding across different modes of interaction

4 Preliminary results

4.2 Grounding across different modes of interaction

We mentioned in section 2 that grounding mechanisms vary according to the communication channel. In our experiments, the subjects use two visual media: the MOO and the whiteboard, the former being text-based, while the latter supports both text and graphics. In each medium, they can negotiate through discussion or through action. This defines 4 modes of negotiation, as illustrated in table 3.

Modes of interaction
	MOO	Whiteboard
Action	1) questioning suspects, looking at objects, moving...	3) creating or modifying rectangles, texts, arrows
Dialogue	2) communicating with each other with 'say' or 'page'	4)posting responsive texte.g. Hercule writes "What do you mean" on whiteboard (see example 18)

The example of dialogue through the whiteboard (4) are very rare (e.g. Hercule writes "What do you mean" on whiteboard - see example 18), primarily due to the easier use of the moo and lack of need to erase this only temporarily relevant information. We focus here on the other spaces: (1) MOO action, (2) MOO discussion and (3) whiteboard usage. The implicit hypothesis underlying our project was that the key function was 3=> 2: drawing a schema visible by both agents facilitates grounding of verbal utterances. This facilitation was expected to be partly due to deictic features. In previous experiments, we analysed the gestures of 8 pairs of subjects using the MEMOLAB system [Dillenbourg94]: the large majority of their gestures have a simple deictic function[5]: "Copy this event", "Put it there", ... The technical choices we made for this experiment actually prevented straightforward deictic gestures: with the "BeingThere" Whiteboard, the subjects do not see each other's cursor, although they could point by drawing or moving a marker. In addition, deictic gestures work when utterances and gestures are synchronous, which was possible in MEMOLAB because communication was vocal, but was not possible in these experiments, since communication was typed. The analysis of the protocols shows that these three spaces have a much richer relationship. It is not simply the whiteboard which support grounding utterances, (3 => 2), but also the utterance being used to ground the information put on the whiteboard (2 => 3). Among all the possible relations between these 3 modes, table 4 examines those that we actually observed in the protocols.
Examples of multi-modal grounding (Numbers refer to the examples below)
Info in row i grounded by act in column j MOO Action MOO dialogue Whiteboard Action
MOO Action A follows B through the MOO: the arrival of A in room X is acknowledged by the fact that B leaves for room Y. MOO actions are grounded through dialogues: The utterance acknowledges the action in examples 23 and 24, it expresses disagreement in example 25.
MOO dialogue Verbal utterances grounded by MOO actions: acknowledgment in example 26 and repair in example 27.

The level of grounding seems different in voice versus MOO based dialogues. Voice dialogues include many grounding around referential problems (example 28) while MOO grounding dialogues are more complex (example 29). Grounding conversation through whiteboard action (acknowledgment in example 19). In example 31, grounding seems to be performed simultaneously through discussion and through whiteboard action.
Whiteboard Action Grounding whiteboard information through conversation (a repair in example 30). In example 31 grounding seems to be performed simultaneously through discussion and through whiteboard action. Whiteboard action grounded by whiteboard action: acknowledgment (example 32) or disagreement (example 33).

Examples of multi-modal grounding (Numbers refer to the examples below)
Info in row i grounded by act in column j	MOO Action	MOO dialogue	Whiteboard Action
MOO Action	A follows B through the MOO: the arrival of A in room X is acknowledged by the fact that B leaves for room Y.	MOO actions are grounded through dialogues: The utterance acknowledges the action in examples 23 and 24, it expresses disagreement in example 25.
MOO dialogue	Verbal utterances grounded by MOO actions: acknowledgment in example 26 and repair in example 27.	The level of grounding seems different in voice versus MOO based dialogues. Voice dialogues include many grounding around referential problems (example 28) while MOO grounding dialogues are more complex (example 29).	Grounding conversation through whiteboard action (acknowledgment in example 19). In example 31, grounding seems to be performed simultaneously through discussion and through whiteboard action.
Whiteboard Action		Grounding whiteboard information through conversation (a repair in example 30). In example 31 grounding seems to be performed simultaneously through discussion and through whiteboard action.	Whiteboard action grounded by whiteboard action: acknowledgment (example 32) or disagreement (example 33).

[5] The pointing to the referred object or location was performed either by the finger or by the mouse cursor.

Grounding in Multi-modal Task-Oriented Collaboration - 3 SEP 1996

Generated with Harlequin WebMaker