Conversation and Coordinative Structures

Cognitive phenomena do not always respect the boundaries that are observed by textbooks and conferences in cognitive science. Consider the mirror neuron system—neurons in the premotor cortex that fire both when observing and performing an action. This discovery has challenged the assumed distinction between perception and action in the brain. While the human capacity to perceive opportunities for action (i.e., affordances; Gibson, 1979) has been extensively documented (see Fajen, Riley, & Turvey, in press, for a recent review), it appears that even non-action-related judgments are in the currency of action. For example, perceived distances appear to be scaled by the metabolic cost associated with traversing a distance rather than a strict function of visual information (e.g., Proffitt, 2006).

Likewise, action appears to embody cognitive processing. Cognitive performance can interfere with concurrent motor performance, and vice versa (see Barsalou, Simmons, Barbey, & Wilson, 2003, for a recent review). Moreover, the inhibition of motor activity during cognitive processing (Frick-Horbury & Guttentag, 1998) or imposed inconsistency between motor activity and cognitive responses (Glenberg & Kaschak, 2001; de’Sperati & Stucchi, 2000) appears to disrupt cognitive performance. Lastly, the time course of even high-level cognitive processes is reflected in the trajectory of action. When clicking “yes” or “no” to questions ranging in fuzzy truth-value, (e.g., Should you brush your teeth everyday? Is murder sometimes justified?), the degree of motor deviation and complexity, measured in computer–mouse trajectories, reflects the degree of cognitive certainty (McKinstry, Dale, & Spivey, 2008). The implication of all these findings is that a full understanding of perception, action, and cognition will not obtain by studying these domains in isolation.

Cognitive phenomena do not always respect the boundaries that we draw between people either. In theory and in experimental practice, cognitive science has been concerned with the individual in isolation. But there have been striking insights into the brain’s sensitivity to social information (for a review, see Cacioppo, Visser, & Pickett, 2006) and the way in which people think and act cooperatively (Hutchins, 1995). For example, Clark (1996) re-conceptualized language as a process spanning individual and social scales, rather than as exclusively an encapsulated cognitive process within an individual. These observations have led to the recent interest in situated cognition (Robbins & Aydede, 2008) and joint action (Sebanz, Bekkering, & Knoblich, 2006; and elsewhere in this issue). An essential feature of joint action is how each person’s nuanced actions shape the other’s as the joint action unfolds. For example, when two people talk, they spontaneously converge in terms of pausing and speaking duration, speaking rate, turn duration, response latency, vocal intensity, and accent. Movements and postures, likewise, appear to be spontaneously coordinated with speech within and across individuals (see Fowler, Richardson, Marsh, & Shockley, 2008, for a review).

In this paper, we review a variety of findings suggesting that in these phenomena not only is action coordinated across individuals but also is cognition. We consider what this observed coordination may be telling us about cognition and the nature of the coordination itself. One explanation for behavioral coordination in the context of conversation is that it exists to align cognitive representations (e.g., Garrod & Pickering, 2004; Pickering & Garrod, 2004; Sebanz et al., 2006). Here, we explore an alternative possibility: interpersonal coordination is a case of an emergent coordinative structure. This explanation draws on models in physics and biology, and need not involve high-level, cognitive representations. For example, when a thin layer of fluid is heated, small-scale high-dimensional interactions result in a large-scale, lower-dimensional order. The molecules form into rolling columns that efficiently distribute heat across the area of the fluid (e.g., Kelso, 1995). There is no temptation here to attribute such coordination to a “central physics executive” that is directing the fluid molecules. The coordinated behavior of molecules emerges naturally from local interactions and constraints. The possibility we raise at the close of this paper is whether a similar process can explain the coordinated behavior between people. We first consider two sources of empirical evidence for such coordination during interaction: postural coordination and coordination of gaze.

Coordination of postural configurations and body movements in the context of language has been observed for decades. Historically, however, the coordination of movements across people has been evaluated using largely subjective methods, such as hand-scoring video tapes of interactions (e.g., Bernieri, 1988). More recently, the spontaneous coordination of conversants’ movements has been objectively quantified using cross-recurrence quantification (CRQ). CRQ is a nonlinear method that quantifies how similarly two observed data series unfold over time (Shockley, Butwill, Zbilut, & Webber, 2002; Zbilut, Giuliani, & Webber, 1998). Like cross-correlation or more recently windowed cross-correlation (Boker, Xu, Rotondo, & King, 2002), CRQ produces measures reflecting coordination, such as how often two data series have similar values or reflect similar system states (called percentage recurrence, or %REC), among other measures. Work in the physical sciences has shown that a benefit of CRQ is its resistance to noise and freedom from assumptions. These features make it well suited to studying noisy behavioral signals generated during naturalistic interaction (see Marwan, Romano, Thiel, & Kurths, 2007, for review of CRQ and related methods). We detail three studies demonstrating postural coordination that use this method.

In a first study, Shockley, Santana, and Fowler (2003) evaluated the shared postural configurations of pairs of people engaged in a conversation by measuring their postural sway trajectories—the subtle, continuous motion of the body during upright standing. Postural sway was taken as a global measure of movement because it reflects the coordination of all body segments required to maintain the center of mass of the body within the base of support (i.e., the extent of the feet on the ground surface) that is required to stand upright. To generate conversation among participants, Shockley et al. (2003) used a puzzle task in which the two participants of a given pair could not see their partner’s picture and had to discuss the pictures to discover the differences (Fig. 1, left). Participants were allowed to gesture, and speak freely, which generated a very natural conversation between members of a pair. To control for any influence of the task itself (i.e., looking and speaking while standing) on the coordination, participants were partnered with either the other participant in the pair or each member of the pair was partnered with an experimental confederate (C1 or C2 in Fig. 1, left). To control for any influence of vision (e.g., seeing another participant sway and/or gesture while doing the task), participants were either oriented toward the other member of the participant pair or they faced away from the other member of the pair (confederates were always out of view) (Fig. 1, left). For measurements taken at the waists of participants, as illustrated in Fig. 1 (right), Shockley et al. (2003) found that participant pairs that performed the task together shared more postural configurations (%REC) and maintained similar postural trajectories for longer (indexed by the CRQ measure termed MAXLINE—the length of the longest common trajectory across the two members of a pair) than participant pairs that performed the task with an experimental confederate, irrespective of whether they were facing the other participant.

In a subsequent study, Shockley, Baker, Richardson, and Fowler (2007) investigated the nature of the coordination observed by Shockley et al. (2003). They considered the possibility that participants may have converged in their speaking patterns over time and that coordinated speech patterns may have mediated coordinated postural sway patterns. Each member of a pair was presented with a target word that was either the same as his/her partner’s (S), a different word for each member of the pair but with the same syllable emphasized (DS) (e.g., ethnic and ancient have an emphasis on the first syllable), or a different word for each member of the pair with a different syllable emphasized (DD) (e.g., ethnic has an emphasis on the first syllable while deserve has an emphasis on the second syllable) (Fig. 2, top). As illustrated in Fig. 2 (bottom), they found greater shared postural configurations when words were more similar than when words were less similar, suggesting that the shared postural activity observed by Shockley et al. (2003) may have indeed been mediated by convergent speaking patterns. They also compared postural sway trajectories of participants who uttered the same sequence of words but compared them with a participant who did the same task with a different partner (i.e., virtual pairs; see Fig. 2, bottom) where they found no influence of word similarity. The implication is that the participants had to be performing the task together, suggesting that the postural coordination observed by Shockley et al. (2003) entailed a social constraint in addition to the constraints of articulation.

Although Shockley et al. (2003) found that it made no difference if participants were facing their task partner, more recently, Giveans, Pelzer, Smith, Shockley, and Stoffregen (2008) did find evidence of visual constraints on shared postural activity. Giveans et al. manipulated the relative distances of the target puzzle pictures for a participant pair and the relative sizes of the target puzzle pictures (both large both small, or one was small while the other was large). These manipulations served to constrain the looking patterns of the conversing participants by having both participants scan a larger or smaller area or have one scan a larger area while the other scanned a smaller area. They found greater shared head configurations when both targets were near participants or large than when both targets were farther away, small, or mismatched in size. The implication of the findings of Giveans and colleagues is that visual constraints influence the observed postural coordination when two persons are conversing. The influence of looking patterns on interpersonal postural coordination is of particular significance because recent research has demonstrated that gaze patterns embody the joint understanding involved in conversation. The following section showcases these studies, which focus on gaze coordination during interaction.

Consider an argument over a map, a debate over a proof written out on a blackboard, a civilized conversation about a painting at a gallery or an idle chat during a commercial break. In all these cases, a visual scene is shared while people talk to each other. Here, we describe research that quantifies such joint attention by measuring the eye movements of two people discussing a common visual scene. We have found that the coordination between their gaze coordination reveals their shared knowledge, their beliefs about each other, and the success of their communication.

In the first quantification of gaze coordination (Richardson & Dale, 2005), the speech and eye movements of one set of participants were recorded as they looked at pictures of six cast members of a TV sitcom and spoke spontaneously about their favorite episode and characters. One-minute segments were played back unedited to a separate set of participants. The listeners looked at the same visual display of the cast members, and their eye movements were also recorded as they listened to the segments of speech. They then answered a series of comprehension questions. CRQ was used to quantify the degree to which speaker and listener eye positions overlapped at successive time lags (Fig. 3). From the moment a speaker looked at a picture, and for the following 6 s, a listener was more likely than chance to be looking at that same picture. The listener was most likely to be looking at the same cast member 2 s after the speaker fixated it. The degree of recurrence between individual speaker–listener pairs correlated with the listeners’ accuracy on the comprehension questions. A second experiment showed that gaze coordination and comprehension were causally connected. Pictures flashing in time with the speakers’ fixations (compared with a randomized version) caused the listeners’ eye movements to look more like the speakers.’ This experimental manipulation improved the speed of listeners’ performance on comprehension questions.

The language use in this study, however, lacked a key element of everyday conversations—interaction. In another study, the gaze of two conversants engaged in a live dialog was tracked simultaneously while they discussed TV shows and surrealist paintings (Richardson, Dale, & Kirkham, 2007a). Conversants’ eye movements were coupled as they looked at a shared display, peaking at a lag of 0 ms. In other words, the conversants were most likely to be looking at the same thing at the same point in time. As in the monolog results, this coupling was above-randomized baseline levels for a period of around 6 s.

Such coordination in conversation can be achieved, Clark (1996) argues, because conversants share not only a stream of words but also a common body of background knowledge that allows them to ground and understand each others’ utterances. This level of “common ground” knowledge was experimentally manipulated, and its effect on gaze coordination assessed (Richardson, Dale, & Kirkham, 2007a). Participants first listened to a 90-s passage describing either to the meaning of a specific painting by Dali or facts from his biography. They then saw Dali’s painting and discussed it while their gaze was tracked. Conversational partners who heard the same information had higher gaze coordination (in the region around 0 ms of lag) than those who heard different information. Interestingly, it did not seem to matter which of the passages the conversants heard. What was important was that they had the same information and knew this to be the case.

This raises an interesting question: How is gaze coordination influenced by what conversants believe about each other’s beliefs? We studied the gaze coordination of conversants while varying what they saw and what they believed each other could see (Richardson, Dale, Tomlinson, & Clark, 2008). In previous “reference game” studies of visual common ground (e.g., Hanna, Tanenhaus, & Trueswell, 2003; Keysar, Barr, Balin, & Brauner, 2000), these factors are not separated, and the linguistic interaction is a monolog directed at objects on display. In our study, participants had a spontaneous conversation about a controversial topic. Before their discussion, pairs of participants watched videos of actors giving opposing views on a topic such as the Iraq War. During their discussion, two factors were manipulated (Fig. 4A). While they talked, either both looked at a blank screen or both looked at pictures of the actors. In addition, either both believed that their partner was looking at a blank screen, or both believed that their partner was looking at the actors. These factors were varied at random across eight conversations. On average, both the presence of the visual scene and beliefs about its presence for another influenced participants’ gaze coordination. Recurrence increased overall when the pictures of actors were onscreen, as, unsurprisingly, conversants were less likely to look at a blank screen. When they were looking at pictures of the actors, conversants’ eye movements were more tightly coordinated if they believed they were talking to someone who was looking at nothing. Conversely, while looking at an empty screen, their eye movements were more closely coordinated if they believed that each other could still see the actors (Fig. 4B). In other words, when they believed there to be an imbalance between their own view and their partners’ view, gaze coordination increased. Analysis of conversants’ speech showed that when they believed they were looking at different things, they used a variety of techniques to boost common ground. They tended to refer to actors’ viewpoints via a nonvisual route, raise the intonation at the end of their phrases (as an implicit request for confirmation that their messages had been understood), and signal understanding to each other via back channel responses (such as saying “ok” or “uh-huh”).

Gaze coordination is causally connected to what people know and remember, and what conversants see and what they believe each other can see. In other words, eye movements embody the cognitive processing involved in cooperative conversation and the shared knowledge among interlocutors. Can these phenomena of gaze coordination and shared knowledge be understood in the same terms as postural coordination?

How do we develop a generic and principled understanding of how minds and bodies move together (Sebanz et al., 2006)? We argue, first, that only by abandoning the historic boundaries among the subdomains of psychology will we achieve a principled understanding interpersonal coordination. Secondly, we have provided a speculative account of how behavioral coordination across individuals can be conceptualized as a functionally defined cross-person coordinative structure that embodies the goals of the joint action system. A challenge for this general approach to understanding joint action is to understand how it might scale up to “representationally hungry” (Clark, 1997) domains at higher levels of linguistic description (Garrod & Pickering, 2004). This, of course, is a challenge shared by all embodied accounts of cognition (Wilson, 2002). We argue that meeting this challenge in the case of interpersonal coordination—understanding its cognitive dynamics in terms of coordinative structures—brings two rewards. First, it offers a principled account of how two cognitive systems can become aligned over time. For example, it may offer an understanding of the gradual entrainment of action and cognition using general principles that explain the entrainment of two physical oscillators or two limb movements. Second, it may allow cognitive coordination to be discussed in the same language as movement coordination. Such a “universal language” of joint action may best capture these itinerant phenomena that wander across boundaries between perception and action, self, and other.