A neural model of sensorimotor experience, and of the representation, storage and communication of events

Many cognitive scientists have advanced ‘embodied’ models of human language, in which language is connected in some way to the sensorimotor (SM) mechanisms that engage with the world. I’ll introduce a particular version of this idea, that has relevance for models of how language interfaces with long-term memory and with the emotional system.

The foundation for my model is Dana Ballard’s (1997) proposal that the SM processes through which an agent engages with the world are structured as deictic routines: well-defined sequences of relatively discrete atomic attentional, sensory or motor actions (called deictic operations). I propose that agents experience sentence-sized ‘events’ in the world through deictic routines, whether they are observing them or participating in them. I further propose that agents represent events in working memory (WM) as prepared deictic routines: that is, as ‘executable’ representations, that can be performed, or simulated. In the model I propose, these executable event representations provide the interface between language and long-term memory (LTM). When an event has been experienced, its complete WM representation can be registered in LTM, and the WM representation can be cleared, ready for the next event. (It will be registered more strongly if it has strong emotional connotations.) A complete WM event representation can also be communicated, by simulating it in a special ‘language mode' where SM signals can trigger output phonology.

This model supports an interesting account of how memory operations surface in language. The key idea here is that operations accessing memory, or putting the agent into other cognitive modes, should also be regarded as ‘deictic operations’ - ones that happen at the very start of a deictic routine. On this model, when experiencing an event, the first thing the agent must do is to decide whether to retrieve an event from memory (or some other cognitive modality like imagination), or to engage with the sensorimotor here-and-now. These different options are each implemented by a deictic operation. (Thus there are separate deictic operations establishing ‘LTM retrieval mode’, ‘experience mode’ and so on.) Crucially, these mode-setting deictic operations are also stored in the WM medium encoding events, which interfaces with language.

This model of the SM system and its interfaces to memory offers some interesting ideas for linguists. At the level of syntax, stored mode-setting operations provide interesting possible denotations for tense mophology in sentences, and of several closed-class/modal verbs (including verbs expressing emotional experience, like ‘feel’). At the level of discourse, the model provides a neural implementation of several ideas from update semantics. I’ll focus on monologue in my talk, but I will outline possible extensions to dialogue.