According to the decomposition thesis, perceptual experiences resolve without remainder into their different modality-specific components. Contrary to this view, I argue that certain cases of multisensory integration give rise to experiences representing features of a novel type. Through the coordinated use of bodily awareness—understood here as encompassing both proprioception and kinaesthesis—and the exteroceptive sensory modalities, one becomes perceptually responsive to spatial features whose instances couldn’t be represented by any of the contributing modalities functioning in isolation. I develop an argument (...) for this conclusion focusing on two cases: 3D shape perception in haptic touch and experiencing an object’s egocentric location in crossmodally accessible, environmental space. (shrink)

The present study investigated how multisensory integration in peripersonal space is modulated by limb posture and limb congruency . This was done separately for the upper limbs and the lower limbs . The crossmodal congruency task was used to measure peripersonal space integration for the hands and the feet. It was found that the peripersonal space representation for the hands but not for the feet is dynamically updated based on both limb posture and limb congruency. Together (...) these findings show how dynamic cues from vision, proprioception, and touch are integrated in peripersonal limb space and highlight fundamental differences in the way in which peripersonal space is represented for the upper and lower extremity. (shrink)

The integration of multisensory information is an essential mechanism in perception and action control. Research in multisensory integration is concerned with how the information from the different sensory modalities, such as the senses of vision, hearing, smell, taste, touch, and proprioception, are integrated to a coherent representation of objects (for an overview, see e.g., Calvert, Spence and Stein, 2004). The combination of information from the different senses is central for action control. For instance, when you grasp for a (...) rubber duck, you can see its size, feel its compliance and hear the sound it produces. Moreover, identical physical properties of an object can be provided by different senses. You can both see and feel the size of the rubber duck. Even when you grasp for the rubber duck with a tool (e.g., with tongs), the information from the proximal hand, from the effective part of the distal tool and from the eyes are integrated in a manner to act successfully (for limitations of this integration see Sutter et al., 2013).Over the recent decade a surge of interest in multisensory integration and action control has been witnessed, especially in connection with the idea of a statistically optimized integration of multiple sensory sources. The human information processing system is assumed to adjust moment-by-moment the relative contribution of each sense’s estimate to a multisensory task. The sense’s contribution depends on its variance, so that the total varia... (shrink)

The Bayesian model of multisensory cue integration proposed by Ernst and Banks provides an attractive model for understanding a way that our sensory systems may interact. Moreover, it has been suggested that the process of multisensory integration that it models underpins conscious experiences with multisensory representational contents merged across modalities. Should we therefore take empirical support for the Bayesian model as evidence of the multimodality of perception? Focusing on evidence of integration across vision and touch, I argue (...) that apparent support for the model does not warrant the rejection of the view that each of our conscious perceptual experiences is associated with one and only one sense modality. (shrink)

Philosophers and scientists have studied sensory perception and, in particular, vision for many years. Increasingly, however, they have become interested in the nonvisual senses in greater detail and the problem of individuating the senses in a more general way. The Aristotelian view is that there are only five external senses—smell, taste, hearing, touch, and vision. This has, by many counts, been extended to include internal senses, such as balance, proprioception, and kinesthesis; pain; and potentially other human and nonhuman senses. This (...) “multisensory turn” has been driven partly by developments in contemporary psychology and neuroscience, which have revealed a host of complex interrelations and interactions between sensory modalities previously thought to be distinct. Contrasts between modalities and other crossmodal phenomena, including multisensory integration, synesthesia, and sensory substitution, have also begun to receive more attention in a burgeoning scientific and philosophical literature on multisensory perception and other crossmodal effects. This article focuses on recent empirically informed contributions to the philosophy of perception, as well as key scientific works that provide important background information and insights into the nature of the senses and sensory perception. Indeed, one of the lessons of the multisensory turn, and of contemporary philosophy of mind more generally, is that philosophers ignore this body of empirical research at their peril because many human and animal senses turn out to be richer and more complex than philosophers and scientists had previously imagined, making this a fruitful area for interdisciplinary interaction and research. (shrink)

Many organisms possess multiple sensory systems, such as vision, hearing, touch, smell, and taste. The possession of multiple ways of sensing the world offers many benefits. However, combining information from different senses also poses many challenges for the nervous system. In recent years there has been dramatic progress in understanding how information from the different senses gets integrated in order to construct useful representations of external space. This volume brings together the leading researchers from a broad range of scientific (...) approaches to present the first overview of this central topic in cognitive neuroscience. (shrink)

Our perception of where touch occurs on our skin shapes our interactions with the world. Most accounts of cutaneous localisation emphasise spatial transformations from a skin-based reference frame into body-centred and external egocentric coordinates. We investigated another possible method of tactile localisation based on an intrinsic perception of ‘skin space’. The arrangement of cutaneous receptive fields (RFs) could allow one to track a stimulus as it moves across the skin, similarly to the way animals navigate using path integration. (...) We applied curved tactile motions to the hands of human volunteers. Participants identified the location midway between the start and end points of each motion path. Their bisection judgements were systematically biased towards the integrated motion path, consistent with the characteristic inward error that occurs in navigation by path integration. We thus showed that integration of continuous sensory inputs across several tactile RFs provides an intrinsic mechanism for spatial perception. (shrink)

This report highlights and explores five questions that arose from the multisensory integration workshop at the University of Toronto on May 9th and 10th, 2014: 1. What Is Multisensory Integration? 2. Do Multisensory Percepts Involve Emergent Features? 3. What Can Multisensory Processing Tell Us about Multisensory Awareness? 4. Is Language Processing a Special Kind of Multisensory Integration? 5. What Is the Purpose of Multisensory Integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Do multisensory percepts involve emergent features?

Suppose that you are at a live jazz show. The drummer begins a solo. You see the cymbal jolt and you hear the clang. But in addition seeing the cymbal jolt and hearing the clang, you are also aware that the jolt and the clang are part of the same event. Casey O’Callaghan (forthcoming) calls this awareness “intermodal feature binding awareness.” Psychologists have long assumed that multimodal perceptions such as this one are the result of a subpersonal feature binding mechanism (...) (see Vatakis and Spence, 2007, Kubovy and Schutz, 2010, Pourtois et al., 2000, and Navarra et al., 2012). I present new evidence against this. I argue that there is no automatic feature binding mechanism that couples features like the jolt and the clang together. Instead, when you experience the jolt and the clang as part of the same event, this is the result of an associative learning process. The cymbal’s jolt and the clang are best understood as a single learned perceptual unit, rather than as automatically bound. I outline the specific learning process in perception called “unitization,” whereby we come to “chunk” the world into multimodal units. Unitization has never before been applied to multimodal cases. Yet I argue that this learning process can do the same work that intermodal binding would do, and that this issue has important philosophical implications. Specifically, whether we take multimodal cases to involve a binding mechanism or an associative process will have impact on philosophical issues from Molyneux’s question to the question of how active or passive we consider perception to be. (shrink)

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What can multisensory processing tell us about multisensory awareness?

A visual analogue, two-route model of somatosensory processing is advanced in this commentary. Touch for perception is seen as separate from, although interconnected with, touch for action. Separate modules are additionally proposed for internal (body) and external (object-related) somatosensation. Here we ask whether dissociation (divide) guarantees better efficiency (impera) in terms of the heuristic model within the somatosensory modality and across modalities.

This chapter offers an indirect defence of the Evansian conception of egocentric space, by showing how it resolves a puzzle concerning the unity of egocentric spatial perception. The chapter outlines several common assumptions about egocentric perspectival structure and argues that a subject’s experience, both within and across her sensory modalities, may involve multiple structures of this kind. This raises the question of how perspectival unity is achieved, such that these perspectival structures form a complex whole, rather (...) than merely disunified set of individually, distinctively structured experiences. The shortcomings of variety of accounts are considered: switch accounts ; sensory accounts; transformation accounts; and ultimate accounts. These shortcomings are addressed by a further kind of account provided by the Evansian conception – an agentive account – according to which egocentrically structured experiences present the world in relation to parts of a single thing, the body as a dynamic unity. (shrink)

In the last decades, the rapid growth of functional brain imaging methodologies allowed cognitive neuroscience to address open questions in philosophy and the social sciences. At the same time, novel insights from cognitive neuroscience research have begun to influence various disciplines, leading to a turn to cognition and emotion in the fields of planning and architectural design. Since 2003, the Academy of Neuroscience for Architecture has been supporting ‘neuro-architecture’ as a way to connect neuroscience and the study of behavioral responses (...) to the built environment. Among the many topics related to multisensory perceptual integration and embodiment, the concept of hapticity was recently introduced, suggesting a pivotal role of touch and haptic imagery in architectural appraisal. Arguments have thus risen in favor of the existence of shared cognitive foundations between hapticity and the supramodal functional architecture of the human brain. Precisely, supramodality refers to the functional feature of defined brain regions to process and represent specific information content in a more abstract way, independently of the sensory modality conveying such information to the brain. Here we highlight some commonalities and differences between the concepts of hapticity and supramodality according to the distinctive perspectives of architecture and cognitive neuroscience. This comparison and connection between these two different approaches may lead to novel observations in regard to people-environment relationships, and even provide empirical foundations for a renewed evidence-based design theory. (shrink)

This paper examines Milner and Goodale’s hypothesis about the two visual streams and raises the questions of whether properties in egocentric space (commonly associated with the vision-for-action, or "dorsal," stream) can be part of the phenomenal content of perceptual experience, or only properties in allocentric space (commonly associated with the vision-for-perception, or "ventral," stream) can play this role, and how (if at all) properties in egocentric space differ from properties in allocentric space. These questions (...) are reminiscent of issues raised by Christopher Peacocke's "equal-sized trees" case: if you are looking at equal-sized trees on the side of a road, the trees will phenomenally look to you to be equally tall. But it is also true that a tree will take up more of the space in your visual field the closer it is to you. So, the equal-sized trees will also phenomenally look to you to have different heights. Your (visual) perceptual experience is not illusory. That is, the trees veridically look to you to have the same height, and they also veridically look to you to have different heights. Call the sort of issue raised here the "problem of dual looks." I provide a solution to the problem of dual looks that helps shed light on whether properties in egocentric space can be part of the phenomenal content of perceptual experience, and how (if at all) properties in egocentric space differ from properties in allocentric space. (shrink)

Our emotional states affect how we perceive the world. If I am stressed, annoyed, or irritated, I might experience the sound of children laughing and screaming as they play around the house in a negative manner — it is unpleasant, loud, piercing, and so on. Yet, if I’m in a relaxed, happy, loving mood, the very same sounds might be experienced as pleasant, playful, warm, and so on. The sounds being made by the children are the same in both cases, (...) but how they are experienced differs. The auditory experience is affected by the emotional state that I am in. I perceive the sounds differently depending on how I’m feeling. Although this might at first seem to be a trivial observation, it certainly is not. We take for granted that our perceptual experiences give rise to justified beliefs and knowledge. Thus, if my perceptual experience of an object can differ based on my emotional state, and I take my perceptual experience to justify my beliefs and even lead to knowledge, then we might have a problem. If emotions — which are not fixed and in large part uncontrolled — affect our ability to accurately perceive the world, then they may undermine the justified beliefs and knowledge gained on the basis of our perceptual experience of the world. My goal here is to explore this potential problem. -/- I consider how we might understand the effect that emotions have on the justification of our perceptual beliefs about the world, beliefs that we acquire from a variety of sensory modalities — audition, gustation olfaction, and so on. I take the problem to be associated with one of two forms of perceptual influence: penetration or multisensory integration. In any given perceptual moment there are multiple sensory modalities and mental states at play, each affecting the overall experience. Whether we understand the influence of emotion on perception as a form of non-perceptual penetration or a form of non-visual sensory perception of the inner body — interoception — the potential epistemological difficulties remain: how can we be said to acquire justified beliefs and knowledge on the basis of such influenced perceptual experience. As has been the norm, only the five exteroceptive senses of vision, audition, olfaction, taste and touch are typically discussed in the context of sensory perception. However, as I argue, there is strong reason to accept the claim that emotional experience is a form of interoception, and that interoception ought to be considered when discussing sensory perception. In this way, then, I propose that clarifying the role played by interoception in sense perception across modalities is necessary if we are to make progress on the epistemological problems at hand. (shrink)

Multisensory processing encompasses all of the various ways in which the presence of information in one sensory modality can adaptively influence the processing of information in a different modality. In Part I of this survey article, I begin by presenting a cartography of some of the more extensively investigated forms of multisensory processing, with a special focus on two distinct types of multisensory integration. I briefly discuss the conditions under which these different forms of multisensory processing (...) occur as well as their important perceptual consequences and interrelations. In Part II, I then turn to examining of some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical attempts to individuate the senses. (shrink)

Neuropsychological findings used to motivate the "two visual systems" hypothesis have been taken to endanger a pair of widely accepted claims about spatial representation in conscious visual experience. The first is the claim that visual experience represents 3-D space around the perceiver using an egocentric frame of reference. The second is the claim that there is a constitutive link between the spatial contents of visual experience and the perceiver's bodily actions. In this paper, I review and assess three (...) main sources of evidence for the two visual systems hypothesis. I argue that the best interpretation of the evidence is in fact consistent with both claims. I conclude with some brief remarks on the relation between visual consciousness and rational agency. (shrink)

We can perceive shapes visually and tactilely, and the information we gain about shapes through both sensory modalities is integrated smoothly into and functions in the same way in our behavior independently of whether we gain it by sight or touch. There seems to be no reason in principle we couldn't perceive shapes through other sensory modalities as well, although as a matter of fact we do not. While we can identify shapes through other sensory modalities—e.g., I may know by (...) smell (the scent of mango) that the object causing my sensory experience is round—this is not perceiving an object as shaped, but rather inferring from the character of one's sensory experience and collateral information that an object of a certain shape caused it. That it is possible to perceive shape by other modalities, however, is suggested by the case of bats and aquatic mammals like dolphins which navigate through their environment by a form of sonar. It is plausible that they have some form of auditory representation of space, and so of shape. These facts about shape perception raise important questions about the relation between those features of perceptual experience which are intrinsic to different sensory modalities and the nature of our perceptual representation of shapes, and, more generally, of the space within which we perceive shaped objects to be located. John Campbell's paper, "Molyneux's Problem" (see above), raises a number of interesting and important questions about the nature of our perception of shape properties, particularly the cross-modal nature of shape perception, and ties them to more general questions about the nature both of perceptual.. (shrink)

The first part of this survey article presented a cartography of some of the more extensively studied forms of multisensory processing. In this second part, I turn to examining some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical debates concerning the modularity of perception, cognitive penetration, and the individuation of (...) the senses. (shrink)

Molyneux's Question (MQ) concerns whether a newly sighted man would recognize/distinguish a sphere and a cube by vision, assuming he could previously do this by touch. We argue that (MQ) splits into questions about (a) shared representations of space in different perceptual systems, and about (b) shared ways of constructing higher dimensional spatiotemporal features from information about lower dimensional ones, most of the technical difficulty centring on (b). So understood, MQ resists any monolithic answer: everything depends on the constraints (...) faced by particular perceptual systems in extracting features of higher dimensionality from those of lower. Each individual question of this type is empirical and must be investigated separately. We present several variations on MQ based on different levels of dimensional integration—some of these are familiar, some novel adaptations of problems known elsewhere, and some completely novel. Organizing these cases in this way is useful because it unifies a set of disparate questions about intermodal transfer that have held philosophical and psychological interest, suggests a new range of questions of the same type, sheds light on similarities and differences between members of the family, and allows us to formulate a much-augmented set of principles and questions concerning the intermodal transfer of spatiotemporal organization. (shrink)

To form a coherent percept of the environment, our brain combines information from different senses. Such multisensory integration occurs in higher association cortices; but supposedly, it also occurs in early sensory areas. Confirming the latter hypothesis, we unequivocally demonstrate supra-additive integration of touch and sound stimulation at the second stage of the auditory cortex. Using high-resolution fMRI of the macaque monkey, we quantified the integration of auditory broad-band noise and tactile stimulation of hand and foot in anaesthetized animals. Integration (...) was found posterior to and along the lateral side of the primary auditory cortex in the caudal auditory belt. Integration was stronger for temporally coincident stimuli and obeyed the principle of inverse effectiveness: greater enhancement for less effective stimuli. These findings demonstrates that multisensory integration occurs early and close to primary sensory areas and—because it occurs in anaesthetized animals—suggests that this integration is mediated by preattentive bottom-up mechanisms. (shrink)

It remains controversial whether touch is a truly spatial sense or not. Many philosophers suggest that, if touch is indeed spatial, it is only through its alliances with exploratory movement, and with proprioception. Here we develop the notion that a minimal yet important form of spatial perception may occur in purely passive touch. We do this by showing that the array of tactile receptive fields in the skin, and appropriately relayed to the cortex, may contain the same basic informational building (...) blocks that a creature navigating around its environment uses to build up a perception of space. We illustrate this point with preliminary evidence that perception of spatiotemporal patterns on the human skin shows some of the same features as spatial navigation in animals. We argue (a) that the receptor array defines a ‘tactile field’, (b) that this field exists in a minimal form in ‘skin space’, logically prior to any transformation into bodily or external spatial coordinates, and (c) that this field supports tactile perception without integration of concurrent proprioceptive or motor information. The basic cognitive elements of space perception may begin at lower levels of neural and perceptual organisation than previously thought. (shrink)

This chapter distinguishes between two kinds of ordinary multisensory experience that go beyond mere co-consciousness of features (e.g., the experience that results from concurrently hearing a sound in the hallway and seeing the cup on the table). In one case, a sensory experience in one modality creates a perceptual demonstrative to whose referent qualities are attributed in another sensory modality. For example, when you hear someone speak, auditory experience attributes audible qualities to a seen event, a person’s speaking motions. (...) The second kind of multisensory experience attributes features experienced in several sensory modalities to one and the same object via a process of amodal perceptual integration, i.e., integration that occurs separately from processing within the individual sensory modalities. Multisensory experiences arising from holding and seeing a tomato or from seeing the Indian curry boil and smelling it are examples of the second kind of multisensory experience. At the end of the chapter we look at synesthesia, a kind of atypical multisensory experience, and argue that one version of this phenomenon may be able to shed light on the neural mechanism underlying amodally integrated multisensory experience. (shrink)

The reach-to-grasp movement is ordinarily performed in everyday living activities and it represents a key behavior that allows humans to interact with their environment. Remarkably, it serves as an experimental test case for probing the multisensory architecture of goal-oriented actions. This review focuses on experimental evidence that enhances or modifies how we might conceptualize the “multisensory” substrates of prehension. We will review evidence suggesting that how reach-to-grasp movements are planned and executed is influenced by information coming from different (...) sensory modalities such as vision, proprioception, audition, taste, and olfaction. The review closes with some considerations about the predominant role of the multisensory constituents in shaping prehensile behavior and how this might be important for future research developments, especially in the rehabilitative domain. (shrink)

A period of exposure to trains of simultaneous but spatially offset auditory and visual stimuli can induce a temporary shift in the perception of sound location. This phenomenon, known as the 'ventriloquist aftereffect', reflects a realignment of auditory and visual spatial representations such that they approach perceptual alignment despite their physical spatial discordance. Such dynamic changes to sensory representations are likely to underlie the brain's ability to accommodate inter-sensory discordance produced by sensory errors (particularly in sound localization) and variability in (...) sensory transduction. It is currently unknown, however, whether these plastic changes induced by adaptation to spatially disparate inputs occurs automatically or whether they are dependent on selectively attending to the visual or auditory stimuli. Here, we demonstrate that robust auditory spatial aftereffects can be induced even in the presence of a competing visual stimulus. Importantly, we found that when attention is directed to the competing stimuli, the pattern of aftereffects is altered. These results indicate that attention can modulate the ventriloquist aftereffect. © 2010 Elsevier B.V. (shrink)

The last decade has seen great progress in the study of the nature of crossmodal links in exogenous and endogenous spatial attention . Exogenous spatial cuing studies of human crossmodal attention and multisensory integration. In C. Spence, & J. Driver , Crossmodal space and crossmodal attention . Oxford, UK: Oxford University Press.], for a recent review). A growing body of research now highlights the existence of robust crossmodal links between auditory, visual, and tactile spatial attention. However, until recently, (...) studies of exogenous and endogenous attention have proceeded relatively independently. In daily life, however, these two forms of attentional orienting continuously compete for the control of our attentional resources, and ultimately, our awareness. It is therefore critical to try and understand how exogenous and endogenous attention interact in both the unimodal context of the laboratory and the multisensory contexts that are more representative of everyday life. To date, progress in understanding the interaction between these two forms of orienting has primarily come from unimodal studies of visual attention. We therefore start by summarizing what has been learned from this large body of empirical research, before going on to review more recent studies that have started to investigate the interaction between endogenous and exogenous orienting in a multisensory setting. We also discuss the evidence suggesting that exogenous spatial orienting is not truly automatic, at least when assessed in a crossmodal context. Several possible models describing the interaction between endogenous and exogenous orienting are outlined and then evaluated in terms of the extant data. (shrink)

The posterior parietal cortex (PPC) is the most likely site where egocentric spatial relationships are represented in the brain. PPC cells receive visual, auditory, somaesthetic, and vestibular sensory inputs; oculomotor, head, limb, and body motor signals; and strong motivational projections from the limbic system. Their discharge increases not only when an animal moves towards a sensory target, but also when it directs its attention to it. PPC lesions have the opposite effect: sensory inattention and neglect. The PPC does not (...) seem to contain a “map” of the location of objects in space but a distributed neural network for transforming one set of sensory vectors into other sensory reference frames or into various motor coordinate systems. Which set of transformation rules is used probably depends on attention, which selectively enhances the synapses needed for making a particular sensory comparison or aiming a particular movement. (shrink)

In consideration of the spatial structures of sensory experiences, an ‘Externality Thesis’ is commonly proposed, according to which awareness of sensory boundaries is also an awareness of the presence of a space beyond these boundaries. The paper evaluates the Externality Thesis in the context of vision and touch. More specifically, relying on mereotopological theories, it is shown that the notion of spatial boundaries is ambiguous as it encompasses various distinct ways in which entities may be connected by a boundary. (...) It is argued that only some of these ways are able to support the Externality Thesis. In particular, it is claimed that while bodily boundaries of which one is aware in tactile sensations are such that the awareness of them provides support for the Externality Thesis, the analogous claim is not true about the boundaries of the visual field. (shrink)

In consideration of the spatial structures of sensory experiences, an ‘Externality Thesis’ is commonly proposed, according to which awareness of sensory boundaries is also an awareness of the presence of a space beyond these boundaries. The paper evaluates the Externality Thesis in the context of vision and touch. More specifically, relying on mereotopological theories, it is shown that the notion of spatial boundaries is ambiguous as it encompasses various distinct ways in which entities may be connected by a boundary. (...) It is argued that only some of these ways are able to support the Externality Thesis. In particular, it is claimed that while bodily boundaries of which one is aware in tactile sensations are such that the awareness of them provides support for the Externality Thesis, the analogous claim is not true about the boundaries of the visual field. (shrink)

This volume is about the many ways we perceive. Contributors explore the nature of the individual senses, how and what they tell us about the world, and how they interrelate. They consider how the senses extract perceptual content from receptoral information. They consider what kinds of objects we perceive and whether multiple senses ever perceive a single event. They consider how many senses we have, what makes one sense distinct from another, and whether and why distinguishing senses may be useful. (...) They consider the extent to which the senses act in concert, rather than as discrete modalities, and whether this influence is epistemically pernicious, neutral, or beneficial. They explore both the familiar five sense modalities and other candidate sense modalities, including a sense for flavour, inner senses (e.g., a sense for pain), and various senses that may result from using sensory substitution devices. Contributors often represent competing views, and the methods they deploy sometimes differ from one essay to the next—with some drawing upon the sciences and engineering and others relying upon conceptual analysis. All contributors agree, nonetheless, that traditional theorizing about the senses is hampered by a neglect of the senses other than vision, and by the misconception that vision is a passive receptacle for an image thrown by a lens. And many contributors believe that it is unduly restrictive to think of perception as a collation of contents provided by individual sense modalities; they think that to understand perception properly, we need to build into our accounts the idea that the senses work together. The ambition of the volume is to begin to develop better paradigms for understanding the senses, and thereby to move toward a better understanding of perception. (shrink)

This project uses machine learning and computer vision techniques and a novel interactive visualization tool to provide street-level characterization of urban spaces such as safety and maintenance in urban neighborhoods. This is achieved by collecting and annotating street-view images, extracting objective metrics through computer vision techniques, and using crowdsourcing to statistically model the perception of subjective metrics such as safety and maintenance. For modeling human perception and scaling it up with a predictive algorithm, we evaluate perception predictions across two points (...) in time separated by economic changes in the urban core of Raleigh, North Carolina, in the aftermath of the 2008 Great Recession. We hypothesize specific socioeconomic processes can be substantially reflected in the built environment of cities and, thus, render themselves visible at the street level. This paper describes the process of incorporating subjective visual ratings across two datasets of temporally separated street-view images, an algorithm, and a visualization tool. This work serves as a case study for utilizing AI and visualization techniques in a richer characterization of urban spaces that includes both objective metrics such as income (that operates at a broader scale) and subjective metrics such as perception of individuals (that operates at a narrower scale at specific locations). We outline an interdisciplinary methodology to test this hypothesis in streetscape data from Raleigh, NC, from 2008 to 2020. We describe the results of training algorithms that utilized image features with crowdsourced human perception ratings. We provide a comparison of the results with income data. The analysis and interpretation of this comparison provide insight into the challenges and opportunities for using AI technology in characterizing changes in urban environments. One challenge is the ability of human domain experts to interpret the output of algorithms through manipulation and to integrate these results into their workflow. This is addressed with a novel interface designed for interactive analysis and visualization. We conclude with a discussion of some of the benefits and limitations of integrating AI models in the human expert’s decision-making process in the presence of both subjective and objective metrics. (shrink)

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is the purpose of multisensory integration?

If a man born blind were to gain his sight in later life would he be able to identify the objects he saw around him? Would he recognise a cube and a globe on the basis of his earlier tactile experiences alone? This was William Molyneux's famous question to John Locke and it was much discussed by English and French empiricists in the eighteenth century as part of the controversy over innatism and abstract ideas. Dr Morgan examines the whole history (...) of this debate: Locke's own (negative) answer to the question, the contributions of Berkeley, Condillac, Diderot and Voltaire and the factual accounts of early cataract operations and modern laboratory studies. He shows how this debate is involved in the development and eventual separation of philosophy and experimental psychology after the eighteenth century and considers why the original question is effectively still unanswered. This is one problem-area with its intricate cluster of connected conceptual and technical difficulties which suggests the need for some reunion or at least collaboration between the two subjects. (shrink)

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Is language processing a special kind of multisensory integration?

This is the editors' Introduction to a special issue of the journal, Multisensory Research. European philosophers of the modern period found multisensory perception to be impossible because they thought that perceptual ideas are defined by how they are experienced. Under this conception, the individual modalities are determinables of ideas—just as colour is a determinable that embraces red and blue, so also the visual is a determinable that embraces colour and (visually experienced) shape. Since no idea is experienced as, (...) for example, both visual and auditory, there can be no such thing as audiovisual perception. This conception of modality is not directly contested, but a variety of perceptual phenomena are listed that could raise interesting questions if treated as multimodal in origin. (shrink)

Object perception deploys a suite of perceptual capacities that constrains attention, guides reidentification, subserves recognition, and anchors demonstrative thought. Objects for perception—perceptual objects—are the targets of such capacities. Characterizing perceptual objects for multisensory perception faces two puzzles. First is the diversity of objects across sensory modalities. Second is the unity of multisensory perceptual objects. This paper resolves the puzzles. Objects for perception are structured mereologically complex individuals. Perceptual objects are items that bear perceptible features and have perceptible parts (...) arranged to form a unified whole. This circumscribes the targets of object perception where neutral talk concerning intentional objects and objects of perception cannot. Nonetheless, it is more permissive than identifying perceptual objects with material bodies, which excludes too much and leans too heavily on vision. The account thus is tailored to capture the role of objects in perception beyond vision: tactual, auditory, and olfactory objects are individuals with differing structures. Its flexibility also enables the account to accommodate shared objects for multisensory perception: multisensory perceptual objects are mereologically complex individuals with hybrid structure. For instance, one can bimodally perceive a common whole with parts accessible to one but not both modalities. Each sense provides a partial perspective on a complex whole that is perceptible through the coordinated use of multiple senses. Understanding perceptual objects as structured mereologically complex individuals thus provides a theoretically useful notion of an object for multisensory perception that resolves the puzzles of diversity and of unity. (shrink)

This chapter examines how our sense modalities interact in the perception of persistence. The chapter concentrates on two questions. The first concerns perceptual processing—do perceptual computations of object persistence ever integrate and compute over representations from more than one modality? It argues that this question should be answered affirmatively. The second question concerns perceptual experience—do experiences of object persistence ever exhibit a constitutively multisensory phenomenal character, or is the phenomenology of object persistence always uniquely associated with just one modality? (...) The chapter argues that the available evidence underdetermines the answer to this question, but suggests ways it might be empirically resolved. (shrink)

In the social world, multiple sensory channels are used concurrently to facilitate communication. Among human and nonhuman pri- mates, faces and voices are the primary means of transmitting social signals (Adolphs, 2003; Ghazanfar and Santos, 2004). Primates recognize the correspondence between species-specific facial and vocal expressions (Massaro, 1998; Ghazanfar and Logothetis, 2003; Izumi and Kojima, 2004), and these visual and auditory channels can be integrated into unified percepts to enhance detection and discrimination. Where and how such communication signals are integrated (...) at the neural level are poorly understood. In particular, it is unclear what role “unimodal” sensory areas, such as the auditory cortex, may play. We recorded local field potential activity, the signal that best correlates with human imaging and event-related potential signals, in both the core and lateral belt regions of the auditory cortex in awake behaving rhesus monkeys while they viewed vocalizing conspecifics. We demonstrate unequivocally that the primate auditory cortex integrates facial and vocal signals through enhancement and suppression of field potentials in both the core and lateral belt regions. The majority of these multisensory responses were specific to face/voice integration, and the lateral belt region shows a greater frequency of multisensory integration than the core region. These multisensory processes in the auditory cortex likely occur via recip- rocal interactions with the superior temporal sulcus. (shrink)

I discuss the relation between egocentric spatial representation and the capacity for bodily activity, with specific reference to Merleau-Ponty.

This chapter critically assesses recent arguments that acquiring the ability to categorize an object as belonging to a certain high-level kind can cause the relevant kind property to be represented in visual phenomenal content. The first two arguments, developed respectively by Susanna Siegel (2010) and Tim Bayne (2009), employ an essentially phenomenological methodology. The third argument, developed by William Fish (2013), by contrast, is supported by an array of psychophysical and neuroscientific findings. I argue that while none of these arguments (...) ultimately proves successful, there is a substantial body of empirical evidence that information originating outside the visual system can nonetheless modulate the way an object’s low-level attributes visually appear. Visual phenomenal content, I show, is not only significantly influenced by crossmodal interactions between vision and other exteroceptive senses such as touch and audition, but also by interactions between vision and non-perceptual systems involved in motor planning and construction of the proprioceptive body-image. (shrink)