The evolution of technology, communication and cognition towards the future of human-computer interaction
Istituto Auxologico Italiano
“ Tor Vergata” University of Rome
Telecom Italia Learning Services
Technical University of Valencia
Ambient Intelligence implies a seamless environment of computing, advanced networking technology and specific interfaces. It is aware of the specific characteristics of human presence and personalities, takes care of needs and is capable of responding intelligently to spoken or gestured indications of desire, and even can engage in intelligent dialogue.
Ambient Intelligence should also be unobtrusive, often invisible: everywhere and yet in our consciousness – nowhere unless we need it. Interaction should be relaxing and enjoyable for the citizen, and not involve a steep learning curve.
Honey, is that YOU or am I experiencing virtual reality?
Certainly adults who are reasonably awake can distinguish vision from television, reality from virtual reality and the artificial world of magic & special effects. Twenty-first century cognitive neuroscientists, with the help of fMRI and other so-called “non-intrusive” brain imaging technology, can peep into our brain to find out what neural activities are associated with perceiving and moving into a real or imagined environment, and as I read about Presence Research and the R&D efforts of artificial intelligence to recreate virtual environments I am reminded of the New Yorker cartoon of 15 years ago where the subject cannot tell the difference between a funny-looking loved one or a believable VR clone.
Over centuries the techniques that artists use to fool our eyes and minds into believing what we see have evolved. Some visual illusion and gestalt techniques stay the same, and rhetoric has a long academic tradition. What is certainly true is that today’s avid consumer of video games and 3D graphic software visualization is a far more discerning and demanding consumer of advanced media technology. Be it for security and safety, for work, fun or everyday life, advanced communication applications have to keep up with the rapid pace of technology innovation, and this is only possible through investment in state-of-the-art scientific research and product development.
The metaphor of Ambient Intelligence (AmI) tries to picture a vision of the future where all of us will be surrounded by intelligent” electronic environments, and this ambient has claims to being sensitive and responsive to our needs. A multitude of sensors and actuators are already embedded in very-small or very large information and communication technologies, and it is only a question of time when better use can be gained from these complex (yet still extraordinarily primitive) technology systems. Pleasant or not, for the lonely human, researchers predict that AmI will be densely populated by IT gadgets and systems with potentially powerful NBIC capabilities (nano- bio- information and communication technology).
Ambient Intelligence without invasion of privacy represents a long-term vision for the EU Information Society Technologies Research programme, with the aim of bringing together researchers across multiple disciplines: computer science, social sciences, physics, biology, engineering, design, architecture & philosophy to name a few. A strong multi-disciplinary and collaborative approach is a key requirement for large scale technology innovation and the development of effective applications.
So far most of the books and papers related to AmI, focus their analysis on the technology potential only. An important feature of this volume is the link between the technology - through the concepts of ubiquitous computing and intelligent interface - and the human experience of interacting in the world - through a neuro-psychological vision centered on the concept of “presence”.
Presence, the sense of being there, is the experience of projecting one's mind through media to other places, people and designed environments. Appropriate presence technologies combine new and old media (like books and story-telling) to create an illusion of «non-mediation» - the closest possible approximation to a sense of physical presence, when physical presence there may be none.
The combination of recent discoveries in cognitive neuroscience - that make it possible to acquire a better understanding of the human aspects of presence, and the breakthroughs at the level of the enabling technologies make it increasingly possible to build novel systems based on this understanding.
Presence comprises not only the sense of being there but also the experience and capability of interacting there with real and virtual physical objects. Early presence researchers investigated tele-operation. Today man-machine interfaces have evolved considerably, and the inherent capacity of presence technologies is to support multiple users’ engagement and bi directionality of exchange: the objectives and communication approach are thus different to control theory. Depending on the degree of presence desired, either high or low-fidelity may be optimal. Mixed reality systems deploy both good quality and high-fidelity representation of the simulated worlds; or on the other hand cartoonification and simplicity of symbolic representation when it’s not a rich experience but an instantaneous response that is required.
In this community, researchers are asking basic scientific or philosophical questions about how human beings distinguish object:subject relationships, and how they have a sense of their own body as a measure. The answers to the question of how the eye and brain sees the outside world in a stable and coherent interpretation (mind, representation, volition, engagement) in relation to representing that world outside our heads, or how haptics and multimodality shapes sensory experiences for us and other creatures are not known in advance.
Yet early results of experiments with presence suggest design choices for new computer-enriched environments that give promise of enhanced features (eg extending human capacity to memorize, or imagine, or redefine spacial positioning, navigation, and adapt to novel situations).
Mental and motivational states are part of the study of human presence, which adds controversy to complexity, since the methods or measurement metrics for presence are not established and the validity of predicting effects is scant if reliable or longitudinal data does not exist.
A better understanding of human cognitive, and affective capacities will very likely lead to new applications/products/services, spanning a whole range of potential industries. There are a number of application areas where a good sense of presence is needed, and in the near future xReality plus broadband access and connectivity will be exploited for successful substitution to actually being there yourself. Representative examples could include specific areas (tele-operation, assembly, maintenance and repairing, new working environments, etc.), biomedicine and neurosciences (assistive surgical operations, neurological rehabilitation, human perceptual augmentation), education & training, surveillance, real time interactive gaming and entertainment, archiving, new communication standards, etc).
The AmI applications are the ones that may benefit more from the presence research since more trust tends to be placed in things that can be recognizably influenced. As Anton Andrews suggests:
“As tailored services and solutions become the norm, people will also demand flexibility in the amount of control and personal information that they hand-over in return for the convenience gained” (Empowering Participation, in The Vision Book p 21, forthcoming).
Today, most AR/VR/MR applications are mainly dedicated to simulated visual phenomena. In the future, x-reality applications will appeal to multisensory modalities and simulate more natural phenomena, following a user-centered and user-in-the-loop approach. The user will have full access to various services from the augmented Presence environment. The services will be delivering results in real-time and the environment will seem fully interactive and reactive, enabling those users who are computer-literate to perceive and (inter)act with applications and services to gain new experiences and better results.
Presence research is still in its infancy. The research challenge is to design develop and test systems that match human cognitive and affective capacities and re-create the different experiences of presence and interaction in mixed reality environments. Research should focus on :
- Understanding different forms of presence, encompassing aspects of perception, cognition, interaction, emotions and affect. Techniques for measuring presence need to be developed taking into account insights from physio- neuro- cognitive and social sciences.
- The ethical aspects and the investigation of possible long-term consequences of using presence technologies need to be investigated.
- Designing and developing essential building blocks that capture the salient aspects of presence and interaction based on the understanding of human presence. These blocks should exploit relevant cutting edge software and hardware technologies (e.g. real time display and high fidelity rendering, 3D representation and compression, real-time tracking and capture, light control, haptic interfaces, 3D audio, wearable and sensor technology, biosensors and biosignals, etc.).
- Developing novel systems, able to generate or support different levels and types of presence and interaction in a multitude of situations. The research focus should be on open system architectures for integrating the above building blocks, with open APIs and source authoring tools for programming presence and for designing novel interaction paradigms.
A number of challenging scenarios are envisioned as tests of whether presence technologies can make a real difference, for example:
- Persistent hybrid communities: constructing large-scale virtual/mixed communities that respond in real-time and exhibit effects of memory and behavioral persistence while evolving according to their intrinsic social dynamics.
- Presence for conflict resolution, allowing people to be immersed and experience situations of conflict or co-operation. By fostering communication and mutual understanding between different parties these presence environments should ultimately be empathy-inducing.
- Mobile mixed reality presence environments: moving freely and interacting in real/augmented populated surroundings through natural and/or augmented mediated tools.
- Personalized learning and training environments, stimulating a combination of imaginary and physical actions and emotions through appropriate sets of embedded non-verbal and multisensory cues for skill acquisition and learning.
There will be other scenarios more imaginative and beyond today’s state of the art. The lack of interdisciplinary research has in the past brought well known failures of over-engineered technologies which maximized pixel-throughput but almost ignored how human perception, cognition, co-operation and communication work. Presence research has evolved from the original focus on “being there” and the effort to reproduce reality with ever increasing realism -the “perceptual illusion of non-mediation” - to include the bias and context of subjective experience.
This future of presence technologies depend on the “acceptance” factor and the perceived potential for interaction that they afford.
The chapters in this book bring the concepts of presence and AmI together so that the readers can decide for themselves what is the current status of research in the field, and what these future and emergent technologies can do for them.
Disclaimer: the views expressed are those of the author only and should not be construed to reflect or represent the position of the European Commission.
Loretta Anania, Ph.D.
IST - Future and Emerging Technologies Programme
Start of the page
Making AmI real is no easy task: as it commonly takes place with a new technology, soon after high-flying visions we are demonstrated with the first pieces of hardware for the intelligent environment. However, making a door knob able to compute and communicate does not make it intelligent: the key (and challenge) to really adding wit to the environment lies in the way how the system learns and keeps up to date with the needs of the user by itself. A thinking machine, you might conclude – not quite but close: if you rely on the intelligent environment you expect it to operate correctly every time without tedious training or updates and management. You might be willing to do it once but not constantly even in the case of frequent changes of objects, inhabitants or preferences in the environment. A learning machine, I'll say.
As underlined by the IST Advisory Group (ISTAG) in 2001, the concept of Ambient Intelligence (AmI) provides a wide-ranging vision on how the Information Society will develop. In fact, the vision of AmI can be considered a landmark for giving direction to ITC research over the coming five-ten years.
The emphasis of AmI is on greater user-friendliness, more efficient services support, user-empowerment, and support for human interactions. Focusing on these features, Europe can exploit its technology advantage in areas such as mobile communications, portable devices, systems integration, embedded computing and intelligent systems design. However, the success of this vision will be based on the social dimension of innovation, the ability as well as the willingness of society to use, absorb or adapt to technological opportunities.
The goal of this volume is to assess the technologies and processes that are behind the AmI vision to help the development of state-of-the-art applications. More in detail, this volume aims at supporting researchers and scientists, interested in the understanding and exploiting the potential of AmI.
For the complexity of this topic, we have put a great deal of effort in the definition of the structure of the book and in the sequence of the contributions, so that those in search of a specific reading path will be rewarded. To this end we have divided the book in four main Sections comprising 15 chapters overall:
1. An Introduction to Ambient Intelligence
2. The Architecture Of Ambient Intelligence:
Towards Pervasive Computing And Context Awareness
3. The Interface Of Ambient Intelligence:
Towards Natural User-System Interaction
4. Ambient Intelligence In Practice: Future Perspectives And Applications
Each chapter begins with a brief abstract, helping the readers in identifying the relationships among its sections.
Section I – An Introduction to Ambient Intelligence
In Chapter 1, Alcañiz and Rey discuss the three recent technologies Ubiquitous Computing, Ubiquitous Communication and Intelligent User Interfaces – that are at the core of the AmI vision.
Ubiquitous Computing means integration of microprocessors into everyday objects like furniture, clothing, white goods, toys, even paint. Ubiquitous Communication enables these objects to communicate each other and the user by means of ad-hoc and wireless networking. An Intelligent User Interface enables the inhabitants of the AmI environment to control and interact with the environment in a natural (voice, gestures) and personalised way (preferences, context).
Chapter 2, by Riva, shifts the focus of the analysis from technology to human factors. In particular, chapter 2 outlines a psychological framework for the concept of Ambient Intelligence (AmI), centered on three concepts: action, situation and presence. Using this framework it is provided a psychological definition of AmI, based on the experience of the user: AmI is the effective and transparent support to the activity of the subjects through the use of information and communication technologies.
In Chapter 3 Gaggioli further develops this vision, by presenting a framework to evaluate user’s experience in AmI systems. Main feature of the framework is the role of attention in actively selecting information from the digital environment. Being a limited resource, attention forces the individual to focus on a restricted number of environmental or internal stimuli per time unit. This approach allows the author to outline a practical method to assess optimal user experience in AmI systems.
Chapter 4, can be considered an Appendix to the first three chapters. Prepared by the IST Advisory Group (ISTAG), the chapter describes AmI as an “emerging property” of intelligent interfaces supported by computing and networking technology, and embedded in everyday objects such as furniture, clothes, vehicles, roads and smart materials. The chapter underlines the role of “future scenario building” as a critical process to support the development of effective AmI applications.
Session II - The Architecture Of Ambient Intelligence: Towards Pervasive Computing And Context Awareness
5 by Cortese, Lunghi and Davide describe Physical
Service Environments, which combine pervasive computing and
context awareness to deliver enhanced, highly usable services
to mobile users. At the communications layer, this kind of
pervasive computing will require adaptive access technologies,
ad-hoc networking and architectures to achieve seamless interoperability
among wireless technologies. As far as concerns middleware,
innovative architectures for pervasive computing will enable
mobile clients, sensors and application servers to interact
in the physical service environment. But context-aware service
environments also require a well designed, standard architecture
for the management of context information.
presents the concept of implicit human computer interaction (iHCI) that
takes the users context into account when creating new user interfaces
for ambient intelligence. Beyond the model, the chapter provides some
examples, and describes some possible application areas.
this section, which tries to outline possible future applications of
AmI, Bettiol and Campi (Chapter
12) tries to understand the relation between an individual
and an ambient intelligent environment. In the chapter, three practical
cases are described using an user-centered approach. The considered
users are the residents of popular housing projects, the students of
a domotica centre of education, and the last generation vehicles maintenance
Computing in Tangibile: Using Artifacts as Components of Ambient
The Simplicity Project: Managing Complexity in a Diverse ICT World,
Bodyarchitecture: The Evolution of Interface towards Ambient Intelligence,
Sociability versus Individualism in the Ageing Society,