A Multimodal Augmented Reality System for Interactive Exploration of Digital Cultural Heritage

Manolya Kavakli-Thorne, professeur à Macquarie University de Sydney
14 Novembre 2014
This presentation reviews a number of Virtual Reality (VR) applications developed at the VR Lab, Department of Computing, Macquarie University to discuss the potential projects for international collaboration. The goal is to summarise our findings related to a recently completed Discovery grant sponsored by the Australian Research Council “A Gesture-Based Interface for Designing in Virtual Reality” and discuss where the biggest challenges remain for the development of multimodal ubiquitous systems in future. The interdisciplinary nature of Human Computer Interaction brings together researchers from a number of fields, such as Computing, Engineering, Architecture, Psychology, and Media. This requires involvement of large research networks as well as the development of an integrated approach to system design. Integrating motion tracking with speech recognition and using augmented reality (AR) technology, today’s mobile devices such as smart tablets and smart phones can allow natural user interactions with a 3D space and a virtual built environment and have a wider outreach than presentations in scientific research labs, simulation centers, and museums. The issue is how to design ubiquitous systems to turn the streets we live in to an open museum with no walls. The expected outcomes of this presentation are strategic partnerships with French Institutes to work on a proposal for a Multimodal Augmented Reality System for Interactive Exploration of Digital Cultural Heritage, as well as joint supervision of PhD students in the domains of common interests. Overall aim of this project is to explore citizens’ interaction with digital cultural heritage sites using the GPS locations provided by a smart tablet and to draw cultural comparisons between traditional and augmented interaction and communication processes. The expected outcomes of this project is a cross-cultural cognitive framework and a system model to support citizens’ communication with digital cultural heritage sites utilizing speech and gesture recognition technologies. The model will be implemented and tested in a number of cultural heritage locations in Europe, Asia, and Australia. The project is closely aligned with The EU Culture programme launched in 2013 and fits the description of smart cities as seen in the Appendix.

Appendix: EU Culture Programme


Dr Manolya Kavakli-Thorne.
Associate Professor Manolya Kavakli is currently the Director of the Postgraduate Coursework Program at the Department of Computing, Macquarie University. She is an active researcher working on Human Computer Interaction (HCI) in the last 25 years with 133 refereed papers (775 citations). She has been the recipient of 10 awards and more than 30 grants from a number of scientific international organisations including the Australian Research Council, the Scientific and Technical Research Council of Turkey, Turkish Republic State Planning Organization. Her research has been profiled in international media 32 times. Dr Kavakli supervised 7 postdoctoral fellows and 47 potsgraduate (11 PhD, 2 MSc, 7 Honours, 18 MIT and 9 French MEng internship) students in HCI. At present, she has been supervising 7 postgraduate students in this domain.
Dr Kavakli started working on HCI in design, following her graduation from the Faculty of Architecture, Istanbul Technical University, in 1987. Her earlier research studies focused on the simulation of designers’ behaviour and interaction with computer aided design systems. She developed two knowledge based systems for the solution of specific design problems to gain her M.Sc. and Ph.D. degrees in 1990 and 1995 from Istanbul Technical University. In her early postdoctoral studies, she focused on computer graphics and sketch recognition to be able to design more effective and intuitive software systems for architects and engineers. She was awarded a NATO Science Fellowship in 1996 with her postdoctoral research project titled “An AI application for the transformation of 2D sketch to 3D geometric model” and started working on the analysis of hand drawn images. She worked at the Colour and Imaging Institute (former Design Research Centre), University of Derby, UK for a year. In 1998 she received a Postdoctoral Fellowship from the University of Sydney, Australia and worked on the differences in cognitive processing between novice and expert designers at the Key Centre of Design Computing and Cognition for 1.5 years.
Until 1999 she was an Associate Professor in Design Science and Methodology at the Faculty of Architecture, Istanbul Technical University, In 2000, she started lecturing in IT at the School of Information Technology, Charles Sturt University. In 2002, she took an active role in establishing the first degree in Computer Science (Games Technology) and became its Acting Course Coordinator. By that time, her research in HCI started having a strong focus on the creation of virtual habitats and novel interaction methods with virtual objects and avatars in computer games and training simulations. In 2003, she became a Senior Lecturer at Department of Computing, Macquarie University, and established a Virtual Reality Lab, as well as VISOR (Virtual and Interactive Simulations of Reality) Research Group. She is currently continuing her research on HCI using virtual reality and motion tracking technologies with a strong focus on the development of training simulations. In 2013 and 2014 Dr Kavakli received two 2-month appointments for Research Professorships at the Virtual Immersion Research Group, Image Institute – Laboratory Le2i, Ecole Nationale Superieure d’Arts et Metiers ParisTech, Chalon Sur Saone, France. Joining CEPET (Centre for Elite Performance, Expertise, and Training) as an associate member, she took an active role in building a Simulation Hub together with her colleagues from the Department of Psychology.