Research aims
The goal has been to develop an intuitive method enables the user to access information while navigating a 3D model, just by looking at it. Moreover we have also applied the method in an outdoor mixed reality setting, using the GPS to provide the location; the system supplied the user with the information related to the portion of the real world being looked at. The first prototype was successfully implemented and we were able to retrieve information just by looking around.
Actually we extend the system on mobile platform.
The core of the method is: using sight to actively query and insert information. This realizes one of the most important goals in our approach: to ease everybody’s access to information.
The information are naturally embedded in 3D spatial contexts, and the most natural way of behaving in such a context is to just look around. By using sight we naturally focus on something (the origin of the name of the method, “I see”); therefore, we propose to use this action as a common language which people can use to understand, query and insert information.
Introduction
Cultural heritage is an interdisciplinary field that draws together several different professions.
Information is gained from different sources and in varying formats. Furthermore, the relationship between the conservation managers, who are often unfamiliar with current documentation techniques, and the providers of the information, who tend to be highly technical practitioners without expertise in cultural heritage, is not easy to handle.
Many objects involved in Cultural Heritage investigations (artworks, sculptures, artefacts, etc.), have a strong 3D component and cannot be easily represented with 2D maps. To keep in account these issues, this research focused on 3D representations. It is clear that the problem of moving from 2D to 3D is complex and it is not conceivable to solve all these issues. Nevertheless, using a 3D environment allows for a closer adherence to the real world (preserving location related data) and permits to respect the spatial relationships among different components (Pillay et al., 2007; Pittarello, 2004).
Also the 2D GIS (Geographic Information System) is beginning to be used in Cultural Assets
application. The problem is that an integrated and high level management of information is missing.
An increasing number of applications need more advanced tools for representing and analyzing the 3D world (Pecchioli & Leitner, 2007).
The method
We started considering what are the requirements for an adaptive and intuitive interface to access information that is connected with a spatial region, but is not reducible to simple numbers (texts, tables,...), information that in GIS is called geo-coded information.
In ISEE, information is associated with regions of the 3D space, which we call Information
Zones (IZ). The Information Zones can not coincide with a 3D object, represented in the model, but they might be just a part of it, or they may include many objects at the same time.
To implement this idea, it is necessary to link each piece of information with a region of the space, find a precise definition of the VZ and define a ranking to find the relevant information.
As this ranking can only depend on the VZ and the location of the information (i.e. on spatial
relationships), we call it spatial relevance.
This choice of information association can have an important advantage as: it is not necessary to subdivide 3D models into logical sub-elements to attach information. One can directly use meshes generated by 3D scanners (or any automatically generated 3D model), without needing any particular post-processing in order to identify sub-portions of the models. The advantages are presented from the indirect connection with 3D objects: it is not always easily possible to find “exact information” related only to a given object; queries can be less precise. Moreover the replacement the model with a new version can make the transfer of information tricky (e.g. if the difference between the new and the old version is larger than the size of the information zone).
Clearly, if information is so strongly related to 3D zones, the definition of a 3D zone is crucial.
For the first issue we shall assume that zones can be approximated. How to represent these approximate zones? Unfortunately, defining 3D zones interactively is not so intuitive, as one has many degrees of freedom. Our first choice was to use spheres, as they are the simplest 3D objects and quite easy to define interactively (just two parameters are needed, radius and centre).
However a sphere, like any other solid object, has a sharp boundary: a point is either inside the sphere or outside it. The zone we actually want to describe is not likely to coincide exactly with the sphere, as the sphere is just meant to represent an approximation of it. Thus especially at the boundary this will be inexact. So defining a smooth transition at the zone boundaries would be preferable. This can be achieved by associating a quantity, that we will call “concentration of information”, to each point of the space, representing “how much” information is contained at that point in the zone.
We decided to use 3D gaussians: this is a function which assigns a value to each point of the space (it can be seen as a fog, more dense in the centre, and less dense on the periphery) which can be used to adequately describe the concentration of information (Figure 1).
Fig. 1 Plot of normalized gaussians with width= 0.5, 1.0 ,2.0 as function of the distance from the centre.
The portion of the 3D model which is being looked at in the interactive 3D Viewer (View Zone - VZ) is also approximated with a normalized 3D gaussian, so that View Zones and Information Zones are described in a similar way. The symmetric treatment of VZs and IZs allow us to use the interactive 3D viewer to visually insert the IZ of a piece of information (authoring), or to jump immediately to the view related to some information (retrieval) (Pecchioli & Mohamed, 2008b).
An innovative aspect of this approach is represented by our definition of the spatial relevance of information. What we want to develop is a continuous measure of the relevance depending on the spatial relationships between VZ and IZ (Figure 2).
Fig. 2 Plot of the overlap between two normalized gaussians of width w1 and w2
Intuitively, the relevance of information should be maximal when its Information Zone (IZ) coincides with the Viewed Zone (VZ), decreasing when they are far apart. Moreover, as information is always associated with extended zones rather than with points, we can also use the size of a zone to decide if it is relevant. An IZ that has a size comparable to the current VZ is probably more interesting than an IZ that has a size (and thus a level of spatial detail(1) ) very different from the current VZ. Using the level of detail as ISEE does (and not only the distance) of the information is a big advantage with high densities of data, because its usage provides an extra means to filter the information reduces the amount of information retrieved (Figure 3).
Fig. 3 Third person view of VZ and IZ from the side. The VZ is represented as a green sphere, the IZ as red spheres, and the view cone is gray
Given two normalized gaussians:
the overlap between them is defined as:
gets smaller if the IZ is far away or very differently sized.
The ISEE method is therefore an improvement on the insufficient filtering performance of distance only methods. These methods typically use the distance from the user not from where he is looking at as extra simplification.
In an interactive 3D model the level of zoom depends on the distance of the object looked at, and in the same scene one can have different levels of detail.
It is clear that in this case a continuos approach that does not classify arbitrarily the information in a given predefined set of discrete level of detail is advantageous.
3D Visualization
The 3D is used as mean to locate the information, thus it is important that it is interactive, but can be a synthetic model. Some pieces of information are about details, other about larger regions.
Classifying the information based on the region is thus a natural idea. Thinking about it
we had the idea of using a simple action of seeing as main mean to define the context. Using this allows a very natural interface which can be easily grasped and explained.
Using a 3D environment allows for a closer adherence to the real world (preserving location related data) and permits to respect the spatial relationships among different components (Carrozzino et al., 2009).
In the present work we will focus in particular on the restoration field, where often the only unifying entity is the object which the information is related to, as several types and kind of documents need are present. Furthermore the objects studied are not so easily reduced to a 2D representation without loss. In this article we will look for example to mural paintings on a vaulted surface.
The crypt in St.Servatius in Quedlinburg
Quedlinburg represents an exceptional example of a medieval European town with high quality of the timber framed buildings, and it is part of the World Heritage List (UNESCO). The Collegiate Church of St. Servatius is one of the masterpieces of Romanesque architecture. It is known not only for the tomb of the emperor Henry I, but also for crypt cycle of wall paintings from the 12th century (Figure 4-5-6).
Fig. 4 Aerial photo of Quedlinburg
Fig. 5 The main nave in St.Servatius
Fig. 6 The crypt
These paintings represent one of the largest painting cycles of that period in Germany (covering an area of more than 120 m2). They were for a long time covered, and only in the beginning of the 20th century, were they discovered again.
The condition of the paintings was very inhomogeneous, largely reduced to the preparation layer, as they had been lime washed for around 350 years. After their uncovering they have been conserved in their reduced surface. Most of the restoration information is about the vaults of the crypt. A 2D representation of the vaults either deforms the surface or looses some spatial relationships.
Additionally a 3D model creating a very detailed cloud of points of the crypt was elaborated by the use of a 3D scanner (Figure 7).
Fig. 7 The cloud of points of the crypt
In the last seven years in the crypt of St.Servatius an important investigation and conservation program has been realized using GIS format extensively as a documentation tool, with the “base map” consisting of high-resolution rectified georeferenced photograph(2).
3D model of the crypt
The investigation and conservation program on the crypt produced lot information stored in several formats, photos, text,... In the time several different naming convention were used to identify the different vaults, making the access to the information more difficult than it should be. In particular using a 3D scanner was produced a cloud of points and rectified images corresponding to the ceiling. Moreover the students had inserted information using a GIS program using these orthophotos as reference (Figure 8 - 9).
Fig. 8 Orthophoto
Fig. 9 A map on the saturating with water
Our idea was to use an interactive 3D model to access the information. To realize this goal the model had to consider the following aspects:
-
1.the model should be simple enough so that it can be interactive;
-
2.it should keep into account as much as possible the detailed cloud of points and the geometry of the crypt given by the cloud of points of the 3D scanner;
-
3.It should keep enough connection with the way the orthophotos were taken, so that the data inserted in GIS format can be transferred automatically to the 3D model.
-
4.The visualization of the vaults should be as natural and intuitive so that users can find their way easily.
Initially we tried to apply spline of interpolation in ArcGIS, but to use this method for all vaults
was impossible due to the amount of points, and the connection with the orthophotos was cumbersome.
So we used a parameterized surface, defined as the maximum of a series of piece-wise halfelliptic tubes, to describe the ceiling of the crypt where the paintings are.
This surface was best-fitted to the points retrieved from a 3D scanner, so that we can directly generate the mesh for XVR (Carrozzino et al., 2005).
Having this parametric surface allows us to easily map the coordinates of the orthophotos to texture coordinates and vice versa. The parameters of the parametric surface (width/heights/slope for each cylinder section) are optimized, minimizing the distance between the parametric surface and the 3D scanner points (non-linear least square fitting). A Python script is used for this. The resulting parametric surface and a sample of the scanner points are shown in Figure 10, the deviation from the cloud of points was at most around 1 cm.
Fig. 10 Fitting solution
The orthophotos have been transformed into textures, giving the possibility to transfer the GIS data connected to them to the 3D model and vice versa.
The user can visualize the mural paintings and their maps of the state of conservation produced in the investigation and conservation program. We are working to give access at all the detailed information available (Figure 11).
Fig. 11 The interface of the crypt of St.Servatius: the mural paintings and the maps of the state of conservation in the same view
The usability test
The method has been tested through a usability test. The possibility to jump from the information to the model and vice-versa, using the link of ranking in the list, was deemed not only useful but also enjoyable. Users found it friendly, and felt they were active members, contributing to the community. The results of the usability test on the whole have been very satisfying, they show as the people have found intuitive the interaction, giving more attention to the content and quantity of the information. Moreover in the past ISEE method has been applied in real world with good results. We tested ISEE on-field using a GPS Compass (Vector CSI Wireless), providing 2D heading and positioning data connected to a laptop in Piazza Napoleone in Lucca, Italy (Figure 12) (Pammer & Radoczky, 2002; Pecchioli et al., 2008a).
Fig. 12 ISEE live in the real Piazza Napoleone
Conclusions
This work presented an intuitive and user-friendly method for accessing, inserting and modifying information in a 3D space. An original aspect is provided by defining the spatial relevance of information, which allows users to retrieve significant information according to their position/orientation in the 3D space. This allows an intuitive interface to be provided, where information is retrieved by looking around and provided in a relevant order. To give access to a larger audience, the method is available on the internet and does not require any specific knowledge to use it. The information is associated with regions of the space, and thus pre-processing of 3D models to subdivide them in suitable logical elements is not needed.
New pieces of information can be inserted in the same way in which they are queried, just by looking. The use of extended zones gives to the proposed ranking algorithm a superior performance than rankings based only on the distance.
With a vaulted surface like that one of the crypt of S.Servatius, a fully 3D approach has the advantage of keeping the spatial relationships between the various information. Also one does not need the various nomenclatures used to classify the information during the time, as the position of the information becomes immediately clear.
In particular ISEE will be useful for the students and professionals of the field, that should benefit from a greater level of freedom to manage and manipulate the information retrieved, with tools specific to their field. They will be able to insert more detailed data and to decide presentation and retrieval modes.
This method allows the easy creation of 3D environments that have a rich information content, and will be useful also for AR and VR applications. The ISEE method was successfully implemented in a prototype application using an interactive smart guide.
-
(1)With level of detail we mean the size of the region the information is related to, for example if it relates to a detail of a painting of to the whole painting. This is not related with the level of geometrical detail of the 3D model.
-
(2)By Prof. Heinz Leitner and his students of the Hochschule für Bildende Künst of Dresden.
References
Carrozzino M., Tecchia F., Bacinelli S., and Bergamasco M. 2005. Lowering the Development Time of Multimodal Interactive Application: the Real-life Experience of the XVR project. In: Proceedings of ACM SIGCHI International Conference ACE, Valencia, Spain, pp. 270-273
Carrozzino, M., Tecchia, F., Bergamasco, M., 2009: Urban procedural modeling for real-time rendering. In: Proceedings of the 3rd ISPRS International Workshop 3D-ARCH 2009: 3D Virtual Reconstruction and Visualization of Complex Architectures, Trento, Italy
Pammer, A. and Radoczky, V. 2002. Multimediale Konzepte für mobile kartenbasierte Fussgängernavigationssysteme. In: Zipf, A. and Strobl, J (Eds.) Geoinformation mobil, 2002 Wichmann,
pages117-126
Pecchioli L., Carrozzino M., Mohamed F., 2008a. ISEE: Accessing Relevant Information by Navigating 3D Interactive Virtual Environments. In: Digital Heritage – Proceedings of the 14th International Conference on Virtual Systems and Multimedia, IEEE VSMM 2008. Limassol (Cyprus), S. 326-331, ISBN/ISSN: 978-963-9911-01-7
Pecchioli L., Mohamed F., 2008b. A method to access the information through an interactive 3D virtual environment. In: Entwicklerforum Geoinformationstechnik - Junge Wissenschaftler forschen – Technische Universität Berlin, Institut für Geodäsie und Geoinformationstechnik, Berlin, Deutschland, (Eds) Shaker Verlag, pp.119 - 129
Pecchioli L., Leitner H., 2007. ICOM2007 International Council of Museums, Managing information in Cultural Heritage using a 3D interface. In: Proceedings of CIDOC2007 - International Committee for Museum Documentation, Vienna, Austria
Pillay, R., Aitken, G., Pitzalis, D. and Lahanier C. 2007. Archive sources visualization and exploitation at the c2rmf. In: Proceedings of ICHIM07 Digital Cultural and Heritage, Toronto, Ontario, Canada
Pittarello, F., 2004: Accessing information through multimodal 3D environments: towards universal access. Computer Science, Springer Berlin/Heidelberg, pp. 189-204
---
Laura Pecchioli, Fawzi Mohamed, Marcello Carrozzino - October 2009
Architectural Image-Based Modeling Web Portal
ISEE: accessing information navigating in a 3D virtual environment
The case study of the crypt in St. Servatius in Quedlinburg, Saxony-Anhalt (Germany)
Laura Pecchioli (a), Fawzi Mohamed (b), Marcello Carrozzino (c)
(a) Technische Universität Berlin, Germany
(b) Humboldt Universitätb, Berlin, Germany
(c) Institutions Markets Technologies IMT Lucca Institute for Advanced Studies, Piazza S. Ponziano, 6 55100 Lucca / PERCRO, Scuola Superiore Sant’Anna, Piazza Martiri della Libertàc, 33, 56127 Pisa, Italy
A new method to access spatial information through the interactive navigation of a synthetic 3D model, reproducing the main features of a corresponding real environment, has been developed in our work. The information is ranked with a novel measure of the relevance that depends on the position/orientation in the 3D space, and allows an intuitive interface. To give access to a larger audience, the method is available on the Internet and does not require special training to be used.
The development of the method started from a first case study: the crypt in St. Servatius in Quedlinburg (Sachsen-Anhalt, Germany). The use case presented different types of information related to the restoration and a cloud of points from a 3D scanner. In particular the information had to be connected to a vaulted surface.
The system has been applied to other cases study related both to outdoor and indoor environments, proving potentially to be also an interesting prototype as a smart guide with the use of augmented reality technologies.
Keywords: interactive 3D interface; AR; XVR 3D engine; normalized Gaussians; spatial relevance; Overlap.
