What do two images tell us about a third one?

From its origin, the photograph always has been considered to be a solution for “recording” the reality. For the disciplines interested in the built heritage documentation, it represents an ideal support for the surface reading, the shape analysis and for the understanding of the elements composing an architectonic organism. From a geometric point of view, a photograph is the projection of a three-dimensional scene on a bidimensional plane. In the course of the past century, photogrammetric techniques based on the correspondence between a perspective projection and an orthographic one, have permitted to establish a tightened relation between the space of the photograph and that one of the representation (C.f. image rectifications, orthophotos). So, if the geometric model of an image is known, it is possible to know the projection of a point in the scene, by starting from the optical center of the related camera and projecting trough to infinity. But, if one has two images of the same scene, 3D coordinates can be retrieved starting from the intersection in space of the projections of two homologous points selected on the images. Starting from these principles (briefly explained here) stereo-photogrammetric techniques were developed. Since approximately thirty years the photogrammetry, like the other measurement systems, has broken with the analogical tradition and rode the wave of the electronics and IT revolutions.

Figure 1. Geometric relations between two images
( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2006 )

But it is really interesting to notice that recent progresses in computer vision, currently a very active research field (that has recovered the progresses made in the field of photogrammetry) has merged the recent methods of algorithmic calculation and the mathematical principles introduced by Poncelet. A digital camera, from a geometric point of view, remains, in fact, a simple “perspective machine”, the same as those described in the treaties of the XIX century. Recent procedures in photogrammetry and computer vision exploit, thus, the a priori knowledge of the projective space in order to lead algorithmic iterations. So what two images tell us about a third one? The answer is based upon the concept of « geometry of multiple images », the laws that define the mutual relations between a 3D scene and all its possible 2D projections. Previous knowledge on existing relations between groups of images is so exploited in order to determine at the same time the geometry of the scene and that of the cameras that photographed it. In effect, the knowledge of common geometry between images forces the problem of the correspondence and, reciprocally, the knowledge on the correspondence between images supplies the information on common geometry. Therefore the problem becomes estimating the geometry of a scene based upon the analysis of the images that represent it. So, in a century of technological progress, the relation between the image and the object to represent was enriched by another dimension: the space of the representation becomes definitively three-dimensional and the photograph, one of the possible points of view of this space.

Figure 2. Relation between architectural elements/spaces and image acquisition.

( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2007 )

Understanding the geometrical nature of architectural elements

Actual Image-Based-Modeling techniques allow the extraction of all the necessary information (coordinates, distances, points and planes for the 2D drawing, relevant vertex or profiles for the 3D reconstruction, textures for enriching the recreated geometry) directly from photographs.

The 3D reconstruction workflow is based upon the definition of the position in space of every characteristic point of the shape to reconstruct and to take in account the way in which these points are connected by geometric entities of various level of complexity.

The choice of the points to select on photos is conditioned at the same time by the objective of the restitution and by the chosen type of geometric representation.

Figure 3. Modeling by primitives : parallelepiped

( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2007 )

Image-Based-Modeling uses previous knowledge of geometric invariants of architectural spaces and elements (parallelism, symmetry, orthogonality, coplanarity, etc.) in order to determine the position and orientation of the objects composing the photographed scene.

The architectural composition of a great number of buildings is based on a reasoned spatial distribution of elements. These elements are often organized by geometrical layouts around the concepts of order, module, stage, etc.

Symmetry is the basic principle of a great number of architectural shapes: correspondences of size, position and orientation that parts and sub-parts of an element have between them and with their unit. In geometrical modeling, symmetry is also an intrinsic property related to extrusion. The combined use of extrusion and scaling functions along the same symmetry axis, allows, for example, a progressive reconstruction of the transitions characterizing the shape along a dominant direction.

Figure 4. Modeling by primitives

Eglise Saint Louis, Mont Dauphin, France
( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2007 )

In a great number of architectural elements, the concept of geometrical transition is closely related to that of moulding: the use of a whole range of atomic units to be combined in order to compose profiles. Understanding the role that mouldings play is essential in order to be able to identify the strategy the most adapted to decomposing very complex elements.

Figure 5. Modeling by progressive extrusion

Palais Royal, Paris
( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2003 )

Although the objects handled by architects through the centuries are primarily three-dimensional, the figurative practice always had the plane as a support. The profiles extracted from relevant planes then constitute the descriptors of surfaces that can be generated by functions of sweeping. A great number of surfaces of classical elements are, in fact, the result of the sweeping of a generating profile along a directing path.

These reflections show the observation and the understanding of the geometric nature of the architectonic elements play here the most important role. Modeling architectural shapes starting from photographs can be an effective cognitive exercise based on the relation between two aspects: in one hand, the understanding of the existing relations between the parts and the sub-parts of the shape, on the other hand, the knowledge related to the geometrical representation of these shapes. This double vision of the modeling problem transforms the 3D reconstruction of a building into a real analysis that is necessarily based on the architectural knowledge. The objective then becomes to explicit, at the same time, the geometric and the architectonic sense of the shape, to add to their geometric properties, some other ones of sensitive expression. The nature of an architectural element will thus be defined by the relationship between geometrical characteristics and architectural qualities.

As they act towards reducing the multiplicity to the intelligibility, the codes of architectural representation offer a privileged support for the identification of the modeling strategy. For each type of element, it is necessary to identify the relevant information for its semantic and geometric description, the more apt information extraction strategy and the more effective geometrical reconstruction technique.

Figure 6. Modeling by profiles

Left : Villa Savoye, Poissy, France. Right : Ex-convento dell’Annunziata, Acoli Piceno, Italia.
( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2008 )

Like other computer graphics techniques, Image-Based-Modeling is not based on the direct representation of the object, but on the formalization of its representation. One can explain this aspect by making a parallel with music. In music it’s easy to notice the relation between a musical piece (the perception of the sounds, of the rhythm and harmony) and the score of this musical piece (the transcription of notes, their duration and their combinations). In the same sense, one can imagine that Image-Based-Modeling applied to the architectural modeling could be an instrument in order to write the « geometrical score » of a building.

Bouncing from real to virtual and vice-versa

In the field of the architectural representation, the elaboration of images that contain interpretative factors is fundamental. In this sense, it is important to focus on a modality that allows integrating different kinds of representation. The Image-Based-Modeling workflow organizes all the analyzed and represented elements (coordinates, measures, characteristic lines, volumes, visual appearance, etc.) into a geometrically and spatially coherent assembly. Taking into account some structuring procedures, that means creating a multi-representation system that structures all of the acquired, analyzed and interpreted information around a model of semantic description of the building. It allows structuring the information in a way that is able to represent the modeled elements in different analysis contexts.

Like Image-Based-Modeling, Image-Based-Rendering results from the integration of principles and techniques coming from various fields (geometric modeling, photogrammetry, computer vision, computer graphics …). It represents the last stage of the technological evolution of the relation between an object in space and its projection on a plane: the possibility to extract textures directly from photographs and to project them on the geometry. The camera, initially a machine used to capture the reality thus becomes able to project the reality.

Figure 7. Image-based textures extraction and projection

Hôtel de Sully, Paris
( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2004 )

That allows the rendering of the visual appearance of the surfaces in order to visually enrich the geometrical model. The texture extraction and projection combined to a semantic structuring of the geometry then becomes a solution for the structured recording of the building’s visual appearance.

Movie 1. Image-based texturing

Chateau Comtal, Carcassonne, France
( 3D modeling by Melanie Chaillou, Texturing by Chawee Bussarayat - UMR CNRS/MCC 694 MAP-Gamsau, 2005 )

It is an especially interesting aspect if one considers that, in the field of historic conservation, a photograph is essentially the testimony of the state of a building in a moment “T”.

It also provides the opportunity to lead a three-dimensional reconstruction starting from historical photos, a particularly interesting possibility when the aim is extracting metric and geometric information related to buildings that no longer exist. In the case of partially destroyed structures, the technique can be employed to integrate historical photos and contemporary ones into a camera calibration and orientation process, based on the building’s parts that still visible today. These are interesting possibilities regarding the analysis of the transformations of a building in time. For example, if spatial relations can be established between photographs of different temporal states of a building, then the changes it undertook can be visually illustrated.

Another interesting aspect concerns the visual representation of the anterior state of a building. It is obvious that today’s computer graphics techniques allow simulating hypothetical materials in a synthetic way. But these simulations imply a high level of uncertainty from a documentation point of view. On the contrary, if a historical photograph is used as support for the geometric reconstruction, the projected textures will originate in the original document. In this way, the 3D model will visually integrate the original testimony of an anterior state of the building. Another parallel with music could be established - the difference between sampling and synthesizing sounds on a digital keyboard.

Movie 2. Image-based-modeling using historic photographs

Trophée des Alpes, La Turbie, France

( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2009 )

The projective relation between photographs and geometry can be also observed in the opposite sense - integrating virtual objects into real scenes.  A real/virtual hybrid representation can be based on a twofold (geometric and photometric) coherence. Regarding the geometrical coherence, the spatial matching of virtual elements into real scenes is normally assured by the Image-Based-Modeling workflow: if photos are used for the geometric modeling of the real scene, then hypothetical elements added to this scene will be directly projected on photos. Regarding the photometrical coherence, Image-Based-Lighting techniques allow extracting values related to the lighting system of a real scene in order to illuminate virtual elements. Global illumination algorithms (that can eventually use High Dynamic Range Images) allow, in fact the spatial distribution of a variety of light emitters and receivers. They sample and restitute the direct and indirect lighting systems present in the real scene with the aim to better integrate virtual elements with real ones.

Figure 8. Real/virtual hybrid representation

Left : Saint Symeon Site, Syria ( Jean-Luc Bischop, Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2008 ).

Right : Abbey of Cluny, France ( On-Situ, Institut Image, 2006 ).

A related field is Augmented Reality. An augmented reality device allows the addition of spatialized information to the real world, for example the comparison between the actual state of a building with a hypothetical one. It’s like the conclusion of a process that starts with the reading of the reality, continues with the study of the shape, the representation of its transformations in time, and ends with a return to reality by means of a visualization device that allows observing the real spaces augmented by their understanding.

Movie 3. Augmented Reality application

Church of Chartreuse de Villeneuve-lez-Avignon, France
( Livio De Luca, Aurélie Favre-Brun - UMR CNRS/MCC 694 MAP-Gamsau, 2009 )

Integrating image-based techniques with other surveying tools

These last years, 3D laser scanners made their appearance in the field of architectural surveying. These devices, characterized by their very important capacities of acquisition (thousands of 3D points per second), allow registering metric information relative to a building in form of dense and precise point clouds. However, the results of a 3D laser survey and the photographic acquisition constitute two different kinds of information, both useful for the analysis and the modeling of elements composing a building. The joint use of 3D point clouds and photographs is today the object of many research works. The suggested solutions generally concentrate on supplementing 3D data using photographs. The aim is exploiting the data complementarity and, in particular, substituting geometrical information coming from a source by more precise or easier to acquire information coming from the other. This fusion can be based on spatial resection solutions (2D/3D correspondences), the orientation of photographic images related to a 3D point cloud.

Figure 9. Point cloud /image integration

Hôtel de Sully, Paris
( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2005 )

Fusion of data sources provides enhanced tools for the geometrical analysis and understanding of the surveyed shapes. For example, the intersection between a dense point cloud and a plane allows extracting a set of points describing relevant profiles. By exploiting the relation between photographs and point clouds, obtained through a spatial resection, it is possible to lead profiles extraction based on the simultaneous observation of the point cloud and the photograph. In this case, the intersecting plane is used, at the same time, as a photo rectification plane. The 3D modeling workflow can consequently use simultaneously the point cloud as metric support (for the choice of relevant profiles) and photographs as visual aids (for the interpretation of shapes) and as sources for further texture extraction and projection.

Movie 4. Nubes Forma : profiles extraction module

( Livio De Luca - UMR CNRS/MCC 694 MAP-Gamsau, 2006 )

Spatial referencing of photographs for architectural heritage databases

Camera calibration and orientation techniques, as well spatial resection solutions allow essentially to spatialize photographic information, which is an important issue related to the building documentation. For the study of building, hundreds of photographs can be acquired, oriented, or exploited to visually enrich geometry. Another possible exploitation of the spatial referencing of photographs relates to the database management:  the possibility to find photographs according to spatial criteria. Storing photos with relative information about their camera geometrical model in a data base allows, for example, comparing the position of an observation point in a 3D scene with the one in the stored photographs. Thus becomes possible finding photographs depicting the portion of space that is being observed in the three-dimensional model. It works by executing a real-time filtering of the database entries according to the point of view in the 3D scene. The spatial referencing of photographs can be also exploited in the opposite sense: projecting the semantic structure of a 3D model on photographs in order to analyze their architectural content.

Movie 4. Nubes Imago : images retrieval module

( Chawee Bussarayat - UMR CNRS/MCC 694 MAP-Gamsau, 2008 )

This allows enriching the informative level of entries in the database in order to allow queries according to semantic criteria. For example, it is possible to search photographs into which a specific architectural element is represented in the database.

That opens at an interesting research track related to ontologies (structured set of terms and concepts related to the sense of a knowledge domain). The purpose can be to construct semantic relations graphs, in order to navigate trough concepts of a knowledge universe while switching between different image-based representations. That would allow “bouncing” from a photograph of a building to its 3D model, from an architectonic element of this model to the collection of photographs that represent this kind of element in other buildings, from a qualitative attribute used in order to describe this element to all other elements that have the same attribute, and so on.

In conclusion, it is really interesting to notice here that the construction of this kind of system is like the construction of a space, the space of the information. It is so because the art of expressing a model or concept of information used in activities that require explicit details of complex system is called: Information Architecture.

References

Al Khalil O. et Grussenmeyer P., Solutions for Exterior Orientation in Photogrammetry :  A Review. The photogrammetric record, ISSN 0031-868X, vol. 17, n°100, pp. 615-634, octobre 2002.

Bénallal M., Système de calibration de Caméra. Localisation de forme polyédrique par vision monoculaire. Thèse de l’École des Mines de Paris. 2002.

Biermann V., Borngässer Klein B., Evers B., Freigang C., Grönert  A., Jobst C., Kremeier J., Lupfer G., Paul J., Ruhl C., Sigel P., Stewering R., Thoenes C. et Zimmer J., Théorie de l’architecture : de la Renaissance à nos jours. Taschen, 2003.

Borshukov G., New Algorithms for Modeling and Rendering Architecture from Photographs. M.S Thesis, EECS department, UC Berkeley, 1997.

Buehrer T., Li Z., Gruen A., Fraser C. et Ruther H., Photogrammetric Reconstruction and 3D Visualisation of Bet Giorgis, a Rock-hewn Church in Ethiopia. Proceedings of Surveying and Documentation of Historic Buildings, Monuments, Sites - Traditional and Modern Methods. CIPA International Symposium, University of Potsdam, Germany, 2001.

Bussarayat C., De Luca L., Véron P. et Florenzano M., An On-line system to Upload and Retrieve Architectural Documents Based on Spatial Referencing. Research in Interactive Design, vol. 3. Proceedings of IDMME-Virtual Concept 2008, Beijing, 2008.

Chen. S. E., QuickTime VR – An Image-Based Approach to Virtual Environnement Navigation. Procedings of
SIGGRAPH ‘95, pp.29-38, 1995.

Chitam R., Gli ordini classici in Architettura. Hoepli, 1998.

Cipolla R., Robertson D. et Boyer E., Photobuilder – 3D Models of Architectural Scenes from Uncalibrated Images. Proceedings of the International Conference on Multimedia Computing and Systems, juin 1999.

Debevec P. et Malik J., Recovering High Dynamic Range Radiance Maps from Photographs. Computer Graphics, vol. 31, pp. 369–378,1997.

Debevec P., Taylor C. J. et Malik J., Modeling and Rendering Architecture from Photographs : A hybrid Geometry and Image-Based Approach. Proceedings of SIGGRAPH ‘96, pp. 11-20, 1996.

De Luca L., Relevé et multi-représentations du patrimoine architectural : définition d’une approche hybride de reconstruction 3D d’édifices. Thèse de doctorat, Arts et Métiers ParisTech, 2006.

De Luca L., Busarayat C., Stefani C., Véron P. et Florenzano M., An Integrated Framework to Describe, Analyze, Document and Share Digital Representations of Architectural Buildings. Proceedings of VAST 2007: Future technologies to empower heritage professionals, Brighton, 2007.

Dudek I., Blaise J.-Y. et Beninstant P., Exploiting the Architectural Heritage’s Documentation :  A Case Study on Data Analysis and Visualisation. Proceedings of I-KNOW ‘03, Austria, 2003.

El-Hakim S. F., Beraldin J.  A., Picard M. et Vettore A., Effective 3D Modeling of Heritage Sites. Proceedings of 4th International Conference on 3-D Digital Imaging and Modeling, pp. 302-309. Banff, Canada, 2003.

Falcidieno B. et Spagnuolo M., A Shape Abstraction Paradigm for Modelling Geometry and Semantics. Proceeding of the international Conference on Computer Graphics. IEEE Computer Society Press, Los Alamitos, California, 1998.

Faugeras O., Three-Dimensional Computer Vision : A Geometric Viewpoint. MIT Press, Cambridge, Massachusetts, 1993.

Foley J. et al., Getting There : The Ten Top Problems Left. IEEE Computer Graphics and Application, vol. 20, n°1, pp. 66-68, 2000.

Gaiani M., Translating the Architecture of the World into Virtual Reality and vice versa : 7 years of Experimentation with « Conservation and Representation » at OFF, Officina infografica, Faculty of Architecture, University of Ferrara. Proceedings of Heritage Applications of 3D Digital Imaging. NRC, Ottawa, 1999.

Genç S. et Atalay V., Texture Extraction from Photographs and Rendering with Dynamic Texture Mapping. Proceedings of the 10th International Conference on Image Analysis and Processing (ICIAO ’99), 1999.

Hartley R. et Zisserman A., Multiple View Geometry in Computer Vision. Cambridge University Press, 2004.

Heckbert P.  S., Survey of Texture Mapping. IEEE Computer Graphics and Applications, 6/11, pp. 56-57, 1986.

Heine E., High Precision Building Documentation: Element Definition and Data Structuring. Proceedings of The ICOMOS & ISPRS Committee for the Documentation of Cultural Heritage, Working Group II, Recife, Brazil, 1999.

Kasser M. et Egels Y., Photogrammétrie numérique. Hermès Science, 2001.

Liebowitz D., Criminisi A. et Zisserman A., Creating Architectural Models from Images. Proceedings of Eurographics ‘99, Computer Graphics Forum, vol. 18, pp. 39-50, 1999.

McMillan L., An Image-Based Approach to Three-Dimensional Computer Graphics. Ph.D. Dissertation. UNC Computer Science Technical Report TR97-013, University of North Carolina, 1997.

Migliari R. et Docci M., Geometria e Architettura. Gangemi Editore, 2000.

Ministère de la Culture et de la Communication, Culture et Recherche n°118-119, 2009.

Mitchel J., The Logic of Architecture : Design, Computation, and Cognition. MIT Press. Cambridge, Massachusetts, 1990.

Oliveira M. M., Image-Based Modeling and Rendering Techniques : A Survey. RITA, vol. IX/2, 2002.

Palladio A., Les quatre livres de l’architecture. Édition originale : Venezia, 1570. Réédition : The Four Books of Architecture. Dover publications, 1965.

Perouse de Montclos J.-M., Architecture vocabulaire. Principes d’analyse scientifique. Imprimerie Nationale, 1972.

Quintrand P., Autran J., Florenzano M., Fregier M. et Zoller J., La conception assistée par ordinateur en architecture. Hermès Science, 1985.

Rattner D., Parallel of the Classical Orders of Architecture.  Acanthus Press, 1998.

Saint-Aubin J.-P., Le relevé et la représentation de l’architecture, Inventaire Général, E. L. P., 1992.

Scholfield P.  H., The Theory of Proportions in Architecture. Cambridge University Press, 1958.

Shum S.Y., Kang. S.B., A Review of Image-based Rendering Techniques. IEEE/SPIE Visual Communications and Image Processing (VCIP), pp. 2-13. Perth, 2000.

Simon G. et Decollogne J., Intégrer des images réelles et des images 3D : postproduction et réalité augmentée. Dunod, 2006.

Snavely N., Seitz S. M. et Szeliski R., Modeling the World from Internet Photo Collections. International Journal of Computer Vision, 2007.

Tzonis A. et Lefaivre L., Classical Architecture : The Poetics of Order. MIT Press, Cambridge, 1986.

Wittkower R., Architectural Principles in the Age of Humanism. John Wiley & Sons, 1998.

---
Livio De Luca - December 2009 - Architectural Image-Based Modeling Web Portal

Architectural Image-Based surveying, modeling and representation

Methodological reflections and research tracks


Livio De Luca, UMR CNRS/MCC 694 MAP-Gamsau
livio.deluca@gamsau.archi.fr

New technologies offer to the cultural world some fresh possibilities of knowledge creation and sharing. The heritage digitization contributes to the development of the information society and to the democratization of the access to culture. In this context, the three-dimensional representation is being asserted as an effective support in order to document the actual state of historical buildings, to supply resources to the researchers who study its evolution and to elaborate multimedia supports for the general public. The acquisition, treatment and visualization of metric data have known a meaningful progress in the past few years. New instruments and techniques for surveying and representation allow unimaginable until today results in terms of accuracy, speed, exhaustiveness and photo-realism. But even if today one disposes of these instruments, the requirements inherent to the fields of architectural heritage documentation, conservation and valorization remain are still challenging. These requirements impose the elaboration of intelligible representations. In this sense, the issue of making these emergent techniques coherent with regard to the codes of the architectural representation is an important one. The problem concerns principally the need to evaluate the way these techniques allow a geometric interpretation of shapes aimed to the understanding of architectural compositions. That corresponds to a praxis in logical continuity with the history of the architectural representation, and coincides with the will not to limit building surveying to the simple control of « machines for taking points ». Among recent digital tools and techniques, it is interesting to focus on all that can be done, all that is being made, and all that will be possible to make starting from photographic images. The purpose is to reason about the effective ways for the application of image-based modeling, rendering and lighting techniques to the surveying, modeling and representation of the architectural heritage.