Algorithm for Image-Based Architectural Element Recognition

This research describes an image-based algorithm for a shape recognition of an architectural element from an image. Given an orthophoto of a particular element such as a window, and a query image of the object facade, the algorithm estimates the level of matching to the shape of an observed window. In order to test and evaluate the algorithm, images of two church facades, containing windows distinguished by their specific shape of a long opening with pointed arches, have been used. The proposed algorithm for shape recognition is semi-automatic and it relies on geometric shape representation.


INTRODUCTION
The analysis of architectural elements is broad and various, and the classification of the main elements of architectural style may have different purposes.In the context of the available technologies, architectural elements can be observed and analysed in many different ways.Nowadays, with a rapid development of the technology and computer graphic tools, architecture is mostly visualized within the virtual reality.Many authors deal with the virtual reality, starting from its incorporation in an education in the fields such as architecture and urban design [1], up to using it as the medium for different aspects of architectural analysis [2].On the other hand, when an object repre-sentation is observed and analysed from an image, computer vision techniques are employed in order to recognize, describe and classify specific elements.
The aim of this paper is to provide an algorithm that recognizes geometric characteristics of a shape of a window from an image.For the architectural element to be observed, we have chosen a window as a common element of every architectural object.The research presented in this paper describes the developed algorithm for shape recognition of a specific window from an image.The proposed algorithm for shape recognition relies on the recognition of a basic geometric elements.First, it extracts a geometric shape of the requested window from an orthophoto image.The goal of the algorithm is to find the previously extracted shape on the image of the object facade which contains the windows of the same shape.In order to achieve this, we propose a semi-automatic algorithm which uses a geometric shape of a particular window in order to calculate the level of matching to the observed shape on the input query image of the object facade.In this paper we explain in detail the steps necessary for the functioning of the algorithm and present the results of the performed experiments.

METHODOLOGY
The presented research uses the architectural element to describe the way of functioning of the algorithm for the image-based shape recognition.It describes in detail a proposed workflow, which is based on two main steps.In the first step, photogrammetry is employed in order to obtain an orthophoto of the observed window.An orthophoto serves to define the exact shape of the window in relation to which the requested form will be compared.
We have photographed the windows distinguished by their specific tall, narrow openings with pointed arches, which are typical of Gothic architectural style [3].Windows were recorded with the previously designed plan that takes into consideration the maximum possible distance of the camera from the object, the windows dimensions, the camera parameters such as the size of the sensor, the focal length, and the direct light, all of which are parameters that affect the automatic generation of the orthophoto.In addition, each window was captured from different viewpoints as well as under different weather conditions.After the photogrammetry is employed, the orthophoto of the window is obtained.
Further, a semi-automatic algorithm is developed in order to recognize the shape of the requested window from an image.First, from the orthophoto is extracted a set of a geometric elements such as straight lines and parts of the circle, all of which define the window shape.When the algorithm recognizes these shape elements on the query image, the matching with the shape of the observed window is achieved.
In order to evaluate the algorithm for shape recognition, the images of two churches from Novi Sad: the Roman Catholic parish church "The Name of Mary Church" (Figure 1) and the Roman Catholic church "Heart of Jesus" in Futog (Figure 2), have been used.The facades of the churches are characterized by specific architectural elements such as a typical Gothic window, distinguished by its long opening with the po-inted arch.The algorithm is semi-automatic in the sense that a user has to mark on an image the previously determined points of interest that define the window shape [4].

ALGORITHM FOR SHAPE RECOGNITION
The proposed algorithm for shape recognition is based on the image-processing techniques, and implemented in the C++ programming language, using the OpenCV library.It uses the orthophoto in order to define and extract geometric characteristics of the shape of the window.Given a query image of the church facade, the algorithm aims to estimate the level of matching to the observed shape of the window, previously extracted from the orthophoto.In this section, we describe the proposed algorithm which can be split into the follo-wing main steps that include image-processing techniques, the extraction of the window shape and the algorithm test function, applied to the orthophoto and/or the query image: • Image processing operations: orthophoto query image • Point selection and coordinate extraction orthophoto query image • Homography -perspective transformation query image • Test function -matching pixel function that examines a content of a non-black pixel of the edges that lie on the previously determined geometric elements which define the window shape.
query image

Preparing and extracting image data
The first step of the algorithm presents the image processing methods applied to the orthophoto.Image processing is a method of preparing an image for further analysis by using well known techniques.The OpenCV library [5] is used, and the following image processing methods have been used: • Image blurring which reduces noise from the input digital image.• Canny edge detector that produces the edge map by using numerical threshold values to appoint pixel as an edge.The method was used to extract edges from the original image input.
• Morphology transformation, which is related to the image edge shape.More precisely, the dilate type of morphological operation was used in order to interconnect the previously detected edges into whole region, which was damaged due to the noise in the image.In this case, thicker edges were applied with the aim to facilitate eaisier identification of the main points that define the window shape from the image.After the input image is prepared, the next step of the algorithm is the selection of the points that define the shape of the windows, which are, at the same time, well recognizable in the input image.Point selection is necessary in order to obtain the coordinates of the points of significance for the shape determination on the orthophoto, as they present the subject of further matching calculation.
The five main points that precisely describe the shape of the observed windows, have been determined.The window shape is defined as the set of geometric elements such as the two straight lines defined by pairs of points -A, B and C, D, and two arches to which the points A, E and D, E are chords.Figure 3 illustrates this concept.The four points (A, B, C, D) are required for a perspective transformation of the orthophoto, while the y coordinate of the point E determines the limit of the observed matching area.In the following part, the given points have to be selected by the user in order to extract coordinates from point A to point D, from the orthophoto input image.In this step, the orthophoto image (Figure 4) of the particular window has been used as an input image.Afterwards, the same process is repeated every time a new image is imported, while through each of the following selections, the points have to be selected in the predetermined order (in this case, from point A to point D).

Test function
After the coordinates of the main points from the orthophoto are known, a query image of the church facade is imported (Figure 5).The methods for the image preparation, described in 3.1 are called again, so the edges are detected and morphed, as demonstrated in Figure 6.
The process of the point selection is repeated, within which the four points have to be selected on the input query image, with respect to the predefined order.The coordinates gained from this selection are required for the homography of the query image.Then, the homography is applied to the input query image, where H is 3 x 3 matrix [6], [7], written as In this case, the goal of the homography is to transform the selected coordinates of the query image into the coordinate system of the orthophoto.The homography has been applied to the input query image, whereas x' and y' refer to the coordinates of the orthophoto points, while x and y are the coordinates of the points of the query input image.The query image transformed by homography is demonstrated in Figure 7.The homography matrix used is shown below.
Using the four selected points, required for the homography, the segments of the shape of the window (straight line defined by the points A, B, straight line defined by the points C, D, as well as two arches formed by points A, E and D, E) are defined on the query image.After the query input image has been warped by applying the previously described homography, the test function can follow.The perspectively transformed query image, containing the previously detected and morphed edges, is used to test if the matching with the window shape, defined on the orthophoto, is achieved.The test function is based on analytic geometry and performs in the following way.It relies on the window shape specified as the set of geometric elements (the line segment defined by the points A and B, the line segment defined by the points C and D, as well as the two arches to which the points A, E and D, E are chords), which is previously extracted from the orthophoto.Then, it uses the query image of the object facade in order to examine a content of a non-black pixels of the edges that lie on the previously determined geo-metric elements.The matching to the corresponding window shape is achieved if the pixels at that locations (x, y) are non-black.The obtained result of the matching calculation is the percentage value of the presence of non-black pixels, presented for each shape segment (the line segment AB, the line segment CD, the arc AE, the arc DE).The test function can be described as the following main steps: • Step 1: The pixel value is evaluated for 100 points of each shape segment of the window: -Line segment defined by A and B -Line segment defined by A and B -Arc between points A and E -Arc between points D and E • Step 2: If the pixel is non-black -at this pixel position is achieved matching to the corresponding shape segment of the window.

EXPERIMENTAL RESULTS
The experiment has been performed on the images of window of two churches: the Roman Catholic parish church "The Name of Mary Church" and the Roman Catholic church "Heart of Jesus" in Futog.The images were taken with various distances and at different angles, and used to test the algorithm.
In the case of the Roman Catholic parish church "The Name of Mary Church", the algorithm is evaluated through its application on four different window positions of two images of the facade, captured from two angles in clear weather and an appropriate lighting conditions (Figure 8).As previously described, the query image is perspectively transformed on the basis of the selected points from the orthophoto.
By using the transformed image, test function examines if the matching with the window shape, defined on the orthophoto, is achieved.In other words, by using the set of the geometric elements extracted from the orthophoto as a reference value, the algorithm analyses the transformed image with the goal to estimate if he detected edges are matched with the previously defined geometric segments of the window shape.The results of the calculated pixel values and the percentage of the matching, are shown in the Table below (Table 1).In two case studies, the shapes were completely matched (100%), and the algorithm achieved a high percentage of matching between the query images and the ortophoto of the specific window.
The positive re-sults can be related to the appearance of the church, dis-tinguished by the contrasts between the brick facade and the surface of the window, which has contributed the mostly to the successful algorithm application.The only two values less than 100% pixel matching refer to the AB and CD line segments of the window shape (Figure 8-b and Figure 8-d), indicating that, due to the view angle, the edges closer to the observer are sheltered and, as such, more difficult to detect than those further away.This is caused by an appearance of a new false edge, which affects the accuracy of the matching results.

Figure 9. Example of the low constrasted query image
In addition, the algorithm has been tested on the special cases of unfavorable conditions for the operation of edge detection.The first input image is low contrasted (Figure 9), while the second is shaded (Figure 10).The experimental results obtained for the images taken in special conditions are displayed in Table 2.As expected, the shaded and occluded segments of the image made the process of edge detection, as well as the function of point selection, difficult to perform, because the detected edges were cut off in those areas of the images.Consequently, this resulted in lower matching values.Yet, it can be observed that the partially shaded image has less impact on the edge detection than the uniform image with low contrast.The well-contrasted segments of the images, i.e.AE and DE arches were completely detected, while edges were disconnected in the shaded parts of the image, which can be seen from the percentage values obtained for the line segments defined by the points A, B and C, D. In the next experiment, the evaluation of the algorithm on the Roman Catholic church "Heart of Jesus" is also performed by using window shape extracted from the orthophoto as a reference value.The algorithm was tested on the images of three different window positions (Figure 11), since, in this case, the facade capturing was constrained by the location conditions.The output parameters are displayed in Table 3. Unlike the desirable appearance of the previously presented church, the Roman Catholic church "Heart of Jesus" is characterized by a homogeneous texture with low contrast which caused difficulties during edge de-tection operation.In this case, the homogeneous texture of the object have presented a challenge for the algo-rithm application.The obtained shape matching results vary in the range from 35% to 100%, which is mainly caused by the homogeneous colors and textures of the surfaces that surround the window.It can be seen from the obtained results that the best matching of the shape is achieved from the same frontal relative positions to the object (Figure 11b), while the previously described problems are mostly expressed in images captured at a certain angle.
Considering the previously discussed experiments, it can be concluded that the performance of the algorithm for shape recognition of the specific windows depends on various conditions, such as the visual characteristics of the observed window.

CONCLUSION
In this paper, a semi-automatic algorithm is proposed with the goal to recognize the specific shape of the previously determined window.The algorithm uses an information about particular window shape from an orthophoto in order to find the observed shape of the window on a query image.The proposed algorithm for shape recognition relies on the shape-based object representation.The developed algorithm is described in detail, and the experiments followed by the results obtained are presented.For the evaluation of the algorithm, the images of the facades of two churches have been used.Given an orthophoto of a particular window and a query image of the church facade composed of the same windows, the algorithm estimates the level of matching to the shape of the observed window.
According to the experiments done, we noticed that the performance of the image-based algorithm for the shape recognition of the specific window was mostly influenced by the visual characteristics of the observed architectural element.

Figure 1 .Figure 2 .
Figure 1.Roman Catholic parish church "The Name of Mary Church" in Novi Sad

Figure 3 .
Figure 3. Key points of the window shape

Figure 4 .
Figure 4. Orthophotos of the windows, obtained by using photogrammetry

Figure 5 .Figure 6 .
Figure 5. Input query image of the church facade

Figure 7 .
Figure 7. Perspective transformation applied to the query image

Figure 10 .
Figure 10.Example of the shaded query image

Figure 11 .
Figure 11.Three different window positions a),b),c) of the Roman Catholic church "Heart of Jesus" in Futog