Deep learning-based building envelope information extraction for BIPV application

Extracting building façade information from a building is a challenging task that is crucial in many building-related studies, especially in the development of building-integrated photovoltaics (BIPV) application studies. Many studies regarding BIPV application in urban environments had been conducted and by examining these studies. It is discovered that reduction factors such as reduction coefficients and window-to-wall ratio (WWR) have been applied to total surface areas on building envelopes to estimate areas that are BIPV compatible. These values used for reduction factors vary differently based on studies which can result in varying BIPV surface area estimation results. Therefore, this can affect the accuracy of the BIPV energy potential estimation when conducting solar energy potential analysis. Artificial intelligence, particularly deep learning framework has been applied in many computer vision problems to recognise objects. Deep learning frameworks have been applied to recognise building element objects such as windows, balconies, and roofs in built environments. Many deep learning application studies for built environments have shown promising results for being able to identify these building elements. Although deep learning has been applied successfully in these studies, not many studies have attempted to utilise deep learning to recognise BIPV applicable surface areas on building facades and roofs for the purpose of BIPV energy potential analysis. There are many research gaps surrounding the issue of extracting BIPV surface areas from building images using deep learning frameworks and performing BIPV solar energy analysis on the extracted surfaces. This research aims to address these research gaps. Through the literature reviews, six popular building elements were identified for the purpose of BIPV applications which can be broken into two macro categories such as facades and roofs. BIPV application studies were reviewed to identify the commonly used reduction factors so that it can serve as a baseline comparison for this study. Deep learning applications in built environments were also reviewed to select the most suitable deep learning model for this task. The Mask Region-Based Convolutional Neural Network (Mask R-CNN) model was identified as the model of choice for this study as it has been proven effective and easy to use. Although many deep learning studies utilised street view images to generate the necessary training data for the model, this study utilised photomesh data due to its advantages of providing a 360-degree view of the selected building. Mask R-CNN framework was deployed for building façade extraction (windows and glazing facades) from image input. This study utilised transfer learning to train the proposed methodology using Common Objects in Context (COCO) dataset with building façade images collected from public photomesh data of Melbourne Central Business District (CBD). The trained model was used to perform segmentation tasks and produce prediction masks on identified building surfaces which were later used for surface area estimation. Model evaluation results show that the trained model achieved a mean average precision of 76.9% at 0.5 Intersection over union (IoU) threshold. The process of the surface area calculation was done using a grid-based system which allowed shading height to be applied to identify unshaded regions on the building façade.  The same process was also applied to building roofs to identify BIPV compatible roof areas. Four buildings in Melbourne were selected to test the model prediction. The prediction results were validated against ground truth data, and it achieved 75.8% overall accuracy for predicting building elements on the ground truth correctly. Solar energy and shading analysis were conducted for the selected buildings using Quantum Geographic Information System (QGIS) to provide a detailed breakdown of suitable surface area for BIPV installation. Solar energy potential on unshaded surface areas of the building can be estimated accurately using the model’s prediction data. BIPV simulations were conducted for two buildings to compare the accuracy of this methodology against conventional BIPV area estimation. The results confirm that the proposed methodology provides BIPV energy potential estimation which is closer to the ground truth data.