Deep learning techniques for the fine art and Australian Aboriginal painting analysis

<p>In recent years, there has been a growing interest in creating autonomous systems that can understand art concepts. With the permanent expansion of digitalised artwork collections by libraries, museums, galleries, and art centres, automated analysis and categorisation of fine-art paintings have become a highly desirable task. However, this represents a significant challenge due to the variations in the interpretation and perception of the different elements of art, smooth transitions between art periods, the nuances separating different artistic categories, and the strong dependency on high-quality annotations made by art experts.</p> <p>In recent years, there has been a growing interest in creating autonomous systems that can understand art concepts. With the permanent expansion of digitalised artwork collections by libraries, museums, galleries, and art centres, automated analysis and categorisation of fine-art paintings have become a highly desirable task. However, this represents a significant challenge due to the variations in the interpretation and perception of the different elements of art, smooth transitions between art periods, the nuances separating different artistic categories, and the strong dependency on high-quality annotations made by art experts.</p> <p>For centuries, Aboriginal people have been using inventive symbols to recount stories and messages. Due to the absence of a written language, Australian indigenous painting constitutes a visual literacy and a fundamental means to transfer knowledge from generation to generation. Although Australian Aboriginal painting has been recognised globally as a unique, highly valued kind of art and essential for preserving Aboriginal culture, they have not been considered in computer vision studies.</p> <p>This research aims to develop efficient deep learning systems that can automatically classify and analyse fine-art paintings, including the study of Australian Aboriginal Paintings and their differences with other artistic styles.</p> <p>The thesis describes and validates two novel style fine-art categorisation methods based on subregion analysis. In the first method, a weighted image patches approach, the final stylistic label of the analysed painting is given by a weighted sum of the individual-patch classification outcomes. The optimal weight values are determined by a numerical optimisation algorithm that maximises the overall system classification accuracy. The second method is a two-stage deep learning approach. In the first stage, sub-regions or patches of the painting are individually classified. Then, a second classifier is trained using the probabilistic outcomes generated at the first stage to obtain the final stylistic label; the second stage is effectively trained to compensate for potential mistakes made during the first stage. Experiments conducted using three art datasets and six different pre-trained CNNs indicate that the proposed methods significantly outperform the existing baseline techniques.</p> <p>The robustness of the two-stage classification approach was evaluated to investigates the effect of partial damage to a painting on the accuracy of the automatic style recognition. It was found that the combination of damaged and non-damaged painting in the training dataset and the strengthening role of the final decision-making classifier increase the system's robustness to partial damages, while at the same time maintaining a relatively high classification accuracy of non-damaged artworks.</p> <p>As shown by the classification research presented in this thesis, the classification outcomes highly depend on the annotation quality made by art experts. Given a very limited access to such labels, the thesis investigated how to provide automatic labelling of fine-art paintings without the need for human annotation. A new unsupervised classification method is proposed. The proposed Adversarial Clustering System (ACS) combines an unsupervised clustering module and a supervised classification module linked via a numerical optimisation procedure that iteratively improves the clusters' quality. Experiments with three different datasets showed that the ACS method presents higher reliability than the classical clustering method. The content analysis of the final clusters indicated meaningful grouping based on scene composition, type, and shape of the object, edge sharpness and direction and colour palette.</p> <p>Stylistic supervised and unsupervised classification of Australian Aboriginal Painting based on the proposed methods mentioned above indicates that it is possible to recognise and distinguish Australian Aboriginal paintings from other fine-arts styles with high accuracy. The clustering of Australian Aboriginal art paintings, which art experts never-before labelled, led to the discovery of categorisation criteria and art categories that could be useful for storage and retrieval. In the final part of the research, other methods for the automatic categorisation of Australian paintings are explored. In addition, the recognition of a selected group of graphic symbols of Australian iconography is investigated.For centuries, Aboriginal people have been using inventive symbols to recount stories and messages. Due to the absence of a written language, Australian indigenous painting constitutes a visual literacy and a fundamental means to transfer knowledge from generation to generation. Although Australian Aboriginal painting has been recognised globally as a unique, highly valued kind of art and essential for preserving Aboriginal culture, they have not been considered in computer vision studies.</p> <p>This research aims to develop efficient deep learning systems that can automatically classify and analyse fine-art paintings, including the study of Australian Aboriginal Paintings and their differences with other artistic styles.</p> <p>The thesis describes and validates two novel style fine-art categorisation methods based on subregion analysis. In the first method, a weighted image patches approach, the final stylistic label of the analysed painting is given by a weighted sum of the individual-patch classification outcomes. The optimal weight values are determined by a numerical optimisation algorithm that maximises the overall system classification accuracy. The second method is a two-stage deep learning approach. In the first stage, sub-regions or patches of the painting are individually classified. Then, a second classifier is trained using the probabilistic outcomes generated at the first stage to obtain the final stylistic label; the second stage is effectively trained to compensate for potential mistakes made during the first stage. Experiments conducted using three art datasets and six different pre-trained CNNs indicate that the proposed methods significantly outperform the existing baseline techniques.</p> <p>The robustness of the two-stage classification approach was evaluated to investigates the effect of partial damage to a painting on the accuracy of the automatic style recognition. It was found that the combination of damaged and non-damaged painting in the training dataset and the strengthening role of the final decision-making classifier increase the system's robustness to partial damages, while at the same time maintaining a relatively high classification accuracy of non-damaged artworks.</p> <p>As shown by the classification research presented in this thesis, the classification outcomes highly depend on the annotation quality made by art experts. Given a very limited access to such labels, the thesis investigated how to provide automatic labelling of fine-art paintings without the need for human annotation. A new unsupervised classification method is proposed. The proposed Adversarial Clustering System (ACS) combines an unsupervised clustering module and a supervised classification module linked via a numerical optimisation procedure that iteratively improves the clusters' quality. Experiments with three different datasets showed that the ACS method presents higher reliability than the classical clustering method. The content analysis of the final clusters indicated meaningful grouping based on scene composition, type, and shape of the object, edge sharpness and direction and colour palette.</p> <p>Stylistic supervised and unsupervised classification of Australian Aboriginal Painting based on the proposed methods mentioned above indicates that it is possible to recognise and distinguish Australian Aboriginal paintings from other fine-arts styles with high accuracy. The clustering of Australian Aboriginal art paintings, which art experts never-before labelled, led to the discovery of categorisation criteria and art categories that could be useful for storage and retrieval. In the final part of the research, other methods for the automatic categorisation of Australian paintings are explored. In addition, the recognition of a selected group of graphic symbols of Australian iconography is investigated.</p>