Seeing the Differences in Artistry among Art Fields

by using Multi-task Learning

Ryo Sato, Fumihiko Sakaue and Jun Sato

Nagoya Institute of Technology, Japan

{r-sato@cv., sakaue@, junsato@}nitech.ac.jp

Keywords:

Multi-Task Learning, Relevance of Art Fields, Analysis of Artistry.

Abstract:

In this paper, we propose a method for analyzing the relevance of artistry among multiple art ﬁelds by using

deep neural networks. Artistry is thought to exist in various man-made objects, such as paintings, sculptures,

architectures, and gardens. However, we are not sure if the artistry or the human aesthetic sensitivities in these

different art ﬁelds is the same or different. Therefore, we in this paper propose a method for analyzing the

relevance of artistry among multiple art ﬁelds by using deep neural networks. In particular, we show that by

using the multi-task learning, the relevance of multiple art ﬁelds can be analyzed efﬁciently.

1 INTRODUCTION

In recent years, various new methods have been de-

veloped in deep learning, and their applications are

advancing in various ﬁelds such as image recognition

and generation. Especially in recent years, it has be-

come possible to generate highly accurate images that

deceive the human eye by using GAN (Goodfellow

et al., 2014). While the technology for generating re-

alistic images has evolved signiﬁcantly, the technol-

ogy for realizing the artistic creativity of human be-

ings is still developing.

As a deep learning network that simulates human

artistry, Elgammal et al. proposed a method for gen-

erating artistic paintings by using a GAN-based net-

work called CAN (Elgammal et al., 2017). They con-

sidered the relationship between the knowledge of

past works and new artistry, and based on that, they

tried to minimize the deviation from the distribution

of artistry while maximizing the deviation from the

existing style for generating creative artistic works.

As a result, they succeeded in creating a work that

does not follow the traditional style of artistry.

While research on the generation of artistic works

is progressing, the analysis of artistry, i.e. artistic

creativity of human beings, has not yet progressed

much. Many methods have been proposed for clas-

sifying artistic paintings according to artist, style and

genre (Tan et al., 2016; Anwer et al., 2016; Bianco

et al., 2019), but these methods simply classify artis-

tic works based on theirsimilarity and cannot measure

the artistic creativity that exists in these artistic works.

Painting

Architecture

Sculpture

Garden

Figure 1: Various art ﬁelds.

The concept of artistry is vague and its deﬁnition

is not clear even among experts (The Metaphysics

Research Lab, 1995). Fig. 2 shows Picasso’s ”The

Weeping Woman” (Picasso, ) and Van Gogh’s ”Sun-

ﬂowers” (van Gogh, ). Both are famous works of art

and are highly evaluated by experts and the general

public, but it is not clear where in these works we

feel the artistry. Also it is not clear why the works

made by ordinary people are not artistic. While it can

be conﬁrmed that the artistry is ambiguous, it is cer-

tain that there is something that fascinates many peo-

ple in art works such as ”The Weeping Woman” and

”Sunﬂower”. That is, it seems that these images have

sufﬁcient information for determining the presence or

absence of artistry. Therefore, in this research, we an-

alyze what the artistry of these works of art is by using

deep neural network.

It is known that paintings are not the only things

that have artistry, but various other man-made objects

such as sculptures, architecture, and gardens are con-

sidered to have artistry as shown in Fig. 1. Therefore,

by clarifying the artistry common to these multiple

ﬁelds, it is considered that the artistry can be grasped

more objectively. However, it is not clear whether

Sato, R., Sakaue, F. and Sato, J.

Seeing the Differences in Artistry among Art Fields by using Multi-task Learning.

DOI: 10.5220/0010902300003124

In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP, pages

609-616

ISBN: 978-989-758-555-5; ISSN: 2184-4321

609

”The Weeping Woman”

by Picasso

”Sunﬂower” by Gogh

Figure 2: Work of art in painting.

Figure 3: Output images of CAN.

the artistry in these different ﬁelds is the same or dif-

ferent. If the artistry is common among the multiple

ﬁelds, the network that can identify the artistry of one

ﬁeld is considered to have the ability to identify the

artistry of other ﬁelds as well.

Thus, in this research, we train classiﬁers that dis-

tinguish between the presence and absence of artistry

in multiple art ﬁelds using multi-task learning (Caru-

ana, 1997). By analyzing the improvement in artistry

identiﬁcation in the multi-task learning, we clarify the

commonalities of artistry among different ﬁelds. We

also analyze the characteristics of the shared layer in

the network after the training and show the relation-

ships of artistry among different ﬁelds.

2 RELATED WORK

Many methods have been proposed for classifying

artistic paintings according to their artist, style and

genre (Tan et al., 2016; Anwer et al., 2016; Bianco

et al., 2019). However, these methods simply classify

artistic works based on their similarity, and they can-

not estimate the artistic creativity that exists in these

artistic works.

Although the analysis of human creativity is dif-

ﬁcult and has not yet progressed much, the study on

generating creative works by using deep neural net-

work has started. Elgammal et al. (Elgammal et al.,

2017) have developed Creative Adversarial Networks

(CAN), which outputs artistic paintings by using gen-

(a) basic network

structure

(b) example of

multi-task learning

(Kendall et al., 2018)

Figure 4: Multi-task learning.

erativeadversarial network (GAN) (Goodfellowet al.,

2014; Isola et al., 2017). In their method, the rela-

tionship between the past works and new artistry is

analyzed, and the deviation from the distribution of

artistry is minimized while the deviation from the ex-

isting style is maximized for generating creative artis-

tic works. As a result, they succeeded in creating

a work that does not follow the traditional style of

artistry as shown in Fig. 3 and making us think that

the work is artistic.

While networks that perform single tasks are be-

ing developed for solving various problems, networks

that efﬁciently perform multiple tasks are also being

developed. This is called multi-task learning (Caru-

ana, 1997). The multi-task learning learns multiple

related tasks at the same time, and the common fea-

tures of these tasks can be obtained in a single net-

work. As a result, the multi-task learning can improve

the accuracy of individual task execution (Kendall

et al., 2018; Liu et al., 2019).

Traditionally, multi-task learning networks in

computer vision follow a simple outline which con-

sists of a global feature extractor made of convolu-

tional layers shared by all tasks followed by an indi-

vidual output branch for each task as shown in Fig. 4

(a). For example, when we want to train the segmen-

tation, object detection, and depth estimation, the net-

work structure of the multi-task learning can be de-

signed as shown in Fig. 4 (b) (Kendall et al., 2018).

More recently, various forms of more complex net-

work structures have been proposed. Since the ob-

jective of this research is to clarify the existence of

artistry common to all art ﬁelds, we use a network

structure for multi-task learning suitable for this ob-

jective.

3 ARTISTRY ANALYSIS IN

MULTIPLE ART FIELDS

In this research, we use multi-task learning for ana-

lyzing the relationships among multiple different art

ﬁelds. As shown in Fig. 1, artistry may exists in

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

610

Figure 5: Network structure of multi-task learning in this

research.

Figure 6: Basis block used in this research.

many man-made objects, such as paintings, sculp-

tures, architecture, and gardens. However, it is un-

clear whether the artistry of those different art ﬁelds is

measured on the same scale or on different scales. In

this research, we analyze the commonality of artistry

among these art ﬁelds by learning the relationships of

artistry among these art ﬁelds using multi-task learn-

ing.

By using multi-task learning, it is possible to learn

common features between tasks and task-speciﬁc fea-

tures at the same time. Therefore, in this research, we

construct a classiﬁer that identiﬁes the magnitude of

artistry of each work in each art ﬁeld by single-task

learning and multi-task learning respectively. In the

multi-task learning, it is possible to learn the com-

mon part in the artistry more efﬁciently. Therefore,

if there exists a commonality of artistry in these art

ﬁelds, it is expected that the accuracy of identifying

the artistry will be improved by using the multi-task

learning compared to the single-task learning.

For example, if there is something in common be-

tween the artistry of painting and the artistry of sculp-

ture, the score of the multi-task learning will be bet-

ter than that of the single-task learning in the artistry

identiﬁcation task for painting and sculpture. That is,

the depth of the relationship of artistry in each art ﬁeld

can be measured by observing the degree of perfor-

mance improvement of multi-task learning with re-

spect to single-task learning.

Figure 7: Network structure for single-task learning.

Figure 8: Network structure for multi-task learning.

In this research, we take up four ﬁelds of art, paint-

ing, architecture, sculpture, and garden, and analyze

the relationship of artistry in these four ﬁelds by using

multi-task learning.

4 MULTI-TASK LEARNING FOR

ARTISTRY IDENTIFICATION

The network structure of multi-task learning used in

this research is as shown in Fig. 5. As shown in Fig. 5,

the network takes images of painting, architecture,

sculpture, and garden as inputs and the magnitude of

their artistry as outputs. In this research, the magni-

tude of artistry is considered to be a binary value with

or without artistry, and so the output of the network is

set to 0 or 1.

Since the appearance of art works in each ﬁeld is

very different, we ﬁrst input the image to the indi-

vidual feature extraction layers that extract the unique

features in each art ﬁeld, as shown in Fig. 5. Since it

is considered that the artistry that has become sufﬁ-

ciently abstract has features that are common among

multiple ﬁelds, the output from the individual feature

extraction layers is passed to the shared common lay-

ers. Since the ﬁnal identiﬁcation of artistry may dif-

fer in each ﬁeld, the output from the common layers

is passed to the individual output layers, and ﬁnally

the presence or absence of artistry is output as 0 or 1.

In addition, since some abstracted artistry is common

and some are unique, the individual feature extraction

layers exist in parallel with the common feature ex-

traction layers as shown in Fig. 5.

Seeing the Differences in Artistry among Art Fields by using Multi-task Learning

611

Table 1: Criteria for judging artistic or non-artistic.

artistic work non-artistic

Pictures Works of famous painters Amateur work

Architecture Works of famous architects Private houses, apartments, buildings

Sculptures Works of famous sculptors Amateur work

Gardens Famous garden Private house garden

Figure 9: Examples of non-artistic paintings.

Figure 10: Examples of artistic painting.

Figure 11: Examples of non-artistic architecture.

Figure 12: Examples of artistic architecture.

Table 2: Accuracy of artistry identiﬁcation.

painting architecture sculpture garden

single-task learning 0.6834 0.8353 0.7280 0.8825

multi-task learning 0.7128 0.8518 0.7482 0.9295

For training the proposed network, we use super-

vised learning with the ground truth of artistry of each

work. Let y

be the output obtained by inputting the

image x of class c in task t to the multi-task network

f as follows:

= f(x) (1)

When the ground truth of y

in that image is ˆy

, the

loss function of each task can be deﬁned by using the

multi-class cross entropy as follows:

Loss

= −

∑

log ˆy

(2)

When we have T tasks in the multi-task learning,

the multi-task learning can be realized by solving the

minimization problem that minimizes the following

multi-task loss:

Loss

multi

∑

t=1

Loss

(3)

where, λ

(t = 1, ··· , T) are hyperparameters.

We next explain the detail of our network. In

this research, we use the combination of Convolution,

Batch Normalization, ReLU, and MaxPoolingas a ba-

sis block of CNN (Fukushima and Miyake, 1982) as

shown in Fig. 6, and the network is constructed by

combining the basis blocks.

We want to evaluate the effectiveness of multi-

task learning when there is a commonality in artistry

among multiple art ﬁelds, so we build a network

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

612

Figure 13: Examples of non-artistic sculpture.

Figure 14: Examples of artistic sculpture.

Figure 15: Examples of non-artistic garden.

Figure 16: Examples of artistic garden.

painting

architecture

sculpture

garden

Figure 17: Changes in accuracy of artistry identiﬁcation.

for multi-task learning and a network for single-task

learning respectively for comparison.

The network structure for single-task learning is

as shown in Fig. 7, and the network structure of multi-

task learning is as shown in Fig. 8.

As shown in Fig. 8, the multi-task learning net-

work has individual blocks for each art ﬁeld and com-

mon blocks common to all art ﬁelds. In this network,

the feature extraction is performed ﬁrst for each art

ﬁeld as in the case of single-task learning, and then

it is divided into individual blocks and a shared com-

mon blocks. The output of these blocks is input to the

individual dense network as in the case of single-task

learning.

5 DATASET

In this research, we use images from four art ﬁelds of

painting, architecture, sculpture, and garden as artis-

tic works. Since the ground truth value of artistry,

i.e. artistic or non-artistic, is very difﬁcult to ﬁx, we

set the ground truth value of artistry of each work

whether it is made by famous artist or not (Sachant

et al., 2016).

Table 1 shows the criteria for judging artistic or

non-artistic in each art ﬁeld. In the case of paintings

and sculptures, those created by famous artists are

considered as artistic works, and those created by am-

ateurs such as students are considered as non-artistic.

Seeing the Differences in Artistry among Art Fields by using Multi-task Learning

613

Figure 18: Painting

Figure 19: Architecture.

In the case of architectures, the works of famous ar-

chitects are considered as artistic works, and ordinary

private houses, apartments, and buildings are consid-

ered as non-artistic. In the case of gardens, the famous

gardens are considered as artistic, and the gardens of

ordinary private houses are considered as non-artistic.

The examples of artistic works and non-artistic

works collected based on these criterion in paint-

ing, architecture, sculpture, and garden are shown in

Figs. 9 to 16. We have collected 1000 images for

artistic work and 1000 images for non-artistic work

in each art ﬁeld. We applied data augmentation to

these images, and obtained 20000 images for artistic

work and 20000 images for non-artistic work in each

art ﬁeld. Of these, 30000 were used for training and

10000 were used for testing in each art ﬁeld.

6 EXPERIMENTS

We next show the experimental results of identify-

ing artistry. In this experiment, both the multi-task

learning network and the single-task learning network

were trained with 150 epochs using images on artis-

tic and non-artistic work shown in section 5, and their

Figure 20: Sculpture.

Figure 21: Garden.

Table 3: Number of parameters in each learning method.

parameter

single-task learning (1) 9,700,722

multi-task learning (1) 26,699,634

multi-task learning (2) 9,184,626

multi-task learning (3) 4,694,898

accuracy of artistry identiﬁcation was compared using

test images.

6.1 Accuracy of Artistry Identiﬁcation

Fig. 17 shows the changes in accuracy of artistry iden-

tiﬁcation for the test data in multi-task learning and

single-task learning respectively, and table 2 shows

the results of the maximum identiﬁcation rate for each

ﬁeld in each learning method. From these results,

we ﬁnd that the accuracy of artistry identiﬁcation is

higher in multi-task learning in all art ﬁelds. The re-

sults indicate that the artistry of these four art ﬁelds is

related to each other. Especially in garden, the multi-

task learning can improve the identiﬁcation rate dras-

tically compared to the single-task learning, indicat-

ing that it is more relevant to other ﬁelds than paint-

ing, architecture and sculpture.

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

614

painting

architecture

sculpture

garden

Figure 22: Comparison of identiﬁcation rate.

Table 4: Accuracy of artistry identiﬁcation.

painting architecture sculpture garden

single-task learning (1) 0.6834 0.8353 0.7280 0.8825

multi-task learning (1) 0.7128 0.8518 0.7482 0.9295

multi-task learning (2) 0.6957 0.8650 0.7820 0.9314

multi-task learning (3) 0.7086 0.8482 0.7490 0.9167

6.2 Visualization of Attention in

Artistry Identiﬁcation

Since we were able to learn a classiﬁer that discrimi-

nates the presence or absence of artistry, we next vi-

sualize the attention in the artistry identiﬁcation task

by using GradCAM (Selvaraju et al., 2017). In partic-

ular, we visualize which part of the work the artistry

identiﬁer focuses on to judge the presence or absence

of artistry. We also compare whether there is a differ-

ence in the place of interest between single-task learn-

ing and multi-task learning.

In the artistry identiﬁcation networks, the output

of the ﬁnal convolution layer is considered to rep-

resent the most important feature in identiﬁcation,

so the ﬁnal convolution layer of block 5 is used for

single-task learning, and the ﬁnal individual layer and

the ﬁnal common layer are used for multi-task learn-

ing to visualize attention by GradCAM.

The output result of GradCAM is shown in Fig. 18

to Fig. 21. From these ﬁgures, we ﬁnd that in paint-

ings, the artistry is identiﬁed mainly by focusing on

the person. In the case of architecture, the artistry

identiﬁer focuses on a single limited place in single-

task learning, but in multi-task learning, the points

of interest are distributed throughout. In the case of

sculpture, it can be conﬁrmed that the points of inter-

est are sparsely distributed throughout both in single-

task learning and multi-task learning. The range of

attention is larger in multi-task learning as in the case

of architectures. Finally, in the case of garden, we

ﬁnd that the artistry identiﬁer was paying attention to

plants and stones. From these results, it is consid-

ered that the multi-task learning pays more attention

to the entire work than the single-task learning, and

this property of multi-task learning makes it possible

to judge the presence or absence of artistry more ac-

curately.

6.3 Number of Parameters

We next evaluate the change in accuracy when the

number of parameters is changed.

In the experiment in section 6.1, the number of

parameters for multi-task learning was larger than the

number of parameters for single-task learning. There-

fore, we next evaluate the accuracy of artistry identiﬁ-

cation in the case where the number of parameters for

multi-task learning is equal to or less than the number

of parameters for single-task learning.

The number of parameters for each learning

method is as shown in table 3. Single-task learning

(1) and multi-task learning (1) are the networks used

in the previous experiment. The multi-task learning

(2) uses a network with the same number of param-

eters as single-task learning (1), and the multi-task

learning (3) uses a network with fewer parameters

than single-task learning (1). The accuracy of artistry

identiﬁcation by these networks are shown in table 4

and Fig. 22.

As shown in table 4 and the graph in Fig. 22,

we ﬁnd that the artistry identiﬁcation rate is higher

in multi-task learning (2) than in other cases and the

number of parameters should not be too large in the

multi-task learning.

7 CONCLUSION

In this paper, we proposed a method for analyzing the

relevance of artistry among multiple art ﬁelds by us-

ing deep neural networks. In particular, we showed

Seeing the Differences in Artistry among Art Fields by using Multi-task Learning

615

that by using the multi-task learning, the relevance

of multiple art ﬁelds can be analyzed quantitatively.

In our experimental results, we showed that the ac-

curacy of artistry identiﬁcation becomes higher in the

multi-task learning. The results show that the artistry

in different art ﬁelds is related to each other.

REFERENCES

Anwer, R. M., Khan, F. S., van de Weijer, J., and Laak-

sonen, J. (2016). Combining holistic and part-based

deep representations for computational painting cate-

gorization. In Proc. International Conference on Mul-

timedia Retrieval, pages 339–342.

Bianco, S., Mazzini, D., Napoletano, P., and Schettini, R.

(2019). Multitask painting categorization by deep

multibranch neural network. Expert Systems with Ap-

plications, 135:90–101.

Caruana, R. (1997). Multi-task learning. Machine Learn-

ing, 28:41–75.

Elgammal, A., Liu, B., Elhoseiny, M., and Mazzone, M.

(2017). Can: Creatice adversarial networks generat-

ing ”art” by learning about styles and deviationg from

style norms. arXiv: 1706.07068.

Fukushima, K. and Miyake, S. (1982). Neocognitron: A

new algorithm for pattern recognition tolerant of de-

formations and shifts in position. Pattern Recognition,

15(6):455–469.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,

Warde-Farley, D., Ozair, S., Courville, A., and Ben-

gio, Y. (2014). Generative adversarial nets. In

Advances in neural information processing systems,

pages 2672–2680.

Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. (2017).

Image-to-image translation with conditional adversar-

ial networks. In The IEEE Conference on Computer

Vision and Pattern Recognition (CVPR), pages 1125–

1134.

Kendall, A., Gal, Y., and Cipolla, R. (2018). Multi-task

learning using uncertainty to weigh losses for scene

geometry and semantics. In Proc. IEEE Conference

on Computer Vision and Pattern Recognition (CVPR).

Liu, S., Johns, E., and Davison, A. J. (2019). End-to-end

multi-task learning with attention. In Proc. IEEE Con-

ference on Computer Vision and Pattern Recognition

(CVPR), pages 1871–1880.

Picasso, P. Weeping woman.

Sachant, P. J., Blood, P., LeMieux, J., and Tekippe, R., edi-

tors (2016). Introduction to Art:Design, Context, and

Meaning. University of North Georgia.

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R.,

Parikh, D., and Batra, D. (2017). Grad-cam: Visual

explanations from deep networks via gradient-based

localization. In ICCV, pages 618–626.

Tan, W. R., Chan, C. S., Aguirre, H. E., and Tanaka, K.

(2016). Ceci n’est pas une pipe: A deep convolutional

network for ﬁne-art paintings classiﬁcation. In Proc.

International Conference on Image Processing.

The Metaphysics Research Lab (1995). Stanford encyclo-

pedia of philosophy.

van Gogh, V. W. Still life - vase with ﬁfteen sunﬂowers.

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

616