Semantic Search with Combination Impression and Image Feature

Query

Desi Amirullah and Lipantri Mashur Gultom

Department of Informatics Engineering, State Polytechnic of Bengkalis, Bengkalis, Indonesia

Keywords: Semantic, Image Search, Combination Query.

Abstract: In this study we propose a new approach in the semantic image search system, namely the combination of

images and impressions query that display results in the form of images that contain impressions. In previous

studies, focusing on semantic image search using combining color and shape features into a single query

which displays the results of images and impressions, and semantic search impression with mecahanism

weighting combination color feature and image feature as query. In this study, the dataset images that we used

were 142 Malay songket motifs, and each songket motif contained impressions. In this study, the dataset and

image query were extracted into metadata using the Hu Moments Invariant method. Based on the results

obtained, we can conclude a success rate of 53% by testing of 10 query combinations.

1 INTRODUCTION

Songket Malay or always known as Songket Siak, and

Songket Bukit Batu, is one of the handicrafts that has

existed for generations in Riau. To maintain the value

of traditional customs that are expected to songket

craftsmen to be able to understand the meaning

contained in every variation of motifs so that the

songket fabric that is made still has a high

philosophical value. At this time information about

the meaning of each songket motif is very limited,

causing the next generation songket craftsmen do not

know the information and cultural values contained in

every songket motif used (Amirullah, Barakbah, &

Basuki, 2015; Amirullah, 2018). Songket craftsmen

should be able to understand well the meaning

contained in every color and shape that is included in

every songket motif they create, so as not to cause

mistakes in the rules of combining motifs on songket

fabric which can result in the resulting songket fabric

being meaningless (Amirullah, 2018).

The development of science in the field of image

processing and artificial intelligence is the main basis

in this research. In previous research, it can be seen

that there are 142 Riau Malay songket motifs,

consisting of several basic colors that also contain

impressions (Amirullah, Barakbah, & Basuki, 2015;

Amirullah, 2018). Impression is a term of the

meaning contained in every image of the Songket

motif. In this system that we propose, focus on

searching songket motif images by querying the

combination (Dinakaran, Annapurna, & Kumar,

2010) of image shape features and impression

features, where the image dataset and image query are

extracted into metadata using the Hu Moments

Invariant method. In this search system presented

images which contained impression.

2 RESEARCH METHODS

In the first stage, the research method carried out as

in the previous research is creating the Impression

Metadata, based on our previous research, which is

conducting research to get impressions on each Motif

Songket Image, from 142 Motif Songket Images, we

found 27 Impressions, and in each image there are 2

to 3 impressions, so that the 142 x 27 metric is created

(Amirullah, Barakbah, & Basuki, 2015; Amirullah,

2018). From this impression data, we do the metric

multiplication with the metadata results from the

shape extraction in each Motif Songket Image, thus

creating a new metric that is 142 x 7.

Search system that uses a combination of image

shape feature and impression feature as queries,

which starting from extracting the Query Image

feature, then extracting the we count similarity with

the impression feature on the database to get the

Amirullah, D. and Gultom, L.

Semantic Search with Combination Impression and Image Feature Query.

DOI: 10.5220/0010352501050109

In Proceedings of the 3rd International Conference on Applied Engineering (ICAE 2020), pages 105-109

ISBN: 978-989-758-520-3

105

impression query options that are the most similar/

closest with the query image entered first, then the

user selects the impression query that is displayed,

then the system takes the metadata from the

impression query and retrieves the extracted metadata

from the query image, then multiplies the metric to

create a new query that is a combination of Image

form features and features Impression, the last

measurement of similarity between the query results

in combination with the image dataset in the database

and displays the results in the form of images. For

more details, can be seen in the following system

design drawings.

Semantic search system stages with a query

combination of image shape features and impression

features:

The user input a query image and

impression.

Based on the query image shape feature, the

system measures the similarity to the shape-

impression metric and displays the 10

closest impressions.

Based on the closest impression, the user

then takes one of 10 impressions for the

impression query.

From the query image shape feature and the

impression- shape feature a metric

multiplication process is performed to

combine the two queries into a single/

combination query.

Furthermore, using the cousine similarity

measurement method, the system will

measure the similarity between the query

combination with the dataset's image shape

feature, and display the 10 closest / most

similar songket motif images.

Finally, we analyze the accuracy of search

results with combination queries.

Figure 1: Semantic Image Search System Design with

queries combination of shape features image and feature

impression.

3 FEATURE EXTRACTION

3.1 Impression Feature Extraction

Most of the general public does not understand the

impression contained in the philosophy in the shape

of rhymes and the meaning of the background color

of each motif, therefore in previous studies we carried

out the impression extraction to be easily understood

(Amirullah, Barakbah, & Basuki, 2015). The results

of the impression extraction create a metric 142 x 27.

142 is the number of images, and 27 is the number of

impression dataset (Amirullah, Barakbah, & Basuki,

2015).

Figure 2: The extraction Impression of Songket Motif.

3.2 Shape Feature Extraction

Extraction of shape features in the image of each

songket motif in the dataset and image query using

the Hu Moment invariant method. This method is

used because it has no effect on rotation, scale and

translation, and produces seven equations

(Amirullah, Barakbah, & Basuki, 2015; Amirullah,

2018). In the feature extraction process of the image

shape using the Hu Moments invariant method

produces metadata metrics based on the number of

images (142 images) and the number of equations (7

equations). Definition The basic equation of object

moments with the Hu Moment Invariant (Amirullah,

Barakbah, & Basuki, 2015; Amirullah, 2018, Ming-

Kuei, 1962). Method is as follows.

(1)

(2)

(3)

(4)

(5)

Then the determine central moment and process

of normalizing the central moment by using the

following equation.

ICAE 2020 - The International Conference on Applied Engineering

106

(6)

(7)

Furthermore, building features of momentary

forms are invariant in object recognition and do not

affect the translation, scale, and rotation of images.

The invariant moment equation is as follows (Ming-

Kuei, 1962).

(8)

(9)

(10)

(11)

(12)

(13)

(14)

3.3 Query Combination Image and

Impression

Query combination of shape features with impression

features are the main topic of the system proposed in

this study. To combine these two different features,

we use the concept of metric multiplication between

image shape features with the impression shape

feature. The results of this metric multiplication

creates the new metric is 1 x 7, and this is a

combination query image query with an impression.

More details as visualized in the Figure 3.

Figure 3: Query Combination Image and Impression

Process

3.4 Similarity Measurement

The measurement of the similarity between the results

of the multiplication of the query metric on the

impression metadata and images with the images

metadata in the dataset is carried out using the cosine

method [1], [2] as follows.

(15)

3.5 Precision Calculation

To calculate the precision for this experiment, we use

a score calculation equation (Amirullah, 2018). The

process begins by measuring the similarity of the

query impressions to the impressions on the image

search results, by giving the weight value to the first

result with a value of 10 and the final value being 1,

then calculating the average for each experimental

process. The equation is as follows (Amirullah,

Barakbah, & Basuki, 2015; Amirullah, 2018).

(16)

The process of score weighting and calculating

the average in the above equation, more details as

shown in the following table.

Table 1: Weighing and calculating average score of system

performance.

Query

Combination

Result

Score

Result

Image

Impression

Selected

K ewibaw--- -

aan

K ewibawaan,

Kerukunan,

Tenggang_Ras

K ewibawaan,

Amanah,

Bijaksana,

Cerdik_Pandai

K ewibawaan,

Ketulusan,

Sopan_Santun

K ewibawaan,

Kerukunan,

Ketaqwaan

ewibawaan,

6 6

Sopan_Santun

Kemakmuran,

Tahu_Diri,

Sopan_Santun

Semantic Search with Combination Impression and Image Feature Query

107

K ewibawaan,

Sopan Santun

4 4

Persaudaraan,

Tenggang_Ra

sa,

Rendah_Hati

K ewibawaan,

Saling_Mengh

ormati

Kebahagiaan,

Sejahtera,

Rendah Hati

Total Score/ Total Score

Result :

49 40

4 RESULTS AND DISCUSSION

On the experimentation process, user inputs image

query and selects one a query impression by result

process image query, and displays the results in the

image of Motif Songket that contained impression.

Based on 10 experiments that we carried out in the

above manner and stages, the results can be known as

in Table 2.

Table 2: Semantic search experiments with image and

impression combination queries

Query Combination

Total

Score

Result

Image Query

Impression

Selected (1 of 10)

Persaudaraan

Kemakmuran

Murah Rezeki

Kesuburan

Rendah Hati

Kerukunan 17

7 Ketulusan 19

8 Kerukunan 12

9 Sopan Santun 34

10 Kewibawaan 46

Average : 29,2

Percentage Acuration : 53%

In the above Experiments conducted 10 times, we

can see that the results of the highest and lowest

scores. And calculated the average results of this

experiment is an average score is 29.2 of 49, and the

percentage is 53% of 100.

5 CONCLUSIONS

This semantic image search system with image and

impression queries the development of previous

research, namely semantic search with impressions

based on the extraction of color and image features,

the merging of color and shape features, with the

weighting mechanism of colors and shapes. The new

proposal in this study is to use a combination of shape

and impression query features. In the process of

getting the most suitable / close to the query semantic

results we do the search process by measuring the

similarity of image and impression queries to the

shape-impression features contained in the dataset,

thus displaying 10 image search results that contain

the closest impressions. In the experiments we

conducted, it was seen that the result of the average

calculation of the experiment was an average score of

29.2 with a success percentage of 53%.

REFERENCES

Amirullah, D., 2018. Sistem Pencarian Semantik Impresi

dengan Mekanisme Pembobotan Kombinasi Fitur

Warna dan Fitur Bentuk, INOVTEK Polbeng - Seri

ICAE 2020 - The International Conference on Applied Engineering

108

Informatika, vol. 3, no. 1, p. 41, 2018, doi:

10.35314/isi.v3i1.332.

Amirullah, D., Barakbah, A. R., Basuki, A., 2015. Semantic

Songket Image Search with Cultural Computing of

Symbolic Meaning Extraction and Analytical

Aggregation of Color and Shape Features. EMOTTER

International Journal of Engineering Technology, vol.

3, no. 1, pp. 115–132, 2015, doi:

10.24003/emitter.v3i1.37.

Dinakaran, B., Annapurna, J., Kumar, C. A., 2010.

Interactive Image Retrieval Using Text and Image

Content. Cybernetics and Information Technologies,

vol. 10, no. 3, pp. 20–30.

Malik A. T. A., Effendy, T., Yunus, H., 2004. Corak Ragi

dan Tenun Melayu Riau, Balai Kajian dan

Pengembangan Budaya Melayu. AdiCita. Yogyakarta.

Ming-Kuei, H. 1962. Visual pattern recognition by moment

invariants. IRE Transactions on Information Theory,

pp. 66– 70.

Semantic Search with Combination Impression and Image Feature Query

109