Search Result Summaries Improved by Structure and Multimedia

Brent Wenerstrom and Mehmed Kantardzic

Department of Comp. Eng. and Comp. Sci., University of Louisville, Louisville, Kentucky, U.S.A.

Keywords:

Structured, Automatic, Summaries.

Abstract:

We previously introduced ReClose which provides summaries with both better content and better visual dis-

play for search engine results. We now seek to further improve summaries with the addition of structured

text and multimedia, more speciﬁcally tables, lists, buttons and images. Currently search engine provided

summaries rarely use structured text and images. We show in this paper that structured text and images lead

to faster comprehension by search engine users and lead to visually more appealing summaries. 70% of non-

expert users made decisions more quickly using summaries preserving document structure and 65% of all

users preferred summaries preserving structure to plain text summaries.

1 INTRODUCTION

The inspiration for this study comes from the jumbled

mess of text that is shown to a user when summary

text is extracted from a table. For example consider

Figure 1(a) in which the summary text shows a ta-

ble of football players invited to the NFL combine.

A search engine user may correctly guess that the

text comes from a table, or one may guess it is only

comma separated values shown in the original docu-

ment. Compare now the impression given by viewing

the same text as a table as the original author of this

text meant it to be viewed as in Figure 1(b). In the

summary preserving structure there is no mistaking

the authors intent of the summary text.

We previously proposed our system ReClose

(Wenerstrom and Kantardzic, 2011b) for summary

generation. ReClose improved summary generation

through a two part summary including query-biased

and query-independent portions. This was further en-

hanced by adding color depth to keywords (Wener-

strom and Kantardzic, 2011a). Together these en-

hancements to summarization led users to be 20%

more accurate when selecting relevant documents

using ReClose summaries compared to using using

Google summaries. We propose to extend ReClose

by incorporating document structure into summaries.

In addition to tables we also explore the usage of

lists, images, buttons, text ﬁelds and hyperlinks. In

the ﬁrst example (Table 1) the purpose of the sentence

“iPhone iPad Android” to be three radio buttons so

that one may search based on different devices.

From Figures 1(a) and 1(b) we see that structured

Ofﬁcial Invite — NFL Combine

www.nﬂcombine.net/players/ofﬁcial-invite-list

jr: player: goes by: school: camp # pos: acho,

emmanuel: texas: lb01: ob: adams, adam: joe:

arkansas: wo01: wo: adams, michael: mike: ohio st:

ol01: ot: alecxih ...

(a) Traditional search engine result observed in Bing.

Ofﬁcial Invite — NFL Combine

www.nﬂcombine.net/players/ofﬁcial-invite-list

—Player —Goes by—School —Camp #—Pos—

—ACHO, EMMANUEL— —TEXAS —LB01 —OB —

—ADAMS, ADAM —JOE —ARK. —WO01 —WO—

—ADAMS, MICHAEL —MIKE —OHIO ST—OL01 —OT —

—ALECXIH, — — — — —

(b) Summary preserving document structure example.

Figure 1: Search engine summary with (b) and without (a)

structure.

text and multimedia adds insight not presentin current

search result summaries. We propose the addition of

structured text and multimedia to current search re-

sults which will have the following beneﬁts for users:

• Additional semantics added to search results.

• Users will have a better understanding of the sum-

marized web pages.

• Users will be better satisﬁed with search results.

This work offers two contributions. First it pro-

vides an alternative basic unit to automatic summa-

rization in search. Rather than all forms of summa-

rization being based only on text, this work expands

that view to include HTML entities such as tables and

lists that contain text. Second, this work provides ex-

181

Wenerstrom B. and Kantardzic M..

Search Result Summaries Improved by Structure and Multimedia.

DOI: 10.5220/0004117901810186

In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval (KDIR-2012), pages 181-186

ISBN: 978-989-8565-29-7

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

Table 1: Comparison of summaries from a traditional search engine versuse summaries with structure.

Buttons

and Text

Field

Chomp. Download Chomp iPhone Android.

iPhone iPad Android

Chomp Download Chomp iPhone Android

iPhone

iPad Android

Images and

Lists

tenerife property sw1. exclusive land for sale;

exclusive land for sale; exclusive land for sale;

exclusive land for ...

Tenerife Property SW1

• exclusive land for sale

• exclusive land for . ..

perimental validation that automatic summarization

systems which output summaries with structured text

and multimedia in addition to unstructured text out-

perform unstructured text alone.

2 SUMMARY FROM HTML

Tranditional search engine summaries coming from

the likes of Google, Yahoo and Bing generally are

made up of plain text. This text often is focused on the

usage of the searched keywords within a document.

In this paper we will mainly focus on HTML

documents, though the ideas extend to all types of

web documents. An HTML document is made up of

HTML tags and text content. These HTML tags pro-

vide text formatting, text structure, hyperlinks, meta

data about the document and more. Traditional search

engine summaries when extracted from web page

content only use the plain text content of the docu-

ment.

Traditional search engines currently extract text

from image descriptions encoded in the HTML tag of

images. When testing the usefulness of multimedia

and structure we replace text extracted from images,

with the original image.

HTML tables may be used to align text or to dis-

play tabular data. The classiﬁcation of HTML ta-

bles into these two categories is not herein described.

We retain table structure in our experiments for tables

found to display tabular data. When table data is ex-

tracted without structure meanings can be lost and the

true value of the content may become fuzzy as was

observed in Figure 1.

Text taken from a list without the list structure

may in fact change the meaning of the content. For

example on a web page describing CMMS software

two bullet points were observed as “work order man-

agement” and “preventative maintenance”. When the

structure of the list was lost through a search engine’s

extraction of content for summary, it was no longer

clear that there were two separate entities. On ﬁrst ob-

servation it was difﬁcult to understand what “work or-

der management preventative maintenance” was. Pre-

serving lists in summaries retains the original mean-

ing of web page authors.

When text appears on buttons or hyperlinks it has

different meaning compared to plain text. For ex-

ample a button that reads “Sign Up” shows that the

action is immediately available, while in plain text

where and how to sign up are not apparent. Hyper-

links show that there is more information available on

a linked document. Some web pages provide value by

linking to high quality content. When a web page is

summarized and text is not shown as a hyperlink the

value of these linking documents is lost on the search

engine user. We preserve buttons and hyperlinks in

summaries.

2.1 HTML Parsing

For the purpose of experimentation we take previ-

ously selected text from a web page and injects struc-

ture and multimedia elements found in the original

HTML web page. We do this by parsing the origi-

nal HTML web page into a convenient data structure

which contains a string representation of the available

text content. The chosen summary text is matched

against the string, then our data structure injects the

HTML that was found between and surrounding the

sequence of characters found to match.

Parsing an HTML web page involves using regu-

lar expressions as found in Java to loop through each

HTML tag and process the text content followed by

the HTML tag. Each HTML tag is processed to re-

move unnecessary formatting. The text content is also

preprocessed. The string representation of the text

content removes all non-letter, non-number charac-

ters and changes all letters to a uniform lower case.

Search engines may use a different letter case than

KDIR2012-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval

182

the original web page, add or remove space charac-

ters and change punctuation. One reason punctuation

is changed by the search engines is to simulate table

or list boundaries found in the HTML.

When a substring of text content matches the cho-

sen summary text, then a new summary is produced

by going character by character and adding to a buffer

the characters referenced by that character informa-

tion object. If the character is the ﬁrst character,

then preceding non-letter, non-number characters are

added ﬁrst, like a starting quotation mark. Then each

middle character adds what characters and tags may

lie between each of them, such as space characters

and HTML tags that we intend to reproduce. When

the last character is reached, than ending context is

also added like the ending punctuation. This may be

repeated if multiple text spans are found, in which

case ellipses (...) are added between text spans.

This whole process produces near identical text to

the text submitted with the addition of HTML tags

that provide text structure and multimedia.

3 RELATED WORKS

The research community has proposed numerous ap-

proaches to displaying search results. Some of these

approaches have been more radical such as docu-

ments appearing in a 2-dimensional particle space

(Kelkar et al., 2009) or using word clouds to represent

results (D¨ork et al., 2009). While other approaches

have aimed to make small changes to the summaries

such as adding images and/or lists to summaries (Joho

and Jose, 2008).

Several studies have focused on the inclusion

of images in search results. (Xue et al., 2008)

found users had higher precision and recall scores

when labeling search results using summaries that

include both summary text and images. (Li et al.,

2008) showed users to be 30% faster at informa-

tional searches when search results included images.

(Dziadosz and Chandrasekar,2002) found users using

thumbnails took longer to make relevance decisions

but were more accurate.

Other approaches have been proposed at provid-

ing context beyond thumbnails and images. (Wen-

erstrom and Kantardzic, 2011b) proposed a two part

text summary, named ReClose, which includes both

the traditional query-biased summary and also a web

page summary, providing additional context. This Re-

Close system was then enhanced with color-coding

(Wenerstrom and Kantardzic, 2011a). The extent

of keyword usage and warnings about topic depar-

ture were encoded with words highlighted in different

shades. The result was the color-enhanced ReClose

summaries leading to users that were 20% more ac-

curate than when using Google summaries.

White et al. (White et al., 2001) add the number of

links, name of ﬁrst non-text object and the size of the

document to search results. D¨ork et al. (D¨ork et al.,

2009) proposed visual widgets for representing search

results. These widgets included maps, timelines and

word clouds.

4 USER EVALUATION

We tested the effectivenessof structured text and mul-

timedia on a number of search engine users. The eval-

uation process was made up of two major steps. The

ﬁrst step involved users reviewing 30 search results

one at a time. Each search result displayed a unique

search query, search question and search engine sum-

mary. The user had three options to use when an-

swering if they “should” click on a result: yes, no or

unsure. Responses and timing information was saved

per summary. The second step asked which system a

user preferred with the only options being the sum-

maries preserving document structure or traditional

summaries.

The search queries used were taken from two main

sources. First 150 queries were obtained from the

website SearchHippo

on Feb. 14 and 22. Second

we obtained the top 20 trending queries on Google

Hot Trends

on Feb. 22, 23, 24 and 28. From these

queries, duplicates and most non-English queries

were removed. Our ﬁnal count was 185 queries.

All 185 queries were submitted to a traditional

search engine through Bing’s Search API

. We ob-

served the URLs and summaries for the top 40 search

results for each of the 185 queries and downloaded the

linked web pages. We were able to download 7,214

web pages. Of these results only 440 summaries cur-

rently contained one of the target HTML formating

tags. From these 440 we removed those where the

formatting would not be noticeable, i.e. small images,

blank images, tables used for text placement or small

tables. We then randomly chose 30 summaries from

each of four groups: forms, tables, lists and images.

There were only 10 usable form summaries. Some

of the summaries used HTML from multiple groups.

The ﬁnal pool of potential summaries included 94

summaries.

Each user within our evaluation was shown 30

summaries randomly selected from our pool of 94.

http://www.searchhippo.com

http://www.google.com/trends/hottrends

http://www.bing.com/toolbox/bingdeveloper/

SearchResultSummariesImprovedbyStructureandMultimedia

183

15 of these summaries were only made up of text and

came directly from Bing. 15 of the summaries were

shown to the users with structured text and/or mul-

timedia present from the original web page. The 30

search results were randomly ordered on a single page

mixing the ordering of Bing summaries with struc-

tured summaries.

We used two pools of people expecting poten-

tially different responses from each group. Our ﬁrst

group was made up of those who effectively think

in the abstract. The ﬁrst pool is made up of 22 stu-

dents from the CECS department at the University of

Louisville. CECS students can be effective at view-

ing source code and placing that source code in men-

tal models of a system. This form of abstract thinking

we do not expect to be common in the general pop-

ulation. The second group, who represent the more

general population, we obtained by sending out ap-

proximately 70 invites through Facebook to acquain-

tances and friends of friends. Our invites resulted in

34 participants for our user evaluation.

The user evaluation was posted online. Users

completed the evaluation using their own computers

outside of a lab setting to better recreate more natural

search engine usage among our subjects.

5 EXPERIMENTAL RESULTS

We evaluated the effectiveness of structured text and

multimedia within search summaries in three dimen-

sions: user time, user accuracy and user preferences.

The values in each of these dimensions have provided

evidence for the usage of structured text and multime-

dia.

5.1 User Decision Time

The more quickly a user can ﬁnd desired answers or

resources the more successful the search engine has

been and the more satisﬁed search engine users will

be. The more quickly the decision can be made with-

out losing accuracy, the better quality the results are.

From the user evaluation conducted, we have obtained

1650 click decision. Each click decision provides in-

formation on whether a user would click or not and

how long it took to decide.

From the results we see a broad range of timings

for each click decision from 1.3 seconds to 2.8 hours

and a median time of 13.3 seconds. The 2.8 hour click

decision time is without a doubt an outlier and was

removed from our time analysis data. We also re-

moved from our time analysis the other 15 timings

with values higher than 78 seconds, which were rare

0 20 40 60 80

Time (s)

0 20 40 60 80

Time (s)

Participants

0 20 40 60 80

Time (s)

0 20 40 60 80

Time (s)

Participants

Figure 2: Box plots of the participants with the ﬁve lowest

median timings and the ﬁve highest median timings.

(16/1650), seem to be extreme values for the deci-

sions made and are more than 3 standard deviations

from the mean.

We see large variations in the timings of each in-

dividual. Consider Figure 2 where we show box plots

of the users with the ﬁve smallest medians and the ﬁve

largest median timings. Here we see that there are in-

dividuals where the majority of their timings fall be-

low 20 seconds. While there are others that carefully

consider their optionsand have a majority of their tim-

ings over 20 seconds. This shows that it is impor-

tant to consider the variation in time due to individual

speed. This is countered by having all users recorded

making decisions for both types of summaries.

When we separate out the two groups of expe-

rienced (computer science group) and inexperienced

(Facebook group) abstract thinkers, we see very dif-

ferent timings. In both groups we compared the tim-

ings when using structured summaries to text only

summaries. We did this by comparing averages. To

avoid the long right tail heavily impacting averages,

we ﬁrst took the logarithm.

First we compare decision times in the computer

science group. This is shown in Figure 3(a). The

square in this chart represents the average decision

time when users are using plain text summaries. A

line is drawn from each square to the average deci-

sion time for structured summaries. If the line goes

up, then plain text summaries were faster for that user.

In the computer science group 9 users were faster us-

ing structured summaries, while 13 were faster using

plain text summaries. From Figure 3(a) we can see

that for most users there was not a big difference be-

tween the two. However, there is a noticeable differ-

ence for the slowest two and the fastest 7. The fastest

group or the 7 with the lowest average decision time,

every one of them was faster with plain text sum-

maries. For the powerusers who make blink decisions

using abstract thinking, they were much faster using

plain text summaries. Except for a couple extreme

cases little other observed differences were present.

KDIR2012-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval

184

5 10 15 20

10 20 30 40

Sorted User IDs

Avg. Decision Time (s)

(a) Computer science students.

0 5 10 15 20 25 30

10 20 30 40

Sorted User IDs

Avg. Decision Time (s)

(b) Facebook group.

Figure 3: The average decision time of users using plain-text summaries are drawn as boxes. A line is drawn from each box

to the average decision time for summaries using structured text.

A paired t-test was performed to check for a statisti-

cal difference, and with a p-value of 0.11, none was

found.

We see a very different effect among the Face-

book pool, see Figure 3(b). A large majority of

the Facebook pool were faster using structured sum-

maries or 23 of 33. Many of the differences were

very small, but there are more noticeable lines point-

ing down in Figure 3(b). First we compared the dif-

ference in values using a paired t-test and there was

not a signiﬁcant difference at the alpha=0.05 range

with a p-value of 0.096. There is a signiﬁcant differ-

ence (p-value=0.041) using a Wilcoxon signed rank

test pairing the averages of users. Visually, it ap-

pears that slower users were most positively impacted

using structured summaries among the inexperienced

abstract thinkers of the Facebook pool.

5.2 User Accuracy

When a user clicks a link that leads to an irrelevant

web page, that user has wasted time and this wasted

time may negatively reﬂect on a search engine’s re-

sults. When judging the impact of structured sum-

maries on decision making we needed to judge the

accuracy of the click selections. We had a judge go

through and create ﬁrst the context of each query such

as for the query katherine jenkins the query question

was “Who is Katherine Jenkins (singer)?” Decisions

were made based mainly on how well the web page

answered the question posed in the search question.

In the pool of summaries, nearly 50% are relevant and

50% are not relevant, meaning that guessing all rele-

vant would result in a low accuracy.

Accuracy was little affected by the introduction

of structure and multimedia in the summary. Accu-

racies by summary type and by group are shown in

Table 2. In this table we see that the more common

search engine user, those inexperienced in abstract

thinking were more accurate using structured sum-

maries, while the abstract thinkers were less accurate.

The inexperiencedabstract thinkers were signiﬁcantly

more accurate than was the experienced group using

Fisher’s exact test with a p-value of 0.038. Perhaps

the experienced abstract thinkers were overly conﬁ-

dent in their mental models, and took time to ignore

the introduced structure, leading to lower accuracies.

Table 2: Comparison of accuracies between summary types

and evaluation groups.

Pool Structured Plain Text

Experienced 65.3% 70.0%

Inexperienced 72.8% 71.7%

5.3 User Preferences

Overall 67% (37/55) prefer the summaries preserv-

ing document structure to plain text summaries. The

per group preferences are shown in Table 3. Ta-

ble 3 shows that 17 out of 22 (77.3%) experienced

abstract thinkers preferred the structured summaries.

At the same time 20 of 33 (60.6%) of the inexperi-

enced abstract thinkers preferred the structured sum-

maries. This is a much higher, but not signiﬁcant,

percentage of experienced abstract users preferring

structured summaries. Perhaps, experienced abstract

thinkers see the value in adding structure, though their

habits do not yet reﬂect it. Part of the inexperienced

pool does not like change and would prefer the sum-

maries to remain familiar, though they comprehend

them more quickly.

We received 38 comments, though it was optional,

providing a lot of feedback about users’ thoughts and

reactions. Tables and images received the most atten-

tion from users. Some users really liked table format-

ted data such as in the comment, “Tabulated data is

ALWAYS easier to read in table format...” One rea-

son for liking or disliking tables is the speed at which

SearchResultSummariesImprovedbyStructureandMultimedia

185

Table 3: Preferences of experienced and inexperienced eval-

uation groups. Bolded values are signiﬁcantly better than

50% using a binomial test and alpha of 0.05.

Pool Structured Plain Text

Experienced 77.3% 22.7%

Inexperienced 60.6% 39.4%

tables are comprehended. One participant wrote, “Be-

cause the information is organized in a table or has il-

lustrations, you can determine more quickly how rele-

vant the info is.” However, one participant felt that ta-

bles added additional mental work as was mentioned

before, “tables require me to decipher another layout.

It’s slower than just reading the information without

extra images, lists, etc.”

Images received a similar reception. Some users

really liked the images, “Also, the embedded images

were useful in determining the content of the web

page.” Another user felt that images were a big dis-

traction, “...however I noticed that the instant I saw

an image my eye jumped immediately to the url with

the image, bypassing the one I was viewing. I needed

words to be sure it was what I was looking for.”

Overall users preferred summaries when possible

to include structure and multimedia. As one user put

it, the summaries preserving document structure were

“More intuitive and easier to read and understand.”

There will always be those that are familiar with the

current approach and would prefer no change take

place, but these users queried in our evaluation over-

all would beneﬁt from the addition of structured text

and multimedia to search engine summaries.

6 CONCLUSIONS

In this study we introduced an extension to our cur-

rent search engine summary generation system, Re-

Close (Wenerstrom and Kantardzic, 2011b). This ex-

tension introduces the use of structured text and mul-

timedia in the form of HTML tables, lists, text ﬁelds,

buttons, hyperlinks and images. We compared these

additions experimentally to traditional summaries ob-

tained from Bing.

The results of our users evaluation suggest that

users will comprehend search results more quickly,

lose no accuracy and prefer the structured summaries

to plain text. From our users evaluation a signiﬁ-

cant number (23 of 33 or 69.7%) of inexperienced

abstract thinkers were faster using structured sum-

maries. While the experienced pool was faster (13/22

or 59.1%) using traditional summaries. No signif-

icant difference existed between the accuracies of

user choices between traditional summaries and struc-

tured, multimedia summaries. Overall a signiﬁcant

number (37 of 55 or 67.3%) of users preferred struc-

tured, multimedia summaries to the traditional, plain-

text summaries. We now plan to explore the use of

HTML entities as the basic unit of summary genera-

tion rather than text entities like sentences.

REFERENCES

D¨ork, M., Williamson, C., and Carpendale, S. (2009). To-

wards visual web search: Interactive query formula-

tion and search result visualization. In Workshop on

Web Search Result Summarization and Presentation.

Dziadosz, S. and Chandrasekar, R. (2002). Do thumbnail

previews help users make better relevance decisions

about web search results? In Proceedings of the 25th

Annual International ACM SIGIR Conference on Re-

search and Development in Information Retrieval, SI-

GIR ’02, pages 365–366, New York, NY, USA. ACM.

Joho, H. and Jose, J. M. (2008). Effectiveness of addi-

tional representations for the search result presenta-

tion on the web. Information Processing & Manage-

ment, 44(1):226 – 241. Evaluation of Interactive In-

formation Retrieval Systems.

Kelkar, S., John, A., and Seligmann, D. (2009). Visualizing

search results as web conversations. In Workshop on

Web Search Result Summarization and Presentation.

Li, Z., Shi, S., and Zhang, L. (2008). Improving relevance

judgment of web search results with image excerpts.

In Proceeding of the 17th international conference on

World Wide Web, WWW ’08, pages 21–30, New York,

NY, USA. ACM.

Wenerstrom, B. and Kantardzic, M. (2011a). Enhanced vi-

sualization for web-based summaries. In Very Large

Data Search (VLDS) 2011 First International Work-

shop on Searching and Integrating New Web Data

Sources, pages 38–43.

Wenerstrom, B. and Kantardzic, M. (2011b). ReClose:

Web page summarization combining summary tech-

niques. International Journal of Web Information Sys-

tems, 7:333–359.

White, R., Jose, J. M., and Ruthven, I. (2001). Query-biased

web page summarisation: a task-oriented evaluation.

In Proceedings of the 24th annual international ACM

SIGIR conference on Research and development in in-

formation retrieval, SIGIR ’01, pages 412–413, New

York, NY, USA. ACM.

Xue, X.-B., Zhou, Z.-H., and Zhang, Z. M. (2008). Improv-

ing web search using image snippets. ACM Transac-

tions on Internet Technology, 8:21:1–21:28.

KDIR2012-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval

186