INTEGRATED SYNTACTIC AND SEMANTIC DATA STRUCTURING

An Abstraction of Intelligent Man-machine Communication

∗

Wladyslaw Homenda

Faculty of Mathematics and Information Science Warsaw University of Technology, Warsaw, Poland

Keywords:

Knowledge representation, Syntax, Semantics, Man-machine communication, Intelligent interfaces.

Abstract:

The paper discusses an approach to intelligent man-machine communication which is a fundamental topic of

intelligent interface of any software. The approach presented in the paper is based on integration of syntactic

and semantic approximations of information structure. In most cases of communication automatic revealing

of the structure of information subjected to communication is not possible due to its complexity. Proposed so-

lution of this problem is based on raw, approximated descriptions of information entities and relations between

them. This approach reveals parallel syntactic and semantic attempts to so called languages of natural com-

munication. Duality of both attempts automatically exposes structure of information and allows a machine for

information maintenance and processing in human-like way. The attempt is reﬂected in the domain of music

information taking music notation as the language of natural communication.

1 INTRODUCTION

For all the time information exchange between human

beings has been done in languages of natural commu-

nication. Natural languages, music notation, gesture

language, etc. are examples of languages of natural

communication. Languages of natural communica-

tion have been created, developed and used prior to

their formal codiﬁcation and - up to now - they have

no formal full deﬁnition.

Since the beginning of the computing era people

have been thinking about computers as their partners

or - less radically - intelligent tools that can act in

a manner similar to a man’s reaction. Computers as

partners of a man require exchanging information and

understanding communication. Up-to-now a form of

man-machine information exchange has been domi-

nated by machines requirements. Due to poor abilities

of machines human beings have had to create commu-

nication tools which could be recognized by comput-

ers. Indeed, languages of formal communication have

been using almost exclusively in man-machine com-

munication. Not only Pascal, C++, Java, but also all

kinds of menus and dialog boxes with all their options

and features often hardly guessed and always easily

forgettable. This kind of tools have been widely ap-

∗

Part of this paper will be included into a forthcoming

chapter of a Springer edited volume.

plied and used despite their inadequacy and uncom-

fortableness. Amazingly, a man’s product has domi-

nated him as nothing before - at least in the aspect of

communication.

Humans have been always thinking about raising

human-like relations with machines, i.e. on his own,

human’s, conditions. Since communication and mu-

tual understanding are the most important features

of such relations, people have been considering ma-

chines as human like behaving artefacts with human’s

features and skills. Such a thinking was a science ﬁc-

tion deliberation rather then nearest future ability. But

now it slowly comes to reality. There are two rea-

sons for making the computing equipment more and

more user-friendly and acting in a human-like man-

ner. The giant increase of the computing equipment’s

power/output allows, on the one hand, for the emula-

tion by brute force of a man’s simple behavior. On the

other hand, advances in research in the ﬁelds of arti-

ﬁcial intelligence, algorithms and computability and

other areas of computer sciences, as well as various

social sciences, break barriers in making computers

more intelligent, barriers unsurmountable only with

respect to increasing power of computing hardware.

In this paper our attention is focused on the aspect

of the computing technologies development which

employs both information exchange and understand-

ing. We will discuss the problem of information ex-

324

Homenda W. (2009).

INTEGRATED SYNTACTIC AND SEMANTIC DATA STRUCTURING - An Abstraction of Intelligent Man-machine Communication.

In Proceedings of the International Conference on Agents and Artiﬁcial Intelligence, pages 324-330

DOI: 10.5220/0001512403240330

 SciTePress

change between man and the machine as well as auto-

matic understanding of information by the machine.

Importantly, communication understood as an infor-

mation exchange is being maintained in some type of

language which we wish to be more a language of nat-

ural communication rather then a language of formal

communication. This meaning of communication in-

cludes not only a simple exchange of ﬁles of informa-

tion represented in the binary form, as - for instance

- the exchange of text ﬁles without looking into their

meaning. It is also understood as a presentation of

a language text contained in the ﬁle, i.e. a language

construction or a sequence of language constructions,

with an expectation that its contents will be analyzed

and a structured space of data representing knowledge

embedded in it will be created and then interactively

used. In other words, we expect that such a kind of

communication will involve a language in which the

given language text was formulated. This language

will perhaps be a language of natural communication.

2 APPROXIMATED DATA

STRUCTURING

The notion of understanding is regarded as the main

feature of intelligent communication and an important

goal of the present paper. We would like to charac-

terize the meaning in which the word understanding

is used in the paper. Understanding is an ability to

identify objects and sets of objects deﬁned by con-

cepts expressed in a given language. The concept’s

description in a given language is what the syntax is.

A mapping which casts the concepts’ description on

the real world objects is what the semantics is. Ability

to recognize the semantics is the meaning of under-

standing. We reﬂect meanings of the above notions in

music notation seen as a language of natural commu-

nication.

2.1 Syntax

Syntactic approach is a crucial stage and a crucial

problem in the wide spectrum of tasks as, for instance,

pattern recognition, translation of programming lan-

guages, processing of natural languages, music pro-

cessing, etc. Syntactic approach is generally based on

the context-free methods which have been intensively

studied. Context-free methods have also been applied

in practice for the processing of artiﬁcial languages

as, for instance, programming languages, in technical

drawings, etc. We can even say that application in this

ﬁeld has been successful.

Unfortunately, natural communication between

people, e.g. communication in a natural language

or using music notation, is too complex to be for-

malized in a context-free way, though it is clear that

such communication is rule-governed, cf. (Bargiela

and Homenda, 2002). Even if there is a deﬁnite set

of rules deﬁning a language of natural communica-

tion, the rules are much more complicated than those

describing artiﬁcial languages of formal communica-

tion. And such rules can often be broken with lit-

tle impact on communication. Thus, a description of

such tools as a natural language or music notation

must deﬁnitely be highly ﬂexible and deeply toler-

ant to natural anarchy of its subjects. With all that

in mind, the proposed approach to describing lan-

guages of natural communication will rely on the sen-

sible application of the proposed context-free meth-

ods applied locally in the structured space of a lan-

guage of natural communication. Moreover, it is as-

sumed that the context-free methods will not be ap-

plied unfairly to generate incorrect constructions of

them. Those assumptions allow for a raw approxima-

tion of languages of natural communication as, for in-

stance, natural language or music notation, which are

far more complex than a context-free tools utilized for

such an approximation. Of course, such assumptions

are real shortcomings in accurate description of a lan-

guage of natural communication and in its process-

ing. These shortcomings must be solved by employ-

ing some other methods, perhaps not context-free.

Below, we present an approximated description

of a local area of music notation. This description is

given in the form of context free grammar. For more

details on context free descriptions of music notation

see (Homenda, 2006; Homenda, 2007).

→ <bl stave>

→ <ks stave>

→ <ts stave>

→ <measure> <barline>

→ <ks measure>

→ <ts measure>

→ <vertical event>

→ <stem>

→ < flags> <note stem>

→ <note stem>

INTEGRATED SYNTACTIC AND SEMANTIC DATA STRUCTURING - An Abstraction of Intelligent Man-machine

Communication

325

→ <flags> <rhythm group> <note stem>

→ <note stem> <rhythm group>

→ right beam <beams>

→ right beam

→ right rh gr <rhythm group>

→ 3

< flags> → flag < flags> | flag

→ note head stem

2.2 Semantics

As mentioned above, people use different tools for

communication: natural languages, programming

languages, artiﬁcial languages, language of gesture,

drawings, photographs, music notation. All those

tools could be seen as tools used for describing a mat-

ter of communication and as information carriers. We

can observe that different tools can be used for en-

coding the same communication matter description.

Immersing our deliberations into music notation we

should be aware that among different tools of natu-

ral communication, natural languages are most uni-

versal. In general, they cover most parts of informa-

tion spaces spanned by other tools. Therefore, a nat-

ural language can alternatively describe constructions

of music notation. Interpreting this observation we

can notice that, for instance, a given score can also

be described in Braille Music (Krolick, 1998), Mu-

sicXML (G. Castan and Roland, 2001) or other for-

mats or even, e.g., in the English language. Moreover,

all such descriptions carry similar information space.

Likewise, a description of a subject (a thing, a

thought, an idea, etc.) may be prepared in differ-

ent natural languages. Such descriptions approximate

the subject bringing its projection onto the language

used for description. And such a description could be

translated to other natural language without a signiﬁ-

cant lost of information. This means that the subject

being described is a meaning of a description. So, a

study on a subject described in a natural language (or

even in any language of natural communication) may

supplement the study on descriptions themselves. In

other words, syntactic analysis of language descrip-

tions may be supplemented by a semantic analysis of

description’s subject.

In this study music notation, as a language of nat-

ural communication, cast onto a space of communica-

tion subjects (i.e. onto musical scores, as texts of the

language of natural communication) is understood as

the semantic approach to music information process-

ing. Formally, assuming that L is a music notation

lexicon and M is music notation, the mapping V de-

scribes semantics of the music notation description:

V : L → M

The mapping V assigns objects of a given musi-

cal score M to items stored in the corresponding lex-

icon L. The lexicon L is a set of local portions of the

derivation tree of the score, c.f. (Homenda, 2006).

3 MAN-MACHINE

COMMUNICATION AS AN

INTELLIGENT INFORMATION

EXCHANGE

As mentioned before, communication is understood

as a presentation or an exchange of information be-

tween two (or more then two) objects of communica-

tion. Essential feature communication is understand-

ing information being exchanged. Understanding re-

quires exact description of relations between informa-

tion entities, what is done in the form of syntactic and

semantic structuring integrated in frames of informa-

tion granulation paradigm.

3.1 Syntactic Analysis - A Tool

Describing Communicated Data

Syntactic analysis is a tool used for data space struc-

turing. As discussed above, syntactic methods cannot

be used for full structuring of complex data spaces

as, for instance, for structuring music information.

Thus, it is used for approximation of data structuring.

Such an approximation is often sufﬁcient for reveal-

ing structures of data that could be extracted form the

data space and possibly subjected to further process-

ing.

Syntactic analysis is a suitable tool for acquiring

user’s choice of data. Selection tool is usually used to

deﬁne user’s choice. A selection done by user could

either be interpreted at the lowest level of data struc-

tures, or may be performed to a part of structured data

space. In Figure 1 we have two rectangle selections in

two upper parts. These selections could be interpreted

as numerical data representing raster bitmaps which

have nothing common with displayed music notation.

ICAART 2009 - International Conference on Agents and Artificial Intelligence

326

Figure 1: Examples of selections: two measures in a system, two measures in a stave, lower voice line in ﬁrst two measures,

triplets on sixteens.

INTEGRATED SYNTACTIC AND SEMANTIC DATA STRUCTURING - An Abstraction of Intelligent Man-machine

Communication

327

<ts_stave>

barline

<ts_stave>barline

<ks_stave><key_signature>

<bl_stave>beg_barline

<stave>

<ts_stave>barline

<ts_measure>

<ts_measure><vertical_event>

<ts_measure>

<vertical_event>

<stem>

<note_stem>

notehead

<note_stem>

notehead

<note_stem>

notehead

stem

<vertical_event>

<ts_measure>

...

3/4

...

...

barline

<vertical_event>

<stem>

<beams>

rightbeam

<stem>

<rhythm_group><beams> <note_stem>

rightbeam rightrhgr notehead stem <rhythm_group><beams> <note_stem>

<stem>

<beams>

rightbeam

leftbeam <rhythm_group>

rightrhgr

leftrhgr notehead stem

...

<vertical_event>

...

<time_signature

<ks_measure>

Figure 2: Derivation tree of the ﬁrst triplet marked in Figure 1.

Figure 3: Examples of transpositions: original score, automatic recognition of the original score, upper voice line moved one

octave up and lower voice line moved one octave down, third measure transposed from D to G.

ICAART 2009 - International Conference on Agents and Artificial Intelligence

328

On the other hand these selections could be under-

stood as a part of music notation, namely: two mea-

sures in a system and two measures in a stave. In

case of lower two parts selections shown as grayed

symbols of music notation cannot be interpreted as a

raw numerical data. These selections are parts of the

structured data space.

Syntactic analysis allows for immersion of user’s

selection into structured information space. Syntactic

interpretation of user’s selection of data gives the ﬁrst

signiﬁcant raise leading to full identiﬁcation of infor-

mation intended to be communicated by man. It needs

to be mentioned that, in this discussion, we drop a

category of technical details like, for instance, which

programming tools are used to point out desired ob-

jects at a computer screen and how to indicate options

of a selection.

Let us look at the selection of two measures in

the stave. It is deﬁned as a part of derivation tree in

a grammar generating the score (part of this gram-

mar is outlined in section 2.1). This selection corre-

sponds to paths from the root to two indicated ver-

texes <measure> of the part of derivation tree shown

in Figure 2. The selection of two triplets in lower

part of Figure 2 is described by the indicated ver-

tex <ts measure>, which is also taken as one vertex

path. Description of voice line selection cannot be

described as easily as other selections, it requires con-

text analysis.

3.2 Semantic Mapping as Identiﬁcation

of Communicated Data

Syntactic descriptions of information entities is a ba-

sis for identiﬁcation of relevant area of information

space. This identiﬁcation is done by casting the lexi-

con of a given score, i.e. the space syntactic granules,

onto the space of objects of sematic granules of the

score. Semantic granules are subjects of understand-

ing and of possible processing.

The meaning of paths from the root to two indi-

cated vertexes <measure> (being syntactic granules)

is deﬁned as follow. It is the crop of all subtrees

of derivation tree, which include both paths together

with subtrees rooted in vertexes ending both paths.

The structure of symbols of music notation that are

included into selected two measures corresponds to

this meaning.

On the other hand, crops of all subtrees of the

derivation tree in Figure 2, which are equal to the sub-

tree deﬁned by the indicated vertex <ts measure>, is

the meaning of syntactic granules (in this case, the

subtree has excluded its part denoted by indicated

multidots vertex).

It is worth to notice that the description of the ﬁrst

semantic granule is a special case of the the descrip-

tion of the second semantic granule. Having a path,

which begins in the root of derivation tree, we can ﬁnd

only one subtree equal this path with subtree rooted in

its ending vertex.

Semantic granules deﬁne meaning of informa-

tion being exchanged and allow for responding to re-

quests. Such responses are outlined in Figure 3. Its

upper part shows original score. two other parts illus-

trate transposition performed on recognized notation.

The middle parts shows three voice lines. The upper

voice line was subjected to transposition by one oc-

tave up. The lower voice line was subjected to trans-

position by one octave up. The third part of shows

transposition of the third measure from D to G.

3.3 Granulation as a Form of

Understanding

Information exchanged in communication is material-

ized in the form of texts of a language of natural com-

munication. Thus, the term text spans not only over

texts of natural languages, but also over constructions

like, for instance, musical scores, medical images, etc.

(we can also apply this term to constructions of lan-

guages of formal communication, e.g. to computer

programs). Revealing recent sections let us say that a

study on how texts are constructed is what we mean

as syntax. A matter described by such a text is what

is understood as semantics. Integrating syntax and se-

mantics leads to information granulation and identiﬁ-

cation of relations between granules of information,

c.f. (Homenda, 2007; Pedrycz and Bargiela, 2005).

Discovering relations between both aspects is seen as

understanding.

The description of music notation as well as

music notation itself could be innately subjected

to the paradigm of granular computing elucidation.

As stated in (Pedrycz, 2001), granular computing

as opposed to numeric computing is knowledge-

oriented. Information granules exhibit different levels

of knowledge abstraction, what strictly corresponds

to different levels of granularity. Depending upon the

problem at hand, we usually group granules of simi-

lar size (i.e. similar granularity) together into a sin-

gle layer. If more detailed (and computationally in-

tensive) processing is required, smaller information

granules are sought. Then, those granules are ar-

ranged in another layer. In the granular processing

we encounter a number of conceptual and algorithmic

layers indexed by the size of information granules.

Information granularity implies the usage of various

techniques that are relevant for the speciﬁc level of

INTEGRATED SYNTACTIC AND SEMANTIC DATA STRUCTURING - An Abstraction of Intelligent Man-machine

Communication

329

granularity.

The meaning of granule size is deﬁned accord-

ingly to real application and should be consistent

with common sense and with the knowledge base

of the application. Roughly speaking, size of syn-

tactic granules is a function of depth of the syntac-

tic structure. Size of the syntactic granule <score>

<score part><page><system> is smaller then size of

<score><score part><page><system><stave> which,

in turn, is smaller then size of <score><score part>

<page><system><measure>.

On the other hand, we can deﬁne size of

semantic granule. It is deﬁned as a quan-

tity of real world objects or a length of con-

tinue concept. Size of the semantic granule

V(<score><score part><page><system>) is greater

than size of V(<score><score part><page><system>

<stave>), which, in turn, is greater then V(<score>

<score part><page><system><stave><measure>).

The relevance between syntactic and semantic gran-

ules has been discussed in (Homenda, 2006; Home-

nda, 2007).

4 CONCLUSIONS

The new framework on man-machine intelligent com-

munication is presented in the paper. The term intelli-

gent communication is understood as information ex-

change with identiﬁed structure of information, which

is presented by a side of communication to his/its

partner(s) or is exchangedbetween sides of communi-

cation. Of course, identiﬁcation of information struc-

ture is a natural feature of human’s side of such com-

munication. An effort is focused on automatic identi-

ﬁcation of information structure based on syntax and

semantics of information description. Syntactic and

semantic descriptions have dual structure revealing

granular character of represented information. Com-

plementary character of both attempts allows for au-

tomation of information structuring and - in conse-

quence - intelligent information maintenance and pro-

cessing, what is the basis of intelligent communica-

tion in man-machine communication process.

In this paper the problem of man-machine intelli-

gent communication is reﬂected in the area of music

notation treated as a language of natural communi-

cation. However, reﬂection of this problem in nat-

ural language as a language of natural communica-

tion give similar conclusions, c.f. (Homenda, 2002).

Thus, we can expect that integrated syntactic and se-

mantic data structuring guides to rational interpreta-

tion of man-machine communication in many areas

of human activity. This framework permits for better

understanding of communication process as well as

leads to practical solutions.

It is worth to notice that man-machine communi-

cation is a basis of intelligent interface of any soft-

ware. An intelligent interface of a computer program

in terms of its way of communication method (graph-

ical, sound, etc.) design is cast on data structures

processed by the program or exchanged between man

and machine. An integration of both elements: man-

machine communication and interface design is an in-

terdisciplinary subject of studies.

ACKNOWLEDGEMENTS

The paper is supported by the University Research

Program Hierarchical methods of information acqui-

sition. Automatic recognition of printed text as a

background element of documents and the Faculty Re-

search Program Intelligent Computing Technologies

REFERENCES

Bargiela, A. and Homenda, W. (2002). Information struc-

turing in natural language communication: syntactical

approach. In Proc. of the International Conference on

Fuzzy Systems and Knowledge Discovery. Singapore.

G. Castan, M. G. and Roland, P. (2001). Extensible markup

language (xml) for music applications: An introduc-

tion. In Walter B. Hewlett and Eleanor Selfridge-Field

(Eds.) The Virtual Score: Representation, Retrieval,

Restoration. The MIT Press.

Homenda, W. (2002). Information structuring in natural

language communication: Syntax versus semantic. In

Rutkowski L. et al (Eds.), Artiﬁcial Intelligence and

Soft Computing - ICAISS, Lecture Notes in Artiﬁcial

Intelligence. Springer.

Homenda, W. (2006). Automatic understanding of images:

integrated syntax and sematic analysis of music no-

tation. In IEEE World Congress on Computational

Intelligence. Vancouver, BC, Canada.

Homenda, W. (2007). Automatic understanding of images:

integrated syntax and sematic analysis of music nota-

tion. In Fourth International Conference on Modeling

Decisions for Artiﬁcial Intelligence MDAI 2007. Ki-

takyushu, Fukuoka, Japan.

Krolick, B. (1998). How to Read Braille Music. Opus Tech-

nologies, London, 2nd edition.

Pedrycz, W. (2001). Granular computing: An introduc-

tion. In Proc. of the Joint 9th IFSA World Congress

and 20th NAFIPS International Conference. Vancou-

ver, BC, Canada.

Pedrycz, W. and Bargiela, A. (2005). Granular map-

pings. In IEEE Transactions on Systems, Man, and

Cybernetics-part A, volume 35, 2.

ICAART 2009 - International Conference on Agents and Artificial Intelligence

330