DETECTION OF INCOHERENCES IN A TECHNICAL AND

NORMATIVE DOCUMENT CORPUS

Susana Martin-Toral

, Gregorio I. Sainz-Palmero

1,2

Computer and Information Technologies Division. Fundaci´on CARTIF

Parque Tecnol´ogico de Boecillo, 47151 Valladolid, Spain

Department of Systems Engineering and Control, School of Industrial Engineering

University of Valladolid, 47011 Valladolid, Spain

Yannis Dimitriadis

GSIC - Group of Intelligent and Cooperative Systems, School of Telecommunications Engineering

University of Valladolid, 47011 Valladolid, Spain

Keywords:

Document corpus, content incoherence, natural language processing, text mining, artiﬁcial intelligence, docu-

ment engineering.

Abstract:

This paper is focused on the problems and effects generated by the use of a document corpus with mistakes,

content incoherences amongst its connected documents and other errors. The problem introduced in this paper

is very relevant in any area of human activity when this corpus is used as base element in the relationships

between company partners, legal support, etc., and the way in which these incoherences can be detected. These

problems can appear in several ways, and the produced effects are different, but a common situation exists in

those areas of activity where many linked documents must be generated, managed and updated by different

authors. This paper describes some examples of this problem in the case of a technical document corpus used

amongst partners, and the solution framework developed for this case. Several types of incoherence have

been detected and formulated, connected with problems described in other research areas such as information

extraction and retrieval, text mining, document interpretation and others, but all of them have been bounded

and introduced from the point of view of document incoherences and their effects, specially in a company

context. Finally the computational architecture and methodology uses are described and some initial results

of incoherence detection are discussed.

1 INTRODUCTION

Documentation, on paper or in electronic format, is a

base element for the information society. It is the most

usual way to store, save and exchange information in

a wide range of human activity contexts, so the infor-

mation and knowledge contained in it has to be right

and clear with no possibility of confusion or contra-

diction. But this goal is not trivial due to several facts.

Some public and private sectors handle documenta-

tion that is not-methodologically generated, suffers

changes and grows in volume and versions.

It is difﬁcult to ﬁnd organizations working with

heterogeneous sets of connected documents that man-

age this movement in a suitable and formal way, with

a unique formulation in their generation, management

and control (see Figure 1), so the problem of incoher-

ences in related documentation appears: mistakes in

the cross references, redundant, contradictory, miss-

ing or wrong information.

The case involvedin this paper comes from the use

in technical documentation by a company of the elec-

trical sector. In this context, an “incoherence” repre-

sents any lack of consistency amongst related docu-

ments or even within the same document.

For example, some types of incoherences appear

when a document does not match the correct struc-

ture, according to the known rules for its genera-

tion. In this situation a structural incoherenceappears.

Other types of incoherence could be numerical. This

concerns the numerical values contained in a docu-

ment that must agree with the values indicated in the

norm, standard or reference document. A contradic-

tion in numerical values between documents with the

282

Martin-Toral S., I. Sainz-Palmero G. and Dimitriadis Y. (2008).

DETECTION OF INCOHERENCES IN A TECHNICAL AND NORMATIVE DOCUMENT CORPUS.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - AIDSS, pages 282-287

DOI: 10.5220/0001699102820287

 SciTePress

Figure 1: Documentation generation scheme in a technical

business.

same concept is not possible. These two examples,

and other types of incoherences, makes up an initial

deﬁnition of an incoherence taxonomy.

The impact of all these problems for an organi-

zation, both in its internal and external relationships,

could cause economic, legal, technical, even serious

social consequences, so when this happens there is a

great interest in eliminating them. Thus, some sec-

tors with economic activity have been seen to show

a growing interest in solving this kind of problem,

though of course, such interest is not exclusive to such

sectors.

- Civil Engineering & Electrical sector, in general

any industrial environment in which collaborative

partners work with related technical documents,

normatives and standards (CARTIF, 2006).

- Software industry, in which the documentation

generated from a uniﬁed software process is

affected by the problems described previously

(Arango, 2003).

- Healthcare sector, the volume of documentation

managed is enormous and very critical due to the

effects of mistakes in such documentation (Ming-

shan and Ching-to, 2002).

- Legal and Law sector, when laws or legal norms

are contradictory, ”legal antinomia” (Ruiz, 2002).

Otherwise, three different parts could be affected

by these problems:

- Owner, economic and legal aspects: incoherent

documentation does not match the application

norms and policies to the sector, which could re-

sult in third party damages.

- User, by employing a contradictory, ambiguous or

even wrong documentation with similar effects to

those commented previously.

- Consumer, using products and services created

by weak and incoherent documentation that can

cause bad quality or dangerous products or ser-

vices.

At this point, the interest concerning the detection

and elimination of these incoherences appears. Doc-

umentation free of incoherences could solve a rele-

vant problem, or at least that focusing on wrong or

confused information, facilitating a coherent manage-

ment of that documentation and obtaining a better

quality of products and services. This is the main mo-

tivation for dealing with the problem.

The organization of the rest of the paper is as

follows: ﬁrst of all a deﬁnition and classiﬁcation of

the detected incoherences in the case involved, and

techniques that could be applied for its detection, are

presented. In the technological framework section

the computational and conceptual architectures devel-

oped in this work are described. Finally, the most in-

teresting results obtained are discussed and the main

conclusions of this work are put forward.

2 DOCUMENT INCOHERENCES.

AN APPROACH

Once the problem and the motivations have been pre-

sented, the next step is to formally deﬁne what is

considered an incoherence in this work, taking into

account the documentation involved (see (CARTIF,

2006)): content incoherence is seen as the weak-

ness of consistency amongst related documents, or

amongst different pieces of the same document, or the

lack or excess of information in a document.

This deﬁnition introduces subjectivity in deciding

what can be considered as an incoherence and its ef-

fects, thus its importance. To assess this issue, an ini-

tial incoherence taxonomy has been deﬁned:

- Structural, concerning the logical rules for doc-

ument generation: differences in style or format

employed when the document was generated or

updated. This aspect is connected with research

into document analysis and the logical and lay-

out document structure (O’Gorman and Kasturi,

1995).

- References, documents use references to other

documents, norms or standards in order to sup-

port the document content or to avoid describing

any aspect explained in the references. The inco-

herence could happen when the reference is not

adequate, or does not exist, or is not referenced.

- Numerical, this concerns the numerical values

contained in a document that must agree with the

values indicated in the norm, standard or docu-

ment of reference. A contradiction between doc-

uments for the same concept is not possible.

DETECTION OF INCOHERENCES IN A TECHNICAL AND NORMATIVE DOCUMENT CORPUS

283

- Measure Units, the units for measuring used in a

document have to work correctly in accordance

with the standard or the International System of

Units.

- Attribute, which is similar to the numerical one,

but applied to attributes such as colors (green,

black, red), shapes (square, triangular), states (liq-

uid, solid, gaseous), etc.

- Denomination or conceptual, it is very important

to use the concepts in a suitable way for the con-

text involved. A conceptual incoherence could

happen if an important concept is denominated of

different ways in the same document or even in

different documents.

- Update, the new version could contain more or

less informationor contradictoryinformation with

regard to the previous version.

- Titles or subtitles of a document do not match the

contents of their sections.

- Dirty words, use of badly written words or words

that do not exist.

In the technical context involved, each of these in-

coherences has a different relevance and effect, which

is usually deﬁned by the domain expert. Generally,

depending on the type of incoherence to be detected

automatically, it will be necessary to apply different

techniques for information processing.

3 TECHNOLOGICAL

FRAMEWORK PROPOSAL

After the theoretical introduction to the problem, this

section deals with an approach to the technological

framework for incoherence detection in documents.

For this aim, a computational architecture has been

proposed (see Figure 2) introducing different levels

of information processing.

- Preprocessing converts the original documenta-

tion to an open format that facilitates its process-

ing. For this task, the standard OASIS Open Doc-

ument Format (ODF) has been selected (OASIS,

2007).

- In extraction techniques module, text and docu-

ment mining techniques are mainly applied to ex-

tract the adequate information from each docu-

ment in order to detect the several types of in-

coherence. Different representations of the same

document must be used to detect the types of in-

coherence considered.

- Representation of atomic information keeps the

suitable representations of the documents. Sev-

eral types of document representation are needed

according to the relevant type of contents or con-

cepts to be checked.

- Document representations by incoherence. Here

the most adequate representations of a document

are taken from the previous module, according to

the incoherence to be detected. Several represen-

tations could be required for an incoherence.

- Comparison techniques module uses text docu-

ment mining techniques for matching different

document representations to detect similarities

and differences between them. This information

will be used as a source of incoherence problems.

Figure 2: Computational architecture.

Once any potential incoherence has been detected

and classiﬁed, it is the domain expert who must take

the ﬁnal decision about the relevance of the detected

problem. Accordingly, the system could be improved

by an adequate feedback.

In this work different techniques have been used

to cover the functionalities of extraction and com-

parison modules. Most of them, mainly for extrac-

tion techniques, are based on the use of heuristic

solutions, with similar criteria as to (Krulwich and

Burkey, 1997).

3.1 Extraction Techniques

The adequate document representations have been

obtained by different techniques of information ex-

traction (Nahm, 2004), information retrieval (Berry,

2004), and document description (Jain et al., 2000).

The proposal, from this point of view, could be seen

as a pattern recognition problem, where every docu-

ment is a pattern of the problem space and each docu-

ment representation is the result of a feature selection

ICEIS 2008 - International Conference on Enterprise Information Systems

284

process, according to the knowledge of the document

corpus. In this way, three main document representa-

tions (conceptual models) have been used:

1. Key term representation, using the Vector Space

Model (VSM) (Salton et al., 1975). This type

of representation facilitates the detection of ref-

erence, conceptual or title incoherences.

2. Reference representation. Every document is de-

scribed according to the references to other doc-

uments. This representation mainly permits the

detection of reference incoherences.

3. Representation using relevant information pat-

terns. Here the information extraction is based on

heuristics, according to information patterns de-

tected inside the document corpora that are rel-

evant in the domain. An example of this is the

representation of a document by its technical data

terms. Each one is represented by an ”N-tuple”,

here N = 4:

< Term ; Operator ; Value ; Units >

Where Term is the word, or set of words, repre-

senting a relevant concept, Operator indicates that

a term is bigger (>), smaller (<), than or equal

to (= ) a speciﬁc value, Value represents the nu-

merical value, or enumerated data (colour, state,

shape) of the term, and ﬁnally, Units is only used

when the value is numerical and with units. Then

the document is summarized by a set of this type

of N-tuple. These N-tuples have been generated

by Episode Rule Mining techniques (ERM) (Man-

nila et al., 1997). An example of a real 4-tuple is:

wire CCX-56-D section ; = ; 54,6 ;

This representation facilitates the detection of nu-

merical, measure and attribute incoherences, ap-

plying suitable matching techniques.

3.2 Comparison Techniques

Matching techniques must be applied to obtain sim-

ilarities, differences, deviations, and trends amongst

document representations. Here, different text min-

ing techniques have been used:

- Classiﬁcation (Berry, 2004), based on VSM rep-

resentation and Naive Bayes, KNN (K Nearest

Neighbour) or TFIDF (Term Frequency - Inverse

Document Frequency) classiﬁers, and using the

libbow library (McCallum, 1996).

- Clustering, using VSM and a Hierarchical

Expectation-Maximization (HEM) algorithm

(Jain et al., 2000).

- Summarization using MEAD, a public domain

portable multi-document summarization system

(Otterbacher et al., 2002).

- Trend detection (Berry, 2004), using the edit dis-

tance, cosine similarity, and summarization tech-

niques (Mani and Bloedorn, 1999).

Figure 3: Conceptual architecture.

In the bibliography,some of these mentioned tech-

niques are used to deal with problems not related to

content incoherence detection. For this reason, the

potential techniques discovered in the bibliography

have been adapted to solve the incoherence problem,

as in Figure 3.

4 EXPERIMENTS AND RESULTS

The experimental work has used the more suitable

document representations and comparison techniques

to detect potential incoherences. The effects and rel-

evance of each one has been decided by the domain

expert.

The document collection used in the experimen-

tal part consists of 873 documents corresponding to

technical manuals (469 TM) and norms (404 N) of

an electrical company. The documentation contains

semistructured and connected documents, with many

numerical Tables and Figures. It has been generated

by multiple authors and branches, so the document

corpus presents incoherence problems.

The experiments developed have covered two

types of analysis, according to the principles of docu-

ment engineering: structural and content analysis.

Structural analysis has been used to detect struc-

tural incoherences in the documentation: lack of

mandatory sections, badly numbered chapters, etc.

The knowledge of the rules about standard generation

of documents permits us to process the document cor-

pus. The results of this experiment have discovered

that almost 30% of the norms present some type of

structural incoherence, so they do not comply with the

deﬁned rules. For example, all the norms must have

a section indicating what other norms are referred to

DETECTION OF INCOHERENCES IN A TECHNICAL AND NORMATIVE DOCUMENT CORPUS

285

in the rest of the document. In this study, a 15.84% of

the norms do not match with the rule because of the

lack of some referred norm (see Table 1).

Table 1: Partial results for strutural incoherences.

Mandatory sections Coh.norms Incoh.norms

Introduction 134 (33.17%) 270 (66.83%)

Referred norms 340 (84.16%) 64 (15.84%)

Application ﬁeld 365 (90.35%) 39 (9.65%)

In content analysis, the main objective is to detect

possible incoherences by processing the documenta-

tion contents, but not its structure. For that end, doc-

ument representations shown in section 3.1, and com-

parison techniques described in section 3.2 have been

used.

The use of the Vector Space Model allows us

to discover relevant terms that should not present

conceptual incoherences. Using this representation,

terms written in different languages can be detected,

and they are also considered as incoherences by the

domain expert.

With classiﬁcation experiments, the trend of doc-

ument clustering can be studied to detect strange be-

haviours. In the results, we can see how some doc-

uments tend to be classiﬁed in a non adequate class

or category. The best results, see Table 2, have been

obtained with the TFIDF classiﬁer.

Table 2: TFIDF classiﬁcation results for norms.

Trial Ok classiﬁed/total % Accuracy

0 92/121 76.03

1 99/121 81.82

2 94/121 77.69

3 95/121 78.51

4 101/121 83.47

5 94/121 77.69

6 90/121 74.38

7 95/121 78.51

8 91/121 75.21

9 89/121 73.55

Average 77.69 stderr 0.94

From the total available norms, a 70% has been

selected to train the classiﬁer, and the rest (121 docu-

ments) to make the test. The average accuracy for 10

trials is 77.69%.

Both document classiﬁcation and clustering, are

developed according to relevant terms of each docu-

ment, so incoherence traces can be detected by ana-

lyzing the content of these bad classiﬁed documents,

in comparison with the rest of the members of the cat-

egory. Similar situations appear when a document is

clustered in an unsuitable cluster. These situations are

potential sources of incoherences. All the norms are

coded according to the material family they represent,

so this codiﬁcation can be used as a natural initial

classiﬁcation. When a norm is classiﬁed or clustered

in a group not belonging to its material family, ac-

cording to the codiﬁcation, this is because its content

is more similar to the other category, so it presents

similarities to an other material family. The analy-

sis of this strange behaviour can report traces of inco-

herence problems. For example, the strange situation

appears when the norm N 72.30.03, that belongs to

the transformer family (code 72), is classiﬁed into the

wire family (code 56) because of its content (in the

electrical sector).

Representing the documentation using the refer-

ence model facilitates the detection of reference inco-

herences. The representation is in the following form:

TM1 N1 N3

TM2 N1 N4 N5 N9

TM3 N2 N5

....

TMN N2

where TM are technical manuals and N are norms.

With this information, unsuitable use of documents

can be detected, and also incoherences related to the

use of non-existent and wrong coded norms. It has

been detected that a 5% of the referenced norms

present a wrong codiﬁcation, and more than a 7%

are non-existent, maybe because they are deprecated

and have been eliminated from the ﬁnal version of the

document corpus.

Finally, the representation using 4-tuples allows

the detection of numerical, measures and attribute in-

coherences. The experimental part has been applied

to obtain all 4-tuples from the documentation, ap-

plying ERM techniques to extract this information

pattern. Matching the representations between two

related documents, or matching the N-tuples of the

same documents, facilitates the detection of this kind

of incoherences. They seem to be the more numer-

ous incoherences, so there is an increasing interest in

their detection. Thus, at the moment, a detailed study

of this type of representation, and all the related tech-

niques, is being considered.

5 CONCLUSIONS

This paper introduces the problem of document inco-

herences from the point of view of the organizations,

companies or environments using document corpus

with mistakes, wrong documents and confused con-

tents, and the effects of this inadequate documenta-

tion: economic, legal, technical and social.

ICEIS 2008 - International Conference on Enterprise Information Systems

286

Taking into account the technical and normative

documentation involved in this work, an attempt to

deﬁne and classify incoherence has been introduced,

which could be used in most technical contexts. From

that, each incoherence has been connected with sev-

eral research areas (information extraction and re-

trieval, document analysis, etc.) in order to ﬁnd

the best way to detect the incoherence by informa-

tion processing. The results obtained by experiments

have allowed us discover several categories of inco-

herences, even some unknown to the domain experts.

The study of new domain and sector documen-

tation could expand and improve the proposed inco-

herence classiﬁcation, but not all incoherences have

the same relevance or the same importance for the af-

fected sectors. The experimentation in this work has

tried to apply the more suitable techniques to detect

those with the most relevant impact in the affected ar-

eas.

To achieve all these objectives, different docu-

ment representations and comparison techniques have

been applied. In this aspect, a new relevant informa-

tion pattern, repeated in technical documentation, has

been used (N-tuples), allowing the detection of one

of the most important and negative incoherence types

found in technical domains: numerical incoherences.

The interpretation and evaluation of the results

have been developed in both unsupervised and super-

vised ways, in this latter case, with the help of the

domain expert. From this evaluation, different levels

of incoherence have been detected:

- Some experimentalresults haveshownstrange be-

haviours, and therefore the presence of potential

incoherences. A deeper study could be needed to

detect speciﬁc incoherences. These results are ob-

tained by classiﬁcation or clustering methods.

- Other results have directly shown potential cases

of incoherences, and the help of the domain expert

is only needed to ensure that the problem exists.

This is the case of incoherences of wrongly coded

and non-existent norms, structural, or content in-

coherences using VSM, and numerical measures

and attribute incoherences using 4-tuples.

Due to the existence of incoherence and its neg-

ative effects, for both to organization and citizens,

future work could deal with the deﬁnition of a new

methodology for the generation of new documenta-

tion free of incoherences, to avoid the initial seed of

the problem.

ACKNOWLEDGEMENTS

This work has been supported in part by the Spanish

Industry, Tourism, and Commerce Ministry through

the project FIT-350100-2006-272.

REFERENCES

Arango, F. (2003). Gestion de inconsistencias en la evolu-

cion e interoperacion de los esquemas conceptuales

OO, en el marco formal de OASIS. PhD thesis.

Berry, M. W. (2004). Survey of Text Mining : Clustering,

Classiﬁcation, and Retrieval. Springer.

CARTIF, F. (2006). Gestor documental de normativa

(DOCNOR). PROFIT project (PROgrama de Fo-

mento de la Investigacin Tecnologica). Project refer-

ence: FIT-350100-2006-272.

Jain, A. K., Duin, R. P. W., and Mao, J. (2000). Statistical

pattern recognition: A review. IEEE Trans. on Pattern

Analysis and Machine Intelligence, 22(1):4–37.

Krulwich, B. and Burkey, C. (1997). The infoﬁnder agent:

Learning user interests through heuristic phrase ex-

traction. IEEE Expert: Intelligent Systems and Their

Applications, 12(5):22–27.

Mani, I. and Bloedorn, E. (1999). Summarizing similarities

and differences among related documents. Informa-

tion Retrieval, 1(1-2):35–67.

Mannila, H., Toivonen, H., and Verkamo, A. I. (1997). Dis-

covery of frequent episodes in event sequences. Data

Min. Knowl. Discov., 1(3):259–289.

McCallum, A. K. (1996). Bow: A toolkit for statistical lan-

guage modeling, text retrieval, classiﬁcation and clus-

tering. http://www.cs.cmu.edu/ mccallum/bow.

Mingshan, L. and Ching-to, A. M. (2002). Consistency in

performance evaluation reports and medical records.

The Journal of Mental Health Policy and Economics,

5(4):191–192.

Nahm, U. Y. (2004). Text mining with information extrac-

tion. PhD thesis. Supervisor-Raymond J. Mooney.

OASIS (2007). OASIS. Organization for the Advance-

ment of Structured Information Standards. URL:

http://www.oasis-open.org. Last visit: February 2007.

O’Gorman, L. and Kasturi, R. (1995). Document Image

Analysis. IEEE Computer Society, Los Alamitos, Cal-

ifornia, USA.

Otterbacher, J., Radev, D., and Luo, A. (2002). Revisions

that improve cohesion in multi-document summaries:

A preliminary study. In Proc. of the Workshop on Au-

tomatic Summarization (including DUC 2002), pages

27–36. Association for Computational Linguistics.

Ruiz, M. (2002). Sistemas jurdicos y conﬂictos normativos.

Dykinson, Universidad Carlos III de Madrid, Instituto

de Derechos Humanos Bartolom de las Casas.

Salton, G., Wong, A., and Yang, C. S. (1975). A vector

space model for automatic indexing. Communication

of the ACM, 18(11):613–620.

DETECTION OF INCOHERENCES IN A TECHNICAL AND NORMATIVE DOCUMENT CORPUS

287