Fuzzy Least Squares and Fuzzy Orthogonal Least Squares Linear

Regressions

Julien Rosset

and Laurent Donz

ASAM Group, University of Fribourg, Boulevard de P

erolles 90, 1700 Fribourg, Switzerland

Keywords:

Fuzzy Least Squares Linear Regression, Fuzzy Orthogonal Least Squares Linear Regression, Fuzzy Orthogo-

Nality, Fuzzy Logic, Fuzzy Methods in Data Analysis.

Abstract:

We examine the well known fuzzy least squares linear regression method. We discuss the constrained and

unconstrained solutions. Based on the concept of fuzzy orthogonality, we propose the fuzzy orthogonal least

squares method to solve fuzzy linear regression problems. We show that, in case of (fuzzy) orthogonal re-

gressors, an important property of the least squares method remains valid. We obtain the same estimates of

the parameters of the model if we regress on all regressors, or on each regressor considered separately. An

empirical application illustrates our methods.

1 INTRODUCTION

Fuzzy regression is no longer a new topic in fuzzy

analysis. Indeed, for decades, all kinds of propo-

sitions have been made to perform fuzzy regression

analyses. Focus has been put particularly on solving

least squares problems or least absolute deviations.

We intend to review some results and complete them

with the so-called orthogonal least squares method.

We especially investigate the fuzzy least squares li-

near regression and the orthogonal fuzzy least squares

linear regression. We provide essentially some strate-

gies to deal with fuzzy data in a regression context.

We proposed constrained and unconstrained solutions

and discussed them.

In some situations, the analyst could proﬁt from

the orthogonality property of the independent vari-

ables. Indeed, by using a proper deﬁnition of fuzzy

orthogonal variables, we show an important feature

of the fuzzy orthogonal least squares method to solve

a linear regression problem. As in the classical case,

with crisp data, in a situation of orthogonal regressors,

the estimates of the model parameters are the same if

the estimation is done with all regressors of the model,

or if we regress the dependent variable on each re-

gressor alone. We verify empirically that, with our

fuzzy orthogonal least squares regression, this prop-

erty holds in a fuzzy context.

https://orcid.org/0000-0002-7883-1512

https://orcid.org/0000-0003-3522-4672

The main contribution of this paper is the proposi-

tion of a fuzzy orthogonal linear least squares regres-

sion method preserving the crisp orthogonal linear

least squares property in a fully fuzzy environment.

A nice feature of the method is to respect the fuzzy

arithmetic. Our work is organized as follow. We be-

gin our study with a small literature review in section

2. Notation is ﬁxed in the next section, 3. Section 4 is

devoted to the problem of the fuzzy least squares re-

gression, proposing two methods to deal with crisp or

fuzzy input and fuzzy output. A discussion of the con-

strained and unconstrained solutions of the two meth-

ods is given in section 5. After brieﬂy deﬁning fuzzy

orthogonality, we present the method of the fuzzy or-

thogonal least squares dealing with crisp and fuzzy

input and fuzzy output in section 6. Section 7 gives us

the possibility to illustrate our methods by empirical

applications. Finally, section 8 concludes our study.

2 LITERATURE REVIEW

In 1982, Tanaka introduced a possibilistic approach

to fuzzy regression analysis. The method consists in

using possibilistic restrictions to minimize the fuzzi-

ness of the model’s fuzzy parameters. In this work,

the quadratic programming approach, which allows

both the minimisation of the estimated deviations of

the central tendency and the minimisation of the es-

timated deviations in the spreads of fuzzy observa-

Rosset, J. and Donzé, L.

Fuzzy Least Squares and Fuzzy Orthogonal Least Squares Linear Regressions.

DOI: 10.5220/0012182700003595

In Proceedings of the 15th International Joint Conference on Computational Intelligence (IJCCI 2023), pages 359-368

ISBN: 978-989-758-674-3; ISSN: 2184-3236

359

tions, will be of concern. (Tanaka and Lee, 1997)

studied the fuzzy linear regression model by means of

quadratic programming to minimise the distances be-

tween the estimated output centres and the observed

outputs while minimising the spreads of the estimated

outputs. (Tanaka and Lee, 1998) proposed an inter-

val regression analysis based on a quadratic program-

ming approach to deal with the problem of fuzzy co-

efﬁcients becoming crisp when using linear program-

ming in possibilistic regression analysis. (Lee and

Tanaka, 1998) proposed a fuzzy regression analysis

based on a quadratic programming approach to again

integrate the central tendency of least squares and the

possibilistic properties of fuzzy regression. (Lee and

Tanaka, 1999) also explored a fuzzy linear regression

model with non-symmetric fuzzy coefﬁcients using

quadratic programming and created a lower and up-

per approximation model. (Donoso et al., 2006) pro-

posed two new fuzzy regression models, the quadratic

possibilistic model and the quadratic non-possibilistic

model, which do not focus on the minimisation of the

uncertainty of the estimated results but on the min-

imisation of the quadratic deviations between the ob-

servations and estimated outputs. However, (Donoso

et al., 2006)’s regression models were only dealing

with fuzzy regressors.

To palliate this, (D’Urso and Massari, 2013) pro-

posed a general linear regression model for studying

the dependence of fuzzy response variable, on a set of

crisp or fuzzy explanatory variables. They also sug-

gested a robust fuzzy regression method, based on the

Least Median Squares estimation approach in an at-

tempt of neutralising the effects of crisp and fuzzy

outliers. In this direction, (Kashani et al., 2021) pro-

posed a penalized estimation method to estimate the

coefﬁcients of a linear regression model with a fuzzy

response variable and crisp explanatory variables. (Li

et al., 2023) constructed a fuzzy multiple linear least

squares regression model based on two distance mea-

sures between LR-type fuzzy numbers. (Stanojevi

and Stanojevi, 2022) described a fuzzy quadratic least

squares regression for a fuzzy response variable and a

single crisp explanatory variable which gives regres-

sion coefﬁcients with positive spreads.

Fuzzy inner product spaces and fuzzy orthog-

onality have been discussed by (Ithoh, 2017) and

(Mostoﬁan et al., 2017). They proposed new deﬁni-

tions of a fuzzy inner product space. They also de-

ﬁned a suitable notion of fuzzy orthogonality in the

fuzzy world and investigated some properties. (Gior-

dani and Kiers, 2004) proposed two extensions of the

classical principal component analysis dealing with

fuzzy symmetric numbers. However, The lack of a

properly deﬁned fuzzy orthogonality made the de-

rived results losing in signiﬁcance. (Yabuuch and

Watada, 2017) performed a principal component anal-

ysis over crisp data belonging to fuzzy groups. In

their work (Yabuuch and Watada, 2017) introduced

new deﬁnitions of expectation, variance and covari-

ance to work with the concept of fuzzy groups. How-

ever, their results are limited to crisp input data.

We wanted to investigate more thoroughly the

fuzzy linear least square regression methods involving

both fuzzy response and explanatory variables since

these have not received much attention except from

(D’Urso and Massari, 2013). However, due to the

complexity of their recursive approach, constructing a

fuzzy orthogonal linear least squares method inspired

from it seemed to be a difﬁcult task. Because we

wanted to preserve certain properties, our approach

was to build a fuzzy linear least squares regression

method that could be, in accordance with the concept

of fuzzy orthogonality, extended to a fuzzy orthogo-

nal linear least squares method allowing the indivi-

dual computation of the estimates.

3 NOTATION

Let us deﬁne by ˜x a fuzzy number. We write by

˜x

(·), the membership function. We consider also

the α-cuts of ˜x denoted by ˜x

or by its equivalent

in interval form by [x

L,α

, x

R,α

]. In practice, triangu-

lar fuzzy numbers are often used. We denote them

by a triplet ˜x = (x

, x, x

) with x

≤ x ≤ x

∈ R. In-

dexed fuzzy triangular number will be denoted by

˜x

= (x

, x

). If not speciﬁed otherwise, lower-

case bold letters with no index will be used for n-

components column vectors of fuzzy numbers, e.g.

˜y = (˜y

, . . . , ˜y

)

′

. A (n × m)-matrix is noted in capital

bold letters, e.g.

X, with the i-th line and j-th column

given respectively by ˜x

= ( ˜x

, . . . , ˜x

)

′

, i = 1, . . . , n,

and ˜x

= ( ˜x

1 j

, . . . , ˜x

n j

)

′

, j = 1, . . . , m.

The fuzzy multiplication operator between two

fuzzy numbers is denoted by

⊗. The fuzzy multipli-

cation between two fuzzy triangular numbers ˜a and

is deﬁned as

˜a

⊗

b =(min(a

, a

, b

, a

ab, max(a

, a

, b

, a

)). (1)

See (Viertl, 2018) for more details.

4 FUZZY LEAST SQUARES

REGRESSION

In the following, several types of fuzzy regression

models will be presented. We decided to estimate the

FCTA 2023 - 15th International Conference on Fuzzy Computation Theory and Applications

360

parameters of the models by the least squares tech-

nique. We will ﬁrst deal with two different cases.

In the ﬁrst one, we will analyse a regression model

with crisp independent variables and a fuzzy depen-

dent variable. The second case will involve both fuzzy

independent and dependent variables. Then we will

discuss their constrained and unconstrained solutions.

As we are only interested in the estimation of the pa-

rameters and not to build a statistical model, we won’t

incorporate random error terms in the speciﬁcation of

the model. The fuzziness of the variables will be mod-

elled by triangular fuzzy numbers, and the estimated

parameters too.

4.1 Case 1: Fuzzy Dependent Variable

and Crisp Independent Variables

This case is solved with (Donoso et al., 2006, p.1305-

1306) approach. Let us consider the following regres-

sion model

˜y

∑

j=1

i j

, i = 1, . . . , n, (2)

where

is the j-th parameter of the model. Note that

is assumed to be fuzzy. Our aim is to ﬁnd fuzzy

triangular estimators

that ﬁt (2) best by minimising

the following weighted sum of squares J given in (3):

J =

∑

i=1

−

∑

j=1

i j

)

+ k

−

∑

j=1

i j

)

+ k

−

∑

j=1

i j

)

= k

∑

i=1

− x

′

β)

+ k

∑

i=1

− x

′

)

+ k

∑

i=1

− x

)

, (3)

under the constraints

(β

− β

), (β

− β

) ≥ 0 for j = 1, . . . , m. (4)

The quantities k

, k

and k

are tuning weights associ-

ated with the three sums of squares corresponding to

the central, left and right values of the fuzzy numbers.

The constraints given in (4) make sure the estimators

are fuzzy triangular numbers, i.e.

= (

β,

4.2 Case 2: Fuzzy Dependent Variable

and Fuzzy Independent Variables

Let us suppose now that the independent variables are

also fuzzy, i.e. ˜x

i j

= (x

i j

, x

i j

, x

i j

), i = 1, . . . , n, j =

1, . . . , m. The fuzzy linear regression model becomes:

˜y

∑

j=1

⊗ ˜x

i j

, i = 1, . . . , n, (5)

where

⊗ denotes the fuzzy multiplication operator.

We ﬁnd the fuzzy triangular estimators

β by min-

imising the sum of squares

∑

i=1



˜y

−

∑

j=1

⊗ ˜x

i j



Rewriting it using the fuzzy multiplication rule yields

the following objective function J given in (6):

J =

∑

i=1

−

∑

j=1

i j

)

+ k

−

∑

j=1

(min(β

i j

, β

i j

, β

i j

, β

i j

))

+ k

−

∑

j=1

(max(β

i j

, β

i j

, β

i j

, β

i j

))

(6)

Minimising (6) is not so direct, and a proper strat-

egy needs to be developed. If we know the sign

of the independent triangular fuzzy variable ˜x

i j

and

the sign of the unknown fuzzy parameters

for all

j = 1, ..., m, we can easily know what will be the mini-

mum and maximum values to be calculated in the ob-

jective function (6) and thus solve the regression prob-

lem under the constraints (4). In practice, the sign of

the observations is evidently known, but not the sign

of the parameters. However, on one hand, by using

the crisp value of the fuzzy triangular observations,

i.e. when α = 1, we can perform the classical least

squares to get the estimates

, j = 1, . . . , m, and if

they are sufﬁciently far away from zero, the sign of

will be the one of

. If, on the other hand,

is close

to zero, one has to ﬁrst estimate

, j = 1, . . . , m, via

(3). We note this ﬁrst estimates by

(1)

. We then pick

their signs to ﬁx the ones of the unknown estimators

and β

and use them to solve problem (6).

Suppose we estimated the sign of the un-

known fuzzy parameters as written above and then

chose, according to the signs of the fuzzy ob-

servation ˜x

i j

and

, i = 1, ..., n, j = 0, ..., m

the corresponding minimum x

min

i j

and maximum

max

i j

that solve min(β

i j

, β

i j

, β

i j

, β

i j

) and

Fuzzy Least Squares and Fuzzy Orthogonal Least Squares Linear Regressions

361

max(β

i j

, β

i j

, β

i j

, β

i j

) respectively. The ob-

jective function (6) can then be rewritten as:

J =

∑

i=1

−

∑

j=1

i j

)

+ k

−

∑

j=1

(β

min

i j

)

−

∑

j=1

(β

max

i j

)

, (7)

where β

min

, β

max

are the coefﬁcients associated to

min

i j

and x

max

i j

respectively. The special case where

i j

> 0, i = 1, . . . , n, deserves our attention. In this

case, β

min

and β

max

are obviously equal to β

and β

respectively.

Notice that, in general, when the fuzzy observations

˜x

i j

are not necessarily all positive, the coefﬁcients

min

or β

max

may appear simultaneously in both left

and right squared terms in (6). This is due to the fuzzy

arithmetic rule for the product that sometimes yields

min

or β

max

as the coefﬁcients, which both maximise

and minimise the fuzzy product. This general case is

no longer a simple quadratic problem. This is why

one recommends adding a constant valued vector to

each covariate to ensure their positivity and thus al-

lowing to solve a proprer quadratic problem. Making

so, only the constant of the problem is affected letting

the slopes unchanged.

5 CONSTRAINED AND

UNCONSTRAINED SOLUTIONS

The constraint (4) assures that the solutions are fuzzy

triangular numbers. If we do not impose (4), we end

up with solutions

which are sometimes of the form

(b, c, a) with b > c > a, a, b, c ∈ R. This clearly goes

against the deﬁnition of fuzzy triangular numbers. A

way to bypass this not plausible solution is to consider

the fuzzy parameters as fuzzy intervals.

In order to avoid confusion, it is very important

to make a clear distinction between a vector of fuzzy

numbers and a fuzzy vector. (Viertl, 2018, p.14) de-

ﬁnes a fuzzy vector as:

Deﬁnition 1 (Fuzzy vector and fuzzy interval).

A k-dimensional fuzzy vector ˜x

∗

with membership

function µ

˜x

∗

is such that:

1. µ

˜x

∗

: R

−→ [0, 1].

2. The support of µ

˜x

∗

is a bounded set.

3. ∀α ∈ (0, 1] the so-called α-cut C

( ˜x

∗

) := {x ∈

|µ

˜x

∗

(x) ≥ α} is non-empty, bounded, and

a ﬁnite union of simply connected and closed

bounded sets.

The set of all k-dimensional fuzzy vectors is de-

noted by F (R

). A k-dimensional fuzzy vector is

called a k-dimensional fuzzy interval if all α-cuts are

connected compact sets.

Applying the concept of fuzzy intervals, we are

able to understand the main difference between the

constrained solution and the unconstrained one. In

the ﬁrst case, when the constraints (4) are satisﬁed,

one ends up with a vector of fuzzy parameters

β sat-

isfying β

≤ β

meaning that, the upper regres-

sion coefﬁcients β

must always give a steeper slope

to the regression line, and conversely, β

must al-

ways give a lower slope. When the constraints (4)

are dropped, one ends up with a fuzzy interval

∗

j = 1, . . . , m. In this case, the ﬁtted solution is such

that

˜y

lower

≤

˜y

upper

. This allows the slopes to violate

(4) so long the lower regression line is below the up-

per one on the interval formed by the observations X

As an example, let us consider a regression model

with two crisp independent variables, the ﬁrst one be-

ing the constant of the model. The unconstrained so-

lution

∗

can be written as:

∗

∗,lower

∗

∗,upper

∗,lower

∗

∗,upper

, (8)

where

∗,lower

is the smaller β

coefﬁcient estimated

and

∗,upper

is the greater β

coefﬁcient estimated.

The coefﬁcients

∗,lower

and

∗,upper

are deﬁned anal-

ogously. It can be easily shown that the estimated

parameters (8) satisfy the deﬁnition of a fuzzy inter-

val. Indeed, let C

be the rectangle formed by the

vertices (β

∗,lower,α

, β

∗,lower,α

), (β

∗,lower,α

, β

∗,upper,α

(β

∗,upper,α

, β

∗,lower,α

), (β

∗,upper,α

, β

∗,upper,α

) where α

stands for the α-cut of the fuzzy number. The more

we reduce the fuzziness, that is the closer to 1 α is,

the closer

∗,α

will be to the crisp solution given when

α = 1. Thus, C

⊆ C

for α

≥ α

. Assuming the ex-

istence of a solution, C

is non-empty, bounded and a

ﬁnite union of simply connected and bounded rectan-

gles. Furthermore, The vector-membership function

∗

: R

−→ [0, 1] deﬁned as :

∗

:= max{α · I

(x) : α ∈ (0, 1]}, ∀x ∈ R

has its support bounded by the rectangle C

By following the same reasoning, one can easily

verify that in the case of m covariates, the solution is

also a fuzzy interval, where C

is the m-dimensional

rectangle which has the form of a hypercube of 2

vertices.

FCTA 2023 - 15th International Conference on Fuzzy Computation Theory and Applications

362

6 FUZZY ORTHOGONAL LEAST

SQUARES

Let us ﬁrst remind the deﬁnition of the fuzzy inner

product and the concept of fuzzy orthogonality, de-

scribed by (Ithoh, 2017, p.13).

Deﬁnition 2 (Fuzzy inner product).

Let X be a nonzero vector space over the ﬁeld R

and

X = {x

|x ∈ X, λ ∈ (0, 1]} be the set of all fuzzy

points in X. A function ⟨·, ·⟩ :

X ×

X → R is said to be

a fuzzy inner product on

X if

1. a) ⟨x

, x

⟩ ≥ 0; b) ⟨x

, x

⟩ = 0 if and only if x = 0;

2. ⟨kx

, y

⟩ = k⟨x

, y

⟩;

3. ⟨x

+ y

, z

⟩ = ⟨x

, z

⟩ + ⟨y

, z

⟩;

4. ⟨x

, y

⟩ = ⟨y

, x

⟩;

5. ⟨x

, y

⟩ ≤ ⟨x

, y

⟩ if 0 < ν ≤ µ ≤ 1;

6. for every x

, y

∈

X and ε > 0, there exists 0 <

δ < µ such that ⟨x

, y

µ−δ

⟩ < ⟨x

, y

⟩ + ε;

7. for every x

, y

∈

X and ε > 0, there exists 0 <

δ < 1 − µ such that ⟨x

, y

µ+δ

⟩ > ⟨x

, y

⟩ − ε.

The pair (X, ⟨·, ·⟩) is called a strong fuzzy inner

product space.

Remark 1.

1. Let (X, ⟨·, ·⟩) be a strong fuzzy inner product

space. If

⟨x

, y

⟩ = ⟨x

, y

⟩ x, y ∈ X,

, µ

∈ (0, 1], i = 1, 2,

(9)

then (X, ⟨·, ·⟩) becomes a usual inner product

space;

2. If we use a stronger condition,

⟨x

+ y

, z

⟩ = ⟨x

, y

⟩ + ⟨y

, z

⟩, (10)

then (X, ⟨·, ·⟩) is just a usual inner product space.

The concept of fuzzy orthogonality is deﬁned as:

Deﬁnition 3 (Fuzzy orthogonality).

Let ˜x and ˜y be vectors in a fuzzy inner product

space. One says that ˜x is fuzzy orthogonal to ˜y if

⟨x

, y

⟩ = 0, for some λ, µ ∈ (0, 1].

The fuzzy orthogonality between two vectors of a

fuzzy inner product space ˜x and ˜y is denoted by ˜x ⊥

˜y.

In case x

is orthogonal to y

for these speciﬁc values

of λ, µ ∈ (0, 1], i.e ⟨x

, y

⟩ = 0, we note it x

⊥ y

We can now introduce the fuzzy orthogonal least

squares estimators of a fuzzy regression model. Our

aim is ﬁrst to transform the independent variables in

such a way that at the end of the procedure, the trans-

formed variables are mutually (fuzzy) orthogonal. We

then can use these transformed variables as new re-

gressors in the regression equation and estimate the

model as described in section 4. The orthogonalisa-

tion procedure will be discussed for two cases: inde-

pendent crisp or fuzzy variables.

6.1 Case 3: Fuzzy Dependent Variable

and Crisp Independent Variables

We can use a classic orthonormalisation procedure

to project X onto an orthonormal basis. Let ¯x =

( ¯x

, ..., ¯x

)

′

be the vector of the means associated to

the m independent variables X

, ..., X

; S the esti-

mated variance-covariance matrix; G the matrix of all

the eigenvectors of S; ι

= (1, . . . , 1)

′

a (n × 1) vec-

tor. We project X onto an orthonormal basis using

the transformation

⊥

= (X − ι

¯x

′

)G. (11)

Then we solve (3) with X

⊥

instead of X. Due to

the orthogonality of X

⊥

, the individual coefﬁcients

of the regression model can be estimated also by

regressing the dependent variable on the variable X

only, j = 1, . . . , m. As a consequence, the orthogonal-

isation of the regressors allows us to ﬁnd uncorrelated

fuzzy estimators

Note that, sometimes, by convention, −G is used

instead of G in (11). One has to be careful with mul-

tiplying or not by a minus sign the orthogonal projec-

tion (11) since it can greatly affect the fuzzy triangu-

lar estimators

. This effect will be discussed more

in depth in section 7.3.

6.2 Case 4: Fuzzy Dependent Variable

and Fuzzy Independent Variables

When both the dependent and independent variables

of the regression model are fuzzy, we ﬁrst need to

solve the min() and max() functions in (6). We per-

form then the orthogonalisation procedure (11) to ﬁnd

⊥min

, X

⊥

and X

⊥max

from X

min

, X and X

max

This allows us to write the objective function (7) with

orthogonal fuzzy data as

J =k

∑

i=1

−

∑

j=1

⊥

i j

)

+ k

∑

i=1

−

∑

j=1

⊥min

i j

)

+ k

∑

i=1

−

∑

j=1

⊥max

i j

)

. (12)

Again, the properties of the orthogonality permit

us to compute the fuzzy least squares individually for

Fuzzy Least Squares and Fuzzy Orthogonal Least Squares Linear Regressions

363

each covariate X

while giving the same solution as

when computed with all covariates X

at once.

Observe that the restructuring of the initial data

into ˜x

i j

= (x

min

i j

, x

i j

, x

max

i j

) allowing us to compute the

fuzzy products in (6) is possible if we know the signs

of the fuzzy unknown parameters

β. This ambiguity

can be solved by adequately permuting the observa-

tions to take into account the fuzzy arithmetic. Af-

ter the orthogonalisation procedure, in case of neg-

ative sign occuring, the signs of the observations

˜x

⊥

i j

= (x

⊥min

i j

, x

⊥

i j

, x

⊥max

i j

) may differ from the signs of

˜x

i j

= (x

min

i j

, x

i j

, x

max

i j

To correct this and to preserve both the fuzzy

arithmetic and the quadratic nature of the problem,

one should shift ˜x

j ⊥

, j = 1, . . . , m, by adding a con-

stant K

= |min( ˜x

j ⊥

)|. This operation ensures that

the signs of the observations ˜x

j ⊥

are positive. As

we already said, this transformation has the effect

of changing the intercept value however, it does not

change the value of the slope coefﬁcients. In practical

applications, one may use this transformation to en-

sure that the signs of the observations

X are positive

as well as ensure that the orthonormalised observa-

tions

⊥

are positive too.

Lastly, note that the fuzzy orthogonal covariates ˜x

j ⊥

meet the deﬁnition of fuzzy orthogonality of (Ithoh,

2017). Indeed, by construction their left part, center

and right part are orthogonal at any given α-cut,

⟨ ˜x

j ⊥ L

, ˜x

k ⊥ L

⟩ = 0,

⟨x

j ⊥

, x

k ⊥

⟩ = 0,

⟨ ˜x

j ⊥ R

, ˜x

k ⊥ R

⟩ = 0, ∀ j, k = 1, . . . , m, j ̸= k, (13)

˜x

j ⊥

satisfying Masuo Itoh’s deﬁnition of fuzzy or-

thogonaly.

7 APPLICATION

Let us apply the above fuzzy least squares regression

methods and discuss them. We considered the fuzzy

data set given in Table 1 inspired by (Tanaka and Lee,

1998).

7.1 Fuzzy Dependent Variable and

Crisp Independent Variables

Let us postulate the following fuzzy regression model

(14):

, i = 1, . . . , 22. (14)

Using k

= k

= 1 in (3), we ﬁnd, following the

methodology described in section 4.1 the fuzzy tri-

angular estimates given in Table 2. In Table 2, The

studied model is displayed in the ﬁrst column while

the second column features the fuzzy regression coef-

ﬁcients obtained with the use of a given model under

constraints 4. These fuzzy estimates are fuzzy trian-

gular numbers since they are solutions of constrained

models. The third column of Table 2 depicts the fuzzy

estimates given by solving the models in the ﬁrst col-

umn under no constraint, thus these fuzzy estimates

are of the form of a fuzzy interval, as explained in

deﬁnition 1.

Table 3 gives the sum of squared residuals. The

ﬁrst column tells which model has been used and if

the fuzzy estimates have been obtained under con-

straints or not. Then, the four remaining columns give

the sum of squared residuals with respect to the left,

center and right values and their addition computed

using the objective function of the respective models.

7.2 Fuzzy Dependent Variable and

Fuzzy Independent Variables

We consider again the data in Table 1. We fuzzify

the covariates X and X

by triangular fuzzy numbers.

Let be

X = (X

, X, X

⊗

X = (X

, X

)

with X

= X −u

, X

= X +u

where u

∼ U(0, 0.5)

and u

∼ U(0, 0.7). Note that

is easily computed

since the observations are positive. The problem is

now given by the fuzzy regression equation (15):

⊗

i = 1, . . . , 22. (15)

Using the methodology of section 4.2 the constrained

and unconstrained solutions are given in Table 2 and

the sum of squared residuals in Table 3.

7.3 Fuzzy Dependent Variable and

Crisp Orthogonal Independent

Variables

By means of an SVD decomposition and using (11),

we orthonormalise X onto X

⊥

. The orthonormalised

covariates X

⊥

and X

⊥

of X and X

are given in Table

1. The model is expressed by the regression equation

(16):

0,i

⊥

1,i

2,i

⊥

, i = 1, . . . , 22. (16)

Following the methodology discussed in section 6.1,

the estimated parameters, without and with a multi-

plication of a minus sign, are given in Table 2. Note

FCTA 2023 - 15th International Conference on Fuzzy Computation Theory and Applications

364

Table 1: Fuzzy Dataset.

Y X

X X

⊥

(15,22.5,30) 1 1 1 -1.93184373 -0.35491140

(20,28.75,37.5) 1 2 4 -1.80915821 -0.25981071

(15,25,35) 1 3 9 -1.67727776 -0.17390496

(25,42.5,60) 1 4 16 -1.53620237 -0.09719415

(25,40,55) 1 5 25 -1.38593203 -0.02967828

(40,52.5,65) 1 6 36 -1.22646676 0.02864264

(55,75,95) 1 7 49 -1.05780654 0.07776863

(70,85,100) 1 8 64 -0.87995139 0.11769968

(80,105,130) 1 9 81 -0.69290129 0.14843578

(90,120,150) 1 10 100 -0.49665625 0.16997695

(115,145,175) 1 11 121 -0.29121627 0.18232317

(140,167.5,195) 1 12 144 -0.07658135 0.18547446

(155,187.5,220) 1 13 169 0.14724851 0.17943080

(175,212.5,250) 1 14 196 0.38027331 0.16419220

(200,240,280) 1 15 225 0.62249305 0.13975866

(240,275,310) 1 16 256 0.87390773 0.10613018

(270,305,340) 1 17 289 1.13451735 0.06330676

(300,342.5,385) 1 18 324 1.40432191 0.01128840

(340,380,420) 1 19 361 1.68332142 -0.04992490

(380,420,460) 1 20 400 1.97151586 -0.12033314

(420,460,500) 1 21 441 2.26890525 -0.19993632

(465,507.5,550) 1 22 484 2.57548957 -0.28873445

that for the unconstrained solution (Fuzzy intervals),

the solutions are the same.

As the regressors of model (16) are mutually or-

thogonal, we can ﬁnd the estimates of the parameter

and

by estimating successively the models:

0,i

⊥

1,i

, i = 1, . . . , 22, (17)

and

0,i

2,i

⊥

, i = 1, . . . , 22. (18)

Finally, remark that one can come back to the es-

timates computed with observations X using a linear

transformation. Let G be the matrix of the eigenvec-

tors of the variance-covariance matrix of the regres-

sors,

Gβ

L,⊥

= β

Gβ

⊥

= β,

Gβ

R,⊥

= β

(19)

This transformation only works for the unconstrained

solutions. In the constrained case, this is in general

not true.

At this point, an important observation has to be

made. When using orthogonal observations X

⊥

ﬁnd a different fuzziness for the fuzzy estimators

and

⊥

= (1.010993, 1.010993, 1.010993)

is crisp while

⊥

= (−130.0262, −111.6804, −93.33466)

is fuzzy. This phenomenon is due to the constraints

(4). They force the fuzzy ”slope” represented by

to be steeper for the upper β

and lower for β

. De-

pending on the reference frame used, this constrains

more or less the solutions. Because of this, one has to

be careful with the convention of multiplying or not

a minus sign after having projected the observations

onto an orthonormal basis. The solution

⊥

of model

(16) would have become

⊥





(165.2273, 192.6705, 220.11364)

(−108.2216, −108.2216, −108.2216)

(111.6804, 111.6804, 111.6804)





(20)

if we choose to multiply by a minus sign, and thus

⊥

would also be crisp. Notice, on the other hand, that

the unconstrained solution is unaffected by the choice

of sign.

Fuzzy Least Squares and Fuzzy Orthogonal Least Squares Linear Regressions

365

7.4 Fuzzy Dependent Variable and

Fuzzy Orthogonal Independent

Variables

The model considered is given by the regression equa-

tion (21):

0,i

⊥

1,i

2,i

⊥

2,i

, i = 1, . . . , 22. (21)

The orthogonal covariates X

⊥

and X

⊥

are obtained by

orthogonalisation of

and

as explained in section

6.2. Then, by application of the method discussed

in section 6.2 the resulting estimated parameters are

given in Table 2 and the associated sum of squared

residuals in Table 3. Note that the estimated parame-

ters can be computed individually for each covariate

and yielding the same results. Moreover, using the

transformation (19) with X

min

and X

max

instead of

X for the left, respectively right fuzzy estimates β

L,⊥

R,⊥

will return the original estimates computed with

min

, X and X

max

7.5 General Discussion

Notice that, in table 2, in case 1, 2 and 3 (with mi-

nus sign), the fuzzy interval estimates seem to bet-

ter preserve the fuzziness than the fuzzy triangular

ones. This show that, empirically constrained mod-

els tend to have crispier estimates than unconstrained

ones. Moreover, as depicted in case 3, we can see how

the freedom of multiplying or not by a minus sign the

orthogonal projection (11) affects the fuzziness of the

estimates.

7.6 Sums of Squared Residuals

Lastly, in Table 3, notice that models (14) and its

orthonormalised version (16) share the same sum of

squared residuals. Moroever, this is also true for mod-

els 15 and 21. This could in fact be resulting from the

transformation 19 which allows one to retrieve the es-

timates found with non orthogonal regressors with the

orthonormalised ones.

8 CONCLUSION

We reexamine the fuzzy least squares method to solve

the so-called fuzzy linear regression problems. We

deal with two cases in particular. First, we consid-

ered that the independent variables are crisp, and sec-

ond, we treat the case of fuzzy independent variables.

In both situation, the dependent variable is fuzzy.

We develop a proper strategy to efﬁciently deal with

the fuzziness appearing in the observations. More-

over, we present and discuss two different types of

solutions arising from constrained and unconstrained

fuzzy least squares regression problems which are re-

spectively fuzzy triangular valued and fuzzy interval

valued.

Then, the extension of the method to orthogonal

fuzzy least squares regression methods has been in-

vestigated. In case of (fuzzy) orthogonal indepen-

dent variables, an important property of the classical

least squares method has been preserved. Due to the

orthogonality of the regressors, the individual coef-

ﬁcients

of the regression model can be estimated

also by regressing the dependent variable on the j−th

covariate only. As a consequence, the orthogonalisa-

tion of the regressors allows us to ﬁnd uncorrelated

fuzzy estimators

. Moreover, in the unconstrained

case, we showed that there exist a linear transforma-

tion to ﬁnd the coefﬁcients associated to the original

regression model, i.e. the model before the orthogo-

nalisation of the regressors. We also highlighted that

the resulting sum of squares residuals of a model and

its orthonormal counterpart are the same. These dis-

coveries seem to be very promising to make progress

in statistical inference with fuzzy data, in particular in

the study of the fuzzy distributions of the estimates.

REFERENCES

Donoso, S., Mar

ın, N., and Villa, M. A. (2006). Quadratic

programming models for fuzzy regression. Intelligent

Data Engineering and Automated Learning – IDEAL

2006, 7th International Conference, Burgos, Spain,

September 20-23, 2006. Proceedings. Lecture Notes

in Computer Science, pages 1304–1311.

D’Urso, P. and Massari, R. (2013). Weighted Least Squares

and Least Median Squares estimation for the fuzzy

linear regression analysis. Metron, 71(3):279–306.

Giordani, P. and Kiers, H. A. (2004). Principal Compo-

nent Analysis of symmetric fuzzy data. Computa-

tional Statistics & Data Analysis, 45(3):519–548.

Ithoh, M. (2017). Fuzzy inner product spaces and fuzzy

orthogonality. Advances in Fuzzy Sets and Systems,

5(1-10).

Kashani, M., Arashi, M., Rabiei, M., D’Urso, P., and Gio-

vanni, L. D. (2021). A fuzzy penalized regression

model with variable selection. Expert Systems with

Applications, 175:114696.

Lee, H. and Tanaka, H. (1998). Fuzzy regression analysis

by quadratic programming reﬂecting central tendency.

Metrika, 25(1):65–80.

Lee, H. and Tanaka, H. (1999). Fuzzy approximations with

non-symmetric fuzzy parameters in fuzzy regression

analysis. Journal of the Operations Research Society

of Japan, 42(1):98–112.

FCTA 2023 - 15th International Conference on Fuzzy Computation Theory and Applications

366

Li, Y., He, X., and Liu, X. (2023). Fuzzy multiple linear

least squares regression analysis. Fuzzy Sets and Sys-

tems, 459:118–143.

Mostoﬁan, E., Azhini, M., and Bodaghi, A. (2017). Fuzzy

inner product spaces and fuzzy orthogonality. Tbilisi

Mathematical Journal, 10(2):157–171.

Stanojevi, B. and Stanojevi, M. (2022). Quadratic least

square regression in fuzzy environment. Elsevier B.V.

Tanaka, H. and Lee, H. (1997). Fuzzy linear regression

combining central tendency and possibilistic proper-

ties. Proceedings of 6th International Fuzzy Systems

Conference, 1:63–68.

Tanaka, H. and Lee, H. (1998). Interval regression analysis

by quadratic programming approach. IEEE Transac-

tions on Fuzzy Systems, 6(1):473–481.

Viertl, R. (2018). Statistical Methods for Fuzzy Data. Wi-

ley.

Yabuuch, Y. and Watada, J. (2017). Fuzzy Principal Com-

ponent Analysis and Its Application. International

Journal of Biomedical Soft Computing and Human

Sciences: the ofﬁcial journal of the Biomedical Fuzzy

Systems Association, 3(1):83–92.

Fuzzy Least Squares and Fuzzy Orthogonal Least Squares Linear Regressions

367

APPENDIX

Table 2: Models estimations

Models Fuzzy triangular estimates

Fuzzy intervals

Case 1 (14)

(15.673, 22.605, 29.537)

(−2.1602, −0.3766, 1.4069)

(1.0109, 1.0109, 1.0109)

(20.032, 22.605, 25.178)

(−3.2499, −0.3766, 2.4966)

(1.0583, 1.0109, 0.9636)

Case 2 (15)

(14.473, 21.298, 30.283)

(−1.3693, −0.0499, 0.7347)

(0.9967, 0.9967, 0.9967)

(21.861, 22.605, 23.275)

(−3.4034, −0.3766, 2.4123)

(1.0849, 1.0109, 0.9259)

Case 3 (16) (without (-)

)

(165.22, 192.67, 220.11)

(100.18, 108.22, 116.26)

(−130.02, −111.68, −93.334)

(165.22, 192.67, 220.11)

(100.18, 108.22, 116.26)

(−130.02, −111.68, −93.334)

Case 3 (16) (with (-)

)

(165.22, 192.67, 220.11)

(−108.22, −108.22, −108.22)

(111.68, 111.68, 111.68)

(165.22, 192.67, 220.11)

(100.18, 108.22, 116.26)

(−130.02, −111.68, −93.334)

Case 4 (21) (without (-))

(165.28, 192.67, 220.11)

(100.24, 108.22, 116.26)

(−124.05, −111.68, −91.63)

(165.28, 192.67, 220.11)

(100.24, 108.22, 116.26)

(−124.05, −111.68, −91.62)

Table displaying the fuzzy regression coefﬁcients obtained via a given fuzzy regression model shown in the ﬁrst column.

Solutions of a given constrained fuzzy least squares regression model.

Solutions of a given unconstrained fuzzy least squares regression model.

The orthogonal projection has not been multiplied by a minus sign.

The orthogonal projection has been multiplied by a minus sign.

Table 3: Sum of squared residuals

Models Constraints Left value

Center value

Right value

Total value

(14) constrained 489.8294 256.9274 555.5718 1302.329

(14) unconstrained 426.2246 256.9274 491.9669 1175.119

(15) constrained 555.5714 303.3096 1189.367 2048.248

(15) unconstrained 599.9359 256.9274 630.1572 1487.021

(16) (with (-)

) constrained 489.8294 256.9274 555.5718 1302.329

(16)(w/o (-)

) constrained 3306.798 256.9274 3372.54 6936.265

(16) unconstrained 426.2246 256.9274 491.9669 1175.119

(21) (w/o (-)) constrained &

unconstrained

599.9359 256.9274 630.1572 1487.021

Table displaying the squared residuals of the different models studied throughout this work.

Squared residuals of the left fuzzy parts.

Squared residuals of the central fuzzy parts.

Squared residuals of the right fuzzy parts.

Sum of the left, center and right squared residuals.

The orthogonal projection has been multiplied by a minus sign.

The orthogonal projection has not been multiplied by a minus sign.

FCTA 2023 - 15th International Conference on Fuzzy Computation Theory and Applications

368