Self-Training using Selection Network for Semi-supervised Learning

Jisoo Jeong

∗ a

, Seungeui Lee

∗ b

and Nojun Kwak

† c

Seoul National University, Seoul, South Korea

Keywords:

Semi-supervised Learning.

Abstract:

Semi-supervised learning (SSL) is a study that efﬁciently exploits a large amount of unlabeled data to improve

performance in conditions of limited labeled data. Most of the conventional SSL methods assume that the

classes of unlabeled data are included in the set of classes of labeled data. In addition, these methods do not

sort out useless unlabeled samples and use all the unlabeled data for learning, which is not suitable for realistic

situations. In this paper, we propose an SSL method called selective self-training (SST), which selectively

decides whether to include each unlabeled sample in the training process. It is designed to be applied to

a more real situation where classes of unlabeled data are different from the ones of the labeled data. For the

conventional SSL problems which deal with data where both the labeled and unlabeled samples share the same

class categories, the proposed method not only performs comparable to other conventional SSL algorithms but

also can be combined with other SSL algorithms. While the conventional methods cannot be applied to the

new SSL problems, our method does not show any performance degradation even if the classes of unlabeled

data are different from those of the labeled data.

1 INTRODUCTION

Recently, machine learning has achieved a lot of suc-

cess in various ﬁelds and well-reﬁned datasets are

considered to be one of the most important factors

(Everingham et al., 2010; Krizhevsky et al., 2012;

Russakovsky et al., 2015). Since we cannot discover

the underlying real distribution of data, we need a

lot of samples to estimate it correctly (Nasrabadi,

2007). However, making a large dataset requires a

huge amount of time, cost and manpower (Chapelle

et al., 2009; Odena et al., 2018).

Semi-supervised learning (SSL) is a method re-

lieving the inefﬁciencies in data collection and an-

notation process, which lies between the supervised

learning and unsupervised learning in that both la-

beled and unlabeled data are used in the learning pro-

cess (Chapelle et al., 2009; Odena et al., 2018). It

can efﬁciently learn a model from fewer labeled data

using a large amount of unlabeled data (Zhu, 2006).

Accordingly, the signiﬁcance of SSL has been stud-

ied extensively in the previous literatures (Zhu et al.,

https://orcid.org/0000-0003-1154-4532

https://orcid.org/0000-0003-3560-785X

https://orcid.org/0000-0002-1792-0327

* Equal contribution

†

Corresponding author.

2003; Rosenberg et al., 2005; Kingma et al., 2014;

Rasmus et al., 2015; Odena, 2016; Akhmedova et al.,

2017). These results suggest that SSL can be a useful

approach in cases where the amount of annotated data

is insufﬁcient.

However, there is a recent research discussing

the limitations of conventional SSL methods (Odena

et al., 2018). They have pointed out that conventional

SSL algorithms are difﬁcult to be applied to real ap-

plications. Especially, the conventional methods as-

sume that all the unlabeled data belong to one of the

classes of the training labeled data. Training with un-

labeled samples whose class distribution is signiﬁ-

cantly different from that of the labeled data may de-

grade the performance of traditional SSL methods.

Furthermore, whenever a new set of data is available,

they should be trained from the scratch using all the

data including out-of-class

data.

In this paper, we focus on the classiﬁcation task

and propose a deep neural network based approach

named as selective self-training (SST) to solve the

limitation mentioned above. To enable learning to se-

The term out-of-class is used to denote the situation

where the new dataset contains samples originated from

different classes than the classes of the old data. On the

other hand, the term in-class is used when the new data con-

tain only the samples belonging to the previously observed

classes.

Jeong, J., Lee, S. and Kwak, N.

Self-Training using Selection Network for Semi-supervised Learning.

DOI: 10.5220/0008940900230032

In Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods (ICPRAM 2020), pages 23-32

ISBN: 978-989-758-397-1; ISSN: 2184-4313

lect unlabeled data, we propose a selection network,

which is based on the deep neural network, that de-

cides whether each sample is to be added or not. Dif-

ferent from (Wang et al., 2018), SST does not di-

rectly use the classiﬁcation results for the data selec-

tion. Also, we adopt an ensemble approach which is

similar to the co-training method (Blum and Mitchell,

1998) that utilizes outputs of multiple classiﬁers to

iteratively build a new training dataset. In our case,

instead of using multiple classiﬁers, we apply a tem-

poral ensemble method to the selection network. For

each unlabeled instance, two consecutive outputs of

the selection network are compared to keep our train-

ing data clean.

In addition, we have found that the balance be-

tween the number of samples per class is quite impor-

tant for the performance of our network. We suggest

a simple heuristics to balance the number of selected

samples among the classes. By the proposed selection

method, reliable samples can be added to the training

set and uncertain samples including out-of-class data

can be excluded. The main contributions of the pro-

posed method can be summarized as follows:

• For the conventional SSL problems, the proposed

SST method not only performs comparable to

other conventional SSL algorithms but also can be

combined with other algorithms.

• For the new SSL problems, the proposed SST

does not show any performance degradation even

with the out-of-class data.

• SST requires few hyper-parameters and can be

easily implemented.

To prove the effectiveness of our proposed

method, ﬁrst, we conduct experiments comparing the

classiﬁcation errors of SST and several other state-of-

the-art SSL methods (Laine and Aila, 2016; Tarvainen

and Valpola, 2017; Luo et al., 2017; Miyato et al.,

2017) in conventional SSL settings. Second, we pro-

pose a new experimental setup to investigate whether

our method is more applicable to real-world situa-

tions. The experimental setup in (Odena et al., 2018)

samples classes among in-classes and out-classes. In

the experimental setting in this paper, we sample un-

labeled instances evenly in all classes. We evaluate

the performance of the proposed SST using three

public benchmark datasets: CIFAR-10, CIFAR-100

(Krizhevsky and Hinton, 2009), and SVHN (Netzer

et al., 2011).

2 BACKGROUND

In this section, we introduce the background of our

research. First, we introduce some methods of self-

training (McLachlan, 1975; Zhu, 2007; Zhu and

Goldberg, 2009) on which our work is based. Then we

describe consistency regularization-based algorithms

such as Π model and temporal ensembling (Laine and

Aila, 2016).

2.1 Self-training

Self-training method has long been used for semi-

supervised learning (McLachlan, 1975; Rosenberg

et al., 2005; Zhu, 2007; Zhu and Goldberg, 2009). It

is a resampling technique that repeatedly labels unla-

beled training samples based on the conﬁdence scores

and retrains itself with the selected pseudo-annotated

data. This process can be formalized as follows. (i)

Training a model with labeled data. (ii) Predicting un-

labeled data with the learned model. (iii) Retraining

the model with labeled and selected pseudo-labeled

data. (iv) Repeating the last two steps.

However, most self-training methods assume that

the labeled and unlabeled data are generated from the

identical distribution. Therefore, in real-world sce-

narios, some instances with low likelihood accord-

ing to the distribution of the labeled data are likely

to be misclassiﬁed inevitably. Consequently, these er-

roneous samples signiﬁcantly lead to worse results in

the next training step. To alleviate this problem, we

adopt the ensemble and balancing methods to select

reliable samples.

2.2 Consistency Regularization

Consistency regularization is one of the popular SSL

methods and has been referred to many recent re-

searches (Laine and Aila, 2016; Miyato et al., 2017;

Tarvainen and Valpola, 2017). Among them, Π model

and temporal ensembling are widely used (Laine and

Aila, 2016). They have deﬁned new loss functions

for unlabeled data. The Π model outputs f (x) and

f (x) for the same input x by perturbing the input

with different random noise and using dropout (Sri-

vastava et al., 2014), and then minimizes the dif-

ference (k f (x) −

f (x)k

) between these output val-

ues. Temporal ensembling does not make different

predictions f (x) and

f (x), but minimizes the differ-

ence (k f

t−1

(x) − f

(x)k

) between the outputs of two

consecutive iterations for computational efﬁciency. In

spite of the improvement in performance, they re-

quire lots of things to consider for training. These

methods have various hyper-parameters such as ‘ramp

ICPRAM 2020 - 9th International Conference on Pattern Recognition Applications and Methods

Figure 1: An overview of the proposed SST. Different shapes represent the input data with different underlying distribution,

and different colors (orange, blue, and green) are for different classes. In the initial training dataset, only three classes with

their corresponding distributions (#, 2, 4) exist and are used for initial training. Then the unlabeled data which include

unseen distribution (F, C) are inputted to the classiﬁcation as well as the selection network. At the bottom right, unlabeled

samples with higher selection network output values than a certain threshold are denoted by yellow and selected to be included

in the training process for the next iteration, while the remaining are not used for training.

up’, ‘ramp down’, ‘unsupervised loss weight’ and

so on. In addition, customized settings for training

such as ZCA preprocessing and mean-only batch nor-

malization (Salimans and Kingma, 2016) are also

very important aspects for improving the performance

(Odena et al., 2018).

3 METHOD

In this section, we introduce our selective self-training

(SST) method. The proposed model consists of three

networks as shown in the bottom part of Figure 1.

The output of the backbone network is fed into two

sibling fully-connected layers —a classiﬁcation net-

work f

(·;θ

) and a selection network f

sel

(·;θ

where θ

and θ

are learnable parameters for each of

them. In this paper, we deﬁne the classiﬁcation result

and the selection score as r

= f

( f

(·;θ

);θ

) and

= f

sel

( f

(·;θ

);θ

), respectively, where f

(·;θ

)

denotes the backbone network with learnable param-

eters θ

. Note that we deﬁne r

as the resultant label

and it belongs to one of the class labels r

∈ Y =

{1, 2, ··· ,C}. As shown in Figure 1, the proposed SST

method can be represented in the following four steps.

First, SST trains the network using a set of the labeled

data L = {(x

, y

) | i = 1, ··· , L}, where x

and y

∈

{1, 2, ··· ,C} denote the data and the ground truth la-

bel respectively, which is a standard supervised learn-

ing method. The next step is to predict all the unla-

beled data U = {x

|i = L +1, ··· , N} and select a sub-

set of the unlabeled data {x

|i ∈ I

} whose data have

high selection scores with the current trained model,

where I

denotes a set of selected sample indices from

= {L + 1, ·· · , N}. Then, we annotate the selected

samples with the pseudo-categories ˆy

evaluated by

the f

(·;θ

) and construct a new training dataset T

composed of L and U

= {(x

, ˆy

)|i ∈ I

}. After that,

we retrain the model with T and repeat this process it-

eratively. The overall process of the SST is described

in Algorithm 1 and the details of each of the four steps

will be described later.

3.1 Supervised Learning

The SST algorithm ﬁrst trains a model with super-

vised learning. At this time, the entire model (all

three networks) is trained simultaneously. f

(·;θ

)

is trained using the softmax function and the cross-

entropy loss as in the ordinary supervised classiﬁca-

tion learning task. In case of f

sel

(·;θ

), the training la-

bels are motivated by discriminator of generative ad-

versarial networks (GAN) (Goodfellow et al., 2014;

Yoo et al., 2017). When x

with y

is fed into the net-

work, the target for f

sel

(·;θ

) is set as:

(

1, if r

= y

for i ∈ I

0, if r

6= y

for i ∈ I

(1)

where I

= {1, ··· , L} represents a set of labeled sam-

ple indices. f

sel

(·;θ

) is trained with the generated tar-

get g

. Especially, we use the sigmoid function for the

ﬁnal activation and the binary cross-entropy loss to

Self-Training using Selection Network for Semi-supervised Learning

Algorithm 1: Training procedure of the proposed SST.

Require: x

, y

: training data and label

Require: L, U: labeled and unlabeled datasets

Require: I

: set of unlabeled sample indices

Require: f

(·;θ

), f

(·;θ

), f

sel

(·;θ

): trainable SST

model

Require: α, ε, K, K

: hyper-parameters

1: randomly initialize θ

, θ

2: train f

(·;θ

), f

(·;θ

), f

sel

(·;θ

) for K epochs

using L

3: repeat

4: initialize r

= −1, I

= ∅

5: for each i ∈ I

6: r

t−1

← r

, r

← f

( f

;θ

);θ

)

7: s

← f

sel

( f

;θ

);θ

)

8: if r

t−1

6= r

then

9: z

← 0

10: end if

11: z

← αz

+ (1 − α)s

12: if z

> 1 − ε then

13: I

← I

∪ {i}

14: assign label for x

using r

15: end if

16: end for

17: update U

with data balancing

18: T ← L ∪ U

19: retrain f

(·;θ

), f

(·;θ

), f

sel

(·;θ

) for K

epochs using T

20: until stopping criterion is true

train f

sel

(·;θ

). Therefore our f

sel

(·;θ

) does not uti-

lize the conventional softmax function because it pro-

duces a relative output and can induce a high value

even for an out-of-class sample. Instead, our f

sel

(·;θ

)

is designed to estimate an absolute conﬁdence score

using the sigmoid activation function. More details

are provided in appendix. Consequently, our ﬁnal loss

function is a sum of the classiﬁcation loss L

and the

selection loss L

sel

total

= L

+ L

sel

. (2)

3.2 Prediction and Selection

After learning the model in a supervised manner, SST

takes all instances of U as input and predicts r

and

, for all i ∈ I

. We utilize r

and s

to annotate and

choose unlabeled samples, respectively. In the context

of self-training, removing erroneously annotated sam-

ples is one of the most important things for the new

training dataset. Thus, we adopt temporal co-training

and ensemble methods for the selection score in order

to keep our training set from contamination. First, let

Table 1: Ablation study with 5 runs on the CIFAR-10

dataset. ‘bal’ denotes the usage of data balancing scheme

during data addition as described in Sec. 3.3, ‘ens’ is for

the usage of previous selection scores as in the 11th line of

Algorithm 1.

method bal ens error

supervised learning 18.97 ± 0.37%

SST

x x 21.44 ± 4.05%

o x 14.43 ± 0.43%

o o 11.82 ± 0.40%

and r

t−1

be the classiﬁcation results of the current

and the previous iterations respectively and we uti-

lize the temporal consistency of these values. If these

values are different, we set the ensemble score z

= 0

to reduce uncertainty in selecting unlabeled samples.

Second, inspired by (Laine and Aila, 2016), we also

utilize multiple previous network evaluations of un-

labeled instances by updating z

= αz

+ (1 − α)s

where α is a momentum weight for the moving aver-

age of ensemble scores. However, the aim of our en-

sembling approach is different from (Laine and Aila,

2016). They want to alleviate different predictions for

the same input, which are resulted from different aug-

mentation and noise to the input. However, our aim

differs from theirs in that we are interested in select-

ing reliable (pseudo-)labeled samples. After that, we

select unlabeled samples with high z

. It is very im-

portant to set an appropriate threshold because it de-

cides the quality of the added unlabeled samples for

the next training. If f

(·;θ

) is trained well on the la-

beled data, the training accuracy would be very high.

Also, since f

sel

(·;θ

) is trained with g

generated from

and y

, s

will be close to 1.0. Therefore, we set the

threshold to 1− ε and control it by changing ε. In this

case, if z

exceeds 1 − ε, the pseudo-label of the unla-

beled sample ˆy

is set to r

3.3 New Training Dataset

When we construct T , we keep the number of sam-

ples of each class the same. The reason is that if

one class dominates the others, the classiﬁcation per-

formance is degraded by the imbalanced distribution

(FernáNdez et al., 2013). We also empirically found

that naively creating a new training dataset fails to

yield good performance. In order to fairly transfer the

selected samples to the new training set, the amount

of migration in each class should not exceed the num-

ber of the class having the least selected samples.

We take arbitrary samples in every class as much as

the maximum number satisfying this condition. T is

composed of both L and U

. The number of selected

unlabeled samples is the same for all classes.

ICPRAM 2020 - 9th International Conference on Pattern Recognition Applications and Methods

Table 2: Classiﬁcation error on CIFAR-10 (4k Labels), SVHN (1k Labels), and CIFAR-100 (10k Labels) with 5 runs using

in-class unlabeled data (* denotes that the test has been done by ourselves).

Method CIFAR-10 SVHN CIFAR-100

Supervised (sampled)* 18.97 ± 0.37% 13.45 ± 0.92% 40.24 ± 0.45%

Supervised (all)* 5.57 ± 0.07% 2.87 ± 0.06% 23.36 ± 0.27%

Mean Teacher (Tarvainen and Valpola, 2017) 12.31 ± 0.28% 3.95 ± 0.21% -

Π model (Laine and Aila, 2016) 12.36 ± 0.31% 4.82 ± 0.17% 39.19 ± 0.36%

TempEns (Laine and Aila, 2016) 12.16 ± 0.24% 4.42 ± 0.16% 38.65 ± 0.51%

TempEns + SNTG (Luo et al., 2017) 10.93 ± 0.14% 3.98 ± 0.21% 40.19 ± 0.51%*

VAT (Miyato et al., 2017) 11.36 ± 0.34% 5.42 ± 0.22% -

VAT + EntMin (Miyato et al., 2017) 10.55 ± 0.05% 3.86 ± 0.11% -

pseudo-label (Lee, 2013; Odena et al., 2018) 17.78 ± 0.57% 7.62 ± 0.29% -

Proposed method (SST)* 11.82 ± 0.40% 6.88 ± 0.59% 34.89 ± 0.75%

SST + TempEns + SNTG* 9.99 ± 0.31% 4.74 ± 0.19% 34.94 ± 0.54%

3.4 Re-training

After combining the labeled and selected pseudo-

labeled data, the model is retrained with the new

dataset for K

epochs. In this step, the label for the

sel

(·;θ

) is obtained by a process similar to Eq. (1).

Above steps (except for Section 3.1) are repeated for

M iterations until (near-) convergence.

4 EXPERIMENTS

To evaluate our proposed SST algorithm, we conduct

two types of experiments. First, we evaluate the pro-

posed SST algorithm for the conventional SSL prob-

lem where all unlabeled data are in-class. Then, SST

is evaluated with the new SSL problem where some

of the unlabeled data are out-of-class.

In the case of in-class data, gradually gathering

highly conﬁdent samples in U can help improve the

performance. On the other hand, in the case of out-

of-class data, a strict threshold is preferred to pre-

vent uncertain out-of-class data from being involved

in the new training set. Therefore, we have experi-

mented with decay mode that decreases the thresh-

old in log-scale and ﬁxed mode that ﬁxes the thresh-

old in the way described in Section 4.2. We have ex-

perimented our method with 100 iterations and deter-

mined ε by cross-validation in decay modes. In case

of ﬁxed modes, ε is ﬁxed and the number of iteration

is determined by cross-validation. The details about

the experimental setup is presented in appendix.

4.1 Conventional SSL Problems with

In-class Unlabeled Data

We experiment with three popular datasets which are

SVHN, CIFAR-10, and CIFAR-100 (Netzer et al.,

2011; Krizhevsky et al., 2014). The settings of labeled

versus unlabeled data separation for each dataset are

the same with (Laine and Aila, 2016; Miyato et al.,

2017; Tarvainen and Valpola, 2017). More details are

provided in appendix. Also, the network is the same

with (Laine and Aila, 2016).

4.1.1 Ablation Study

We have performed experiments on CIFAR-10 dataset

with the combination of two types of components. As

described in Table 1, these are whether to use data

balancing scheme described in Section 3.3 (balance),

whether to use selection score ensemble in the 11th

line of Algorithm 1 (ensemble). First, when SST does

not use all of these, the error 21.44% is higher than

that of the supervised learning which does not use

any unlabeled data. This is due to the problem of un-

balanced data mentioned in subsection 3.3. When the

data balance is used, the error is 14.43%, which is bet-

ter than the baseline 18.97%. Adding the ensemble

scheme results in 11.82% error. Therefore, we have

used only balance and ensemble schemes in the fol-

lowing experiments.

4.1.2 Experimental Results

Table 2 shows the experiment results of supervised

learning, conventional SSL algorithms and the pro-

posed SST on CIFAR-10, SVHN and CIFAR-100

datasets. Our baseline model with supervised learning

performs slightly better than what has been reported

in other papers (Laine and Aila, 2016; Tarvainen and

Valpola, 2017; Luo et al., 2017) because of our dif-

ferent settings such as Gaussian noise on inputs, opti-

mizer selection, the mean-only batch normalizations

and the learning rate parameters. For all the datasets,

we have also performed experiments with a model of

SST combined with the temporal ensembling (Tem-

Self-Training using Selection Network for Semi-supervised Learning

Figure 2: SST result on CIFAR-10, SVHN, and CIFAR-100 datasets with 5 runs. The x-axis is the iteration, the blue circle is

the average of the number of data used for training, and the red diamond is the average accuracy.

(a) (b)

Figure 3: Result of new SSL problems on CIFAR-10 dataset with 5 runs. (a) number of data with iteration in decay mode

(b) accuracy with iteration in decay mode (c) number of data with iteration in ﬁxed mode(d) accuracy with iteration in ﬁxed

mode. % means the ratio of the number of non-animal classes in the unlabeled data.

pEns) and SNTG, labeled as SST+TempEns+SNTG.

For the model, the pseudo-labels of SST at the last it-

eration is considered as the true class label. Figure 2

shows the number of samples used in the training and

the corresponding accuracy on the test set for each

dataset.

CIFAR-10: The baseline network yields the test

error of 18.97% and 5.57% when trained with 4,000

(sampled) and 50,000 (all) labeled images respec-

tively. The test error of our SST method reaches

11.82% which is comparable to other algorithms

while SST+TempEns+SNTG model results 1.83%

better than the SST-only model.

SVHN: The baseline model for SVHN dataset is

trained with 1,000 labeled images and yields the test

error of 13.45%. Our proposed method has an er-

ror of 6.88% which is relatively higher than those

of other SSL algorithms. Performing better than SST,

SST+TempEns+SNTG reaches 4.74% of error which

is worse than that of TempEns+SNTG model. We

suspect two reasons for this. The ﬁrst is that SVHN

dataset is not well balanced, and the second is that

SVHN is a relatively easy dataset, so it seems to be

easily added to the hard labels. With data balancing,

the SST is still worse than other algorithms. We think

this phenomenon owes to the use of hard labels in

SST where incorrectly estimated samples deteriorate

the performance.

CIFAR-100: While the baseline model results in

40.24% of error rate through supervised learning with

sampled data, our method performs with 34.89% of

error, enhancing the performance by 5.3%. We have

observed that the performance of TempEns+SNTG is

lower than TempEns, and when TempEns+SNTG is

added to SST, performance is degraded slightly. Al-

though TempEng+SNTG shows better performance

than TempEng without augmentation in (Luo et al.,

2017), its performance is worse than that of TempEng

with augmentation in our experiment. The reason for

this can be conjectured that the hyper-parameter in the

current temporal ensembling and SNTG may not have

been optimized

We have reproduced tempEns+SNTG model with a Py-

torch implementation, and have veriﬁed of its performance

on CIFAR-10 and SVHN akin to what is reported in (Luo

et al., 2017). However, for CIFAR-100 dataset, since the

experimental results without data augmentation are not re-

ported, we thus report our reproduced results.

ICPRAM 2020 - 9th International Conference on Pattern Recognition Applications and Methods

Table 3: Classiﬁcation error for new SSL problems on CIFAR-10 and CIFAR-100 dataset with 5 runs. ’%’ means the ratio of

the number of non-animal classes.

dataset CIFAR-10 CIFAR-100

method SST(decay) SST(ﬁxed) SST(decay) SST(ﬁxed)

supervised 22.27 ± 0.47% 34.62 ± 1.14%

0% 14.99 ± 0.54% 17.84 ± 0.39% 28.01 ± 0.44% 32.16 ± 0.64%

25%

17.93 ± 0.33% 18.38 ± 0.52% 29.94 ± 0.45% 32.28 ± 0.58%

50% 20.91 ± 0.53% 19.04 ± 0.63% 31.78 ± 0.62% 32.60 ± 0.67%

75% 22.72 ± 0.42% 20.07 ± 0.98% 34.44 ± 0.85% 32.32 ± 0.52%

100% 26.78 ± 1.35% 20.24 ± 0.15% 37.17 ± 1.08% 32.62 ± 0.63%

4.2 New SSL Problems with

Out-of-class Unlabeled Data

We have experimented with the following settings

for real-world applications. The dataset is categorized

into six animal and four non-animal classes as simi-

larly done in (Odena et al., 2018). In CIFAR-10, 400

images per animal class are used as the labeled data

(total 2,400 images for 6 animal classes) and a pool

of 20,000 images with different mixtures of both an-

imal and non-animal classes are experimented as an

unlabeled dataset. In CIFAR-100, 5,000 labeled data

(100 images per animal class) and a total of 20,000

unlabeled images of both classes with different mixed

ratios are utilized. Unlike the experimental setting in

(Odena et al., 2018), we have experimented according

to the ratio (%) of the number of out-of-class data in

the unlabeled dataset.

As mentioned above, in the presence of out-of-

class samples, a strict threshold is required. If all of

the unlabeled data is assumed to be in-class, the de-

cay mode may be a good choice. However, in many

real-applications, out-of-class unlabeled data is also

added to the training set in the decay mode and causes

poor performance. In avoidance of such matter, we

have experimented on a ﬁxed mode of criterion thresh-

old on adding the unlabeled data. Unlike the decay

mode that decrements the threshold value, SST in

the ﬁxed mode sets a ﬁxed threshold at a reasonably

high value throughout the training. Our method in the

ﬁxed mode should be considered more suitable for

real-applications but empirically shows lower perfor-

mances in Figure 3 and Table 3 than when running

in the decay mode. The difference between the decay

mode and the ﬁxed mode are an unchangeable ε and

the initial ensemble.

Setting a threshold value for the ﬁxed mode is

critical for a feasible comparison against the decay

mode. Figure 3 shows the average of the results ob-

tained when performing SST ﬁve times for each ratio

in CIFAR-10. As shown in Figure 3(a), as the number

of iteration increases, the threshold in the decay mode

decreases and the number of additional unlabeled data

increases. Obviously, while the different percentage

of the non-animal data inclusion show different trends

of training, in the cases of 0 ∼ 75% of non-animal

data included in the unlabeled dataset, the addition-

ally selected training data shows an initial increase at

∼ 40

iteration. Also, when the unlabeled dataset

is composed of only the out-of-class data, selective

data addition of our method initiates at 55

∼ 65

training iteration. This tendency has been observed in

previous researches on classiﬁcation problems and we

have set the threshold value ﬁxed at a value between

two initiating points of data addition as similarly done

in the works of (Viola and Jones, 2001; Zhang and Vi-

ola, 2008). We have set the ﬁxed threshold based on

47th iteration (between 40 and 55). For a more re-

liable selection score, we have not added any unla-

beled data to the new training set and have trained our

method with the labeled data only for 5 iterations.

As it can be seen in Table 3, in the case of SST in

the decay mode, the performance has been improved

when the unlabeled dataset consists only in-class an-

imal data, but when the unlabeled pool is ﬁlled with

only out-of-class data, the performance is degraded.

For the case of SST with a ﬁxed threshold value, sam-

ples are not added and the performance was not de-

graded at 100% non-animal ratio as shown in Figure

3(c). Furthermore, at 0% of out-of-class samples in

the pool, there is more improvement in the perfor-

mance than at 100 % of out-of-class samples while

still being inferior to the improvement than the decay

mode. Because less but stable data samples are added

by SST with a ﬁxed threshold, the performance is im-

proved for all the cases compared to that of supervised

learning. Therefore, it is more suitable for real appli-

cations where the origin of data is usually unknown.

5 CONCLUSION

We proposed selective self-training (SST) for semi-

supervised learning (SSL) problem. SST selectively

samples unlabeled data and trains the model with a

subset of the dataset. Using selection network, reli-

able samples can be added to the new training dataset.

In this paper, we conduct two types of experiments.

Self-Training using Selection Network for Semi-supervised Learning

First, we experiment with the assumption that unla-

beled data are in-class like conventional SSL prob-

lems. Then, we experiment how SST performs for

out-of-class unlabeled data.

For the conventional SSL problems, we achieved

competitive results on several datasets and our

method could be combined with conventional algo-

rithms to improve performance. The accuracy of SST

is either saturated or not depending on the dataset.

Nonetheless, SST has shown performance improve-

ments as a number of data increases. In addition, the

results of the combined experiments of SST and other

algorithms show the possibility of performance im-

provement.

For the new SSL problems, SST did not show any

performance degradation even if the model is learned

from in-class data and out-of-class unlabeled data.

Decreasing the threshold of the selection network in

new SSL problem, performance degrades. However,

the output of the selection network shows different

trends according to in-class and out-of-class. By set-

ting a threshold that does not add out-of-class data,

SST has prevented the addition of out-of-class sam-

ples to the new training dataset. It means that it is pos-

sible to prevent the erroneous data from being added

to the unlabeled dataset in a real environment.

ACKNOWLEDGEMENTS

This work was supported by IITP grant funded by the

Korea government (MSIT) (No.2019-0-01367).

REFERENCES

Akhmedova, S., Semenkin, E., and Stanovov, V. (2017).

Semi-supervised svm with fuzzy controlled coopera-

tion of biology related algorithms. In ICINCO (1),

pages 64–71.

Blum, A. and Mitchell, T. (1998). Combining labeled and

unlabeled data with co-training. In Proceedings of the

eleventh annual conference on Computational learn-

ing theory, pages 92–100. ACM.

Chapelle, O., Scholkopf, B., and Zien, A. (2009).

Semi-supervised learning (chapelle, o. et al., eds.;

2006)[book reviews]. IEEE Transactions on Neural

Networks, 20(3):542–542.

Everingham, M., Van Gool, L., Williams, C. K., Winn, J.,

and Zisserman, A. (2010). The pascal visual object

classes (voc) challenge. International journal of com-

puter vision, 88(2):303–338.

FernáNdez, A., LóPez, V., Galar, M., Del Jesus, M. J., and

Herrera, F. (2013). Analysing the classiﬁcation of im-

balanced data-sets with multiple classes: Binarization

techniques and ad-hoc approaches. Knowledge-based

systems, 42:97–110.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,

Warde-Farley, D., Ozair, S., Courville, A., and Ben-

gio, Y. (2014). Generative adversarial nets. In

Advances in neural information processing systems,

pages 2672–2680.

Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling,

M. (2014). Semi-supervised learning with deep gen-

erative models. In Advances in Neural Information

Processing Systems, pages 3581–3589.

Krizhevsky, A. and Hinton, G. (2009). Learning multiple

layers of features from tiny images. Technical report,

Citeseer.

Krizhevsky, A., Nair, V., and Hinton, G. (2014). The

cifar-10 dataset. online: http://www. cs. toronto.

edu/kriz/cifar. html.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Im-

agenet classiﬁcation with deep convolutional neural

networks. In Advances in neural information process-

ing systems, pages 1097–1105.

Laine, S. and Aila, T. (2016). Temporal ensem-

bling for semi-supervised learning. arXiv preprint

arXiv:1610.02242.

Lee, D.-H. (2013). Pseudo-label: The simple and efﬁcient

semi-supervised learning method for deep neural net-

works. In Workshop on Challenges in Representation

Learning, ICML, volume 3, page 2.

Luo, Y., Zhu, J., Li, M., Ren, Y., and Zhang, B.

(2017). Smooth neighbors on teacher graphs

for semi-supervised learning. arXiv preprint

arXiv:1711.00258.

McLachlan, G. J. (1975). Iterative reclassiﬁcation proce-

dure for constructing an asymptotically optimal rule

of allocation in discriminant analysis. Journal of the

American Statistical Association, 70(350):365–369.

Miyato, T., Maeda, S.-i., Koyama, M., and Ishii, S. (2017).

Virtual adversarial training: a regularization method

for supervised and semi-supervised learning. arXiv

preprint arXiv:1704.03976.

Nasrabadi, N. M. (2007). Pattern recognition and ma-

chine learning. Journal of electronic imaging,

16(4):049901.

Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and

Ng, A. Y. (2011). Reading digits in natural images

with unsupervised feature learning. In NIPS workshop

on deep learning and unsupervised feature learning,

volume 2011, page 5.

Odena, A. (2016). Semi-supervised learning with

generative adversarial networks. arXiv preprint

arXiv:1606.01583.

Odena, A., Oliver, A., Raffel, C., Cubuk, E. D., and

Goodfellow, I. (2018). Realistic evaluation of semi-

supervised learning algorithms.

Rasmus, A., Berglund, M., Honkala, M., Valpola, H., and

Raiko, T. (2015). Semi-supervised learning with lad-

der networks. In Advances in Neural Information Pro-

cessing Systems, pages 3546–3554.

ICPRAM 2020 - 9th International Conference on Pattern Recognition Applications and Methods

Rosenberg, C., Hebert, M., and Schneiderman, H. (2005).

Semi-supervised self-training of object detection

models. In WACV/MOTION, pages 29–36.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S.,

Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern-

stein, M., et al. (2015). Imagenet large scale visual

recognition challenge. International Journal of Com-

puter Vision, 115(3):211–252.

Salimans, T. and Kingma, D. P. (2016). Weight normaliza-

tion: A simple reparameterization to accelerate train-

ing of deep neural networks. In Advances in Neural

Information Processing Systems, pages 901–909.

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and

Salakhutdinov, R. (2014). Dropout: a simple way to

prevent neural networks from overﬁtting. The Journal

of Machine Learning Research, 15(1):1929–1958.

Tarvainen, A. and Valpola, H. (2017). Mean teachers are

better role models: Weight-averaged consistency tar-

gets improve semi-supervised deep learning results. In

Advances in neural information processing systems,

pages 1195–1204.

Viola, P. and Jones, M. (2001). Rapid object detection using

a boosted cascade of simple features. In Computer Vi-

sion and Pattern Recognition, 2001. CVPR 2001. Pro-

ceedings of the 2001 IEEE Computer Society Confer-

ence on, volume 1, pages I–I. IEEE.

Wang, K., Lin, L., Yan, X., Chen, Z., Zhang, D., and

Zhang, L. (2018). Cost-effective object detection: Ac-

tive sample mining with switchable selection criteria.

IEEE Transactions on Neural Networks and Learning

Systems, (99):1–17.

Yoo, Y., Park, S., Choi, J., Yun, S., and Kwak, N. (2017).

Butterﬂy effect: Bidirectional control of classiﬁcation

performance by small additive perturbation. arXiv

preprint arXiv:1711.09681.

Zhang, C. and Viola, P. A. (2008). Multiple-instance

pruning for learning efﬁcient cascade detectors. In

Advances in neural information processing systems,

pages 1681–1688.

Zhu, X. (2006). Semi-supervised learning literature survey.

Computer Science, University of Wisconsin-Madison,

2(3):4.

Zhu, X. (2007). Semi-supervised learning tutorial. In In-

ternational Conference on Machine Learning (ICML),

pages 1–135.

Zhu, X., Ghahramani, Z., and Lafferty, J. D. (2003). Semi-

supervised learning using gaussian ﬁelds and har-

monic functions. In Proceedings of the 20th Inter-

national conference on Machine learning (ICML-03),

pages 912–919.

Zhu, X. and Goldberg, A. B. (2009). Introduction to semi-

supervised learning. Synthesis lectures on artiﬁcial

intelligence and machine learning, 3(1):1–130.

APPENDIX

The Basic Settings of Our Experiments

The basic settings of our experiments are as follows.

Different from (Laine and Aila, 2016; Luo et al.,

2017), we use stochastic gradient descent (SGD) with

a weight decay of 0.0005 as an optimizer. The mo-

mentum weight for the ensemble of selection scores is

set to α = 0.5. Also, we do not apply mean-only batch

normalization layer (Salimans and Kingma, 2016)

and Gaussian noise. We follow the same data augmen-

tation scheme in (Laine and Aila, 2016) consisting

of horizontal ﬂips and random translations. However,

ZCA whitening is not used. In the supervised learning

phase, we train our model using batch size 100 for 300

epochs. After that, in the retraining phase, we train us-

ing the same batch size for 150 epochs with the new

training dataset. The learning rate starts from 0.1. In

the supervised learning phase, it is divided by 10 at

the 150-th and 225-th epoch. In the retraining phase,

it is divided by 10 at the 75-th and 113-th epoch.

The number of training iteration and thresholding ε

are very important parameters in our algorithm and

have a considerable correlation with each other. In the

ﬁrst experiment, the iteration number remains ﬁxed

and the growth rate of ε is adjusted so that the valida-

tion accuracy saturates near the settled iteration num-

ber. While the validation accuracy is evaluated using

the cross-validation, we set the number of training it-

eration to be 100 so that the model is trained enough

until it saturates. ε is increased in log-scale and be-

gins at a very small value (10

−5

) where no data is

added. The growth rate of ε is determined according

to when the validation accuracy saturates. The stop-

ping criterion is that the accuracy of the current it-

eration reaches the average accuracy of the previous

20 steps. If the stopping iteration is much less than

100 times, the ε growth rate should be reduced so that

the data is added more slowly. If the stopping itera-

tion signiﬁcantly exceeds 100 iterations, the ε growth

rate should be increased so that the data is added more

easily. We allow 5 iterations as a deviation from 100

iterations and the growth rate of ε is left unchanged in

this interval. As a result, the ε is gradually increased

in log-scale by 10 times every 33 iterations in CIFAR-

10 and SVHN. In the case of CIFAR-100, the ε is in-

creased by 10 times in log-scale every 27 iterations.

In the second experiment, we leave the ε ﬁxed and

simply train the model until the stopping criteria are

satisﬁed. Other details are the same as those of the

ﬁrst experiment.

Self-Training using Selection Network for Semi-supervised Learning

Table 4: Classiﬁcation error and the number of added unlabeled data of softmax and sigmoid for new SSL problems on

CIFAR-10 with 5 runs.

Error Added data

method softmax sigmoid softmax sigmoid

supervised 22.27 ± 0.47% -

0% 18.27± 0.52% 17.84 ± 0.39% 4,306 2,338

25% 18.35 ± 0.86% 18.38 ± 0.52% 3,350 1,470

50% 18.72 ± 0.36% 19.04 ± 0.63% 2,580 811

75% 20.33 ± 0.82% 20.07 ± 0.98% 1,711 315

100% 20.71 ± 0.19% 20.24 ± 0.15% 864 1

Data Details

We have experimented with CIFAR-10, SVHN, and

CIFAR-100 datasets that consist of 32 × 32 pixel

RGB images. CIFAR-10 and SVHN have 10 classes

and CIFAR-100 has 100 classes. Overall, standard

data normalization and augmentation scheme are

used. For data augmentation, we used random hori-

zontal ﬂipping and random translation by up to 2 pix-

els. In the case of SVHN, random horizontal ﬂipping

is not used. To show that the SST algorithm is com-

parable to the conventional SSL algorithms, we ex-

perimented with the popular setting (Laine and Aila,

2016; Miyato et al., 2017; Tarvainen and Valpola,

2017). The validation set in the cross-validation to ob-

tain the reduction rate of ε is extracted from the train-

ing set by 5000 images. After the ε is obtained, all the

training datasets are used. The following is the stan-

dard labeled/unlabeled split.

CIFAR-10: 4k labeled data (400 images per class),

46k unlabeled data (4,600 images per class), and 10k

test data.

SVHN: 1k labeled data (100 images per class),

72,257 unlabeled data (it is not well balanced), and

26,032 test data.

CIFAR-100:10k labeled data (100 images per class),

40k unlabeled data (400 images per class), and 10k

test data.

Comparison of Softmax with Sigmoid

Eq 3 and Eq 4 are the formula of softmax function

and sigmoid function, respectively. Eq 3 can be repre-

sented in the form shown in Eq 5.

so f tmax

(v) =

∑

(3)

sigmoid(v) =

1 + e

−v

(4)

so f tmax

(v) =

1 + (e

−v

) × (

∑

i−1

j=1

∑

j=i+1

)

(5)

If v

is comparably larger than the other v

, the soft-

max function performs like a sigmoid function. Also,

even if v

with moderately high values, the softmax

output still becomes close to 1 when having relatively

and extremely small values of v

. Eq 6 represents such

case.

so f tmax

(v) ≈

1 + (e

−v

) × 0

= 1 (6)

We experiment and compare softmax outputs of

(·;θ

) against sigmoid outputs of f

sel

(·;θ

) when

sampling unlabeled data. Table 4 shows the classiﬁ-

cation error and the number of added unlabeled data

with sampling based on outputs of the softmax and the

sigmoid. The threshold for sampling in softmax out-

put is set same to the sigmoid threshold. Although the

thresholds are very high enough, in softmax case, an

average of 864 unlabeled datas are added for the case

of 100% of the non-animal data. Furthermore, with

0% of the non-animal data, error rate of using soft-

max is larger than that of using sigmoid even when

the added data is larger. To the best of our knowledge,

the addition of data with high softmax outputs does

not affect the loss much, which leads to the small per-

formance improvement. This shows the limitation of

thresholding with softmax.

ICPRAM 2020 - 9th International Conference on Pattern Recognition Applications and Methods