The Data Deconﬂation Problem: Moving from Classical to Emerging

Solutions

Roger A. Hallman

1,2

and George Cybenko

Thayer School of Eningeering, Dartmouth College, Hanover, New Hampshire, U.S.A.

Naval Information Warfare Center (NIWC) Paciﬁc, San Diego, California, U.S.A.

Keywords:

Data Deconﬂation, Deconvolution, Blind Source Separation, Cocktail Party Problem, Simple Data, Complex

Data, Deep Learning, Deep Reinforcement Learning (DRL), Generative Adversarial Networks (GANs).

Abstract:

Data conﬂation refers to the superposition data produced by diverse processes resulting in complex, combined

data objects. We deﬁne the data deconﬂation problem as the challenge of identifying and separating these

complex data objects into their individual, constituent objects. Solutions to classical deconﬂation problems

(e.g., the Cocktail Party Problem) use established linear algebra techniques, but it is not clear that those

solutions are extendable to broader classes of conﬂated data objects. This paper surveys both classical and

emerging data deconﬂation problems, as well as presenting an approach towards a general solution utilizing

deep reinforcement learning and generative adversarial networks.

1 INTRODUCTION

The proliferation of Internet-connected devices has

led to a ﬂood of complex, conﬂated data objects from

which we can glean a wealth of useful information.

For example, distributed sensor networks–critical

to large-scale Internet of Things (IoT) systems–

continually report real-time data that may be repre-

sentative of co-located individuals (Wan et al., 2016).

Similarly, data reported by medical wearables may be

contaminated by patient movements or external in-

ﬂuences, or report excess noise due to insufﬁciently

tuned sensors (Tariq et al., 2018). Those conﬂated

data objects must ﬁrst be separated into their con-

stituent components before any meaningful analysis

can be conducted.

Recent advances in deep learning have led to

breakthroughs in many classiﬁcation, recognition,

and decision-making tasks; however those results

have been limited to tame datasets and performance

in relatively benign environments. As a purely mo-

tivational example, consider the conﬂated illustration

in Figure 1. While even a human child can identify at

least one of the constituent objects (seen individually

in Figure 2) in this conﬂated image, a MATLAB im-

plementation of the well-known Alexnet Object Clas-

siﬁer (Krizhevsky et al., 2012; MathWorks, 2020) is

unable to identify any object and returns the following

probabilities:

Figure 1: Multiple images have been conﬂated in such a

way that state-of-the-art classiﬁers cannot identify a single

constituent image.

The current known solutions to data deconﬂation

problems rely on well-established linear algebra tech-

niques, but it is not at all clear that these techniques

can be generally extended. For instance, behavioral

tracking tasks will often generate non-additive su-

perpositions and categorical data that is neither real-

valued nor sampled from a uniform spatial or tempo-

ral grid. As illustrated by Alexnet’s inability to clas-

sify any of the constituent images in Figure 1, even

current deep learning networks are unlikely to provide

Hallman, R. and Cybenko, G.

The Data Deconﬂation Problem: Moving from Classical to Emerging Solutions.

DOI: 10.5220/0010530403750380

In Proceedings of the 6th International Conference on Internet of Things, Big Data and Security (IoTBDS 2021), pages 375-380

ISBN: 978-989-758-504-3

375

Table 1: Alexnet probabilities for Figure 1.

Category Probability

jigsaw puzzle 0.2270

wreck 0.1252

mud turtle 0.1249

loggerhead 0.0842

terrapin 0.0566

(a) Barn (b) Otter

Figure 2: The constituent images that were conﬂated in Fig-

ure 1. Alexnet classiﬁes the subject of these individual im-

ages with high probability.

a satisfactory, more generalized solution to the data

deconﬂation problem. (Tangentially, consider Good-

fellow et al.’s (Goodfellow et al., 2014b), demonstra-

tion that even the addition of seemingly imperceptible

noise can lead to misclassiﬁcations.)

To that end, we present our vision for a solution

to the data deconﬂation problem which can be be ex-

tended to tasks which are beyond currently-known so-

lutions. We believe that a promising approach to the

general deconﬂation problem can be based on itera-

tions between estimating what signal component or

element is contributed by what process (accomplished

by using a trained deep reinforcement network) and

ﬁltering done by using a generative network seeded

by small signal samples.

Contribution and Organization. Our primary

contribution in this paper is the proposal of what we

believe to be a general solution to the data decon-

ﬂation problem, iteratively using deep reinforcement

learning and generative adversarial networks, not only

during training but also in the deconﬂation and clas-

siﬁcation phase. As far we are aware, there is no gen-

eral solution for deconﬂation problems dealing with

non-additive superpositions or categorical valued data

objects, as often occur in spatial tracking and behav-

ioral deconﬂation problems.

The remainder of this paper is organized as fol-

lows: many deconﬂation problems, including the

cocktail party problem, and their solutions are de-

scribed in Section 2. Our approach to a general solu-

tion to the data deconﬂation problem is given in Sec-

tion 3, and concluding remarks are given in Section

2 BACKGROUND AND RELATED

WORK

We begin by presenting a brief survey of current so-

lutions to deconﬂation problems, as well as reinforce-

ment learning and generative adversarial networks.

2.1 Blind Source Separation

Blind source separation (BSS) is the process of sep-

arating unknown signals that have been mixed in an

unknown way (Koﬁdis, 2016). Speciﬁcally, a mixture

u(n) = F (a(n), v(n), n)

mixes N source signals

a(n) = [a

(n), a

(n), ..., a

(n)]

and K noise signals

v(n) = [v

(n), v

(n), ..., v

(n)]

by a mixing system F (·, ·, ·), which yields

u(n) = [u

(n), u

(n), ..., u

N×K

(n)]

BSS problems have long been an active research topic

in both analog and digital signal processing, with nu-

merous demonstrated solutions (O’grady et al., 2005;

Comon and Jutten, 2010). BSS problems are vector

representative and additive, which means that there

are a number of solutions that utilize established lin-

ear algebra techniques. Techniques utilized in classi-

cal BSS solutions include singular value decomposi-

tions, principal component analysis, sparsity enforce-

ment, or other dimensionality reduction methods. For

instance, the Joint Approximation Diagonalization of

Eigen-matrices algorithm has been implemented to

accomplish BSS for both image (Hughes, 2015a) and

audio (Hughes, 2015b) samples.

AI4EIoTs 2021 - Special Session on Artiﬁcial Intelligence for Emerging IoT Systems: Open Challenges and Novel Perspectives

376

2.1.1 The Cocktail Party Problem

Perhaps the most well-known BSS problem is the

Cocktail Party Problem (CPP) (Cherry, 1953), that is

the human ability to selectively focus attention on a

single voice in a noisy environment. In a typical for-

mulation, an attendee at a cocktail party hears their

name spoken by an unknown person outside of their

vision and they attempt to identify that person. The

CPP has been extended to visual data as well as audi-

tory. Shapiro et al., showed that people have an ability

to recognize their own name in otherwise unattended

information (Shapiro et al., 1997).

Haykin and Chen (Haykin and Chen, 2005) frame

the problem in terms of understanding how the human

brain solves this problem and determining whether it

is possible to build a machine that can satisfactorily

solve it. Their survey of computational approaches

detail solutions via (i) independent component analy-

sis (ICA) and general BSS approaches, (ii) temporal

binding and oscillatory correlation, and (iii) cortronic

networks. They note that while ICA and BSS solu-

tions enjoy decades of support in literature, the ap-

proach is not analogous to actual biological solutions.

On the other hand, approaches (ii) and (iii) are in-

spired by biological processes but rely on the assump-

tion of some prior knowledge (e.g., the language be-

ing spoken).

Qian et al. (Qian et al., 2018) survey more recent

approaches to the CPP (including deep learning-based

solutions). They highlight many impressive results,

while pointing out limitations that are analogous to

the current solutions’ shortcomings mentioned in Sec-

tion 1. For instance, they highlight greater improve-

ments in recognition for mixed-gender speech than

for same-gender speech; inferring that same-gender

speech tracing is a more difﬁcult task.

2.2 Process Query Systems

Process Query Systems (PQS) (Cybenko and Berk,

2007) are a more recent solution to deconﬂation prob-

lems that are especially well suited to networked sys-

tems, where extracting meaningful information is par-

ticularly challenging. By paying attention to process

descriptions, PQS are able to solve complex informa-

tion retrieval tasks within the network. Speciﬁcally,

PQS take input from arbitrary nodes in a network and

build hypotheses about observed events that answer a

user’s process queries. Multiple hypotheses and mod-

els are used to separate observed events, optimally

matching them with ongoing processes, and identify-

ing process states.

PQS have been applied to tasks in network admin-

istration, including security monitoring (Berk et al.,

2003; Berk and Fox, 2005), covert channel detection

(Giani et al., 2005), and autonomic server monitoring

(Roblee et al., 2005). Additionally, PQS have been

used for vehicle tracking using acoustic sensor net-

works (Berk et al., 2003).

While PQS provide a more general solution to

tasks that are beyond the capabilities of BSS and clas-

sical deconﬂation solutions, they are not a general so-

lution. A PQS requires a priori models for underly-

ing processes, as well as heuristics for estimating the

number of processes, when those processes begin and

end, and track assignments.

2.3 Reinforcement Learning

Reinforcement Learning (RL) is a ﬁeld of machine

learning that seeks to understand, automate, and op-

timize goal-directed decision making (Sutton and

Barto, 2018). Deep Reinforcement Learning (DRL)

(Franc¸ois-Lavet et al., 2018) involves harnessing the

power of deep neural networks for RL tasks and has

led to groundbreaking results, including super-human

results in gameplay.

In spite of the successes in relatively tame and op-

timized environments, RL and DRL face a multitude

of challenges in adoption for real-world tasks (Dulac-

Arnold et al., 2019). One such challenge which has

recently seen breakthrough results is the credit assign-

ment problem where there are delays between agent

actions and rewards (Hung et al., 2019). Speciﬁcally,

Hung et al. developed an agent memory function that

credits past actions and enables them to solve pre-

viously intractable problems. Deep Reinforcement

Relevance Networks (He et al., 2016) and Dialog

State Tracking and Management (Zhao and Eskenazi,

2016) have shown phenomenal success in state track-

ing and credit assignment in natural language.

2.4 Generative Adversarial Networks

Generative Adversarial Networks (GANs) (Goodfel-

low et al., 2014a) are a deep learning framework

where two deep neural networks, a generator and a

discriminator, are simultaneously trained against each

other. Speciﬁcally, the discriminator is trained to de-

tect real from synthetic data (e.g., differentiating an

authentic image versus a synthetic image of a human

face (Tariq et al., 2018)) while the generator is trained

to generate authentic “looking” synthetic data from a

low-dimension seed.

In order to take a low-dimensional data seed and

generate synthetic data capable of fooling the discrim-

The Data Deconﬂation Problem: Moving from Classical to Emerging Solutions

377

inator, GANs must effectively impute missing data.

Lee et al. (Lee et al., 2019), developed a GAN which

converts image imputation into a multi-domain trans-

lation task, enabling a single generator and discrim-

inator to successfully estimate missing image data.

Following on successes in image data imputation,

GANs are being utilized for time series data impu-

tation. Time series data from many sensor networks

have an average missing data rate of around 80% and

the imputation of that missing data is critical to any

analysis efforts. Luo et al. (Luo et al., 2018) im-

plemented a gated recurrent unit (GRU), modiﬁed to

model temporal irregularity, into their GAN architec-

ture. Furthermore, they developed a loss function that

provides a ﬁtness measure for imputed values. Zhang

et al. (Zhang et al., 2021) incorporate real data forcing

and an encoder network into their GAN architecture

to create imputed synthetic data that performs well in

numerous downstream tasks.

3 OUR APPROACH TO A

GENERAL DATA

DECONFLATION SOLUTION

We have now deﬁned BSS and surveyed existing so-

lutions, thus we ﬁrst propose a generalization of the

BSS problem before we present our vision for a gen-

eral solution.

3.1 From BSS to General Data

Deconﬂation

Data can be conﬂated in space (e.g., Figure 1), time,

and semantics as well as in any combinations of these

dimensions. The most common manifestation of the

multi-target tracking problem can be both spatial (as

arises in occlusion) and temporal (as in track assign-

ment). Pattern of life analyses have to deal with con-

ﬂated semantics in which, for example, a commuter

combines a trip to work with an in-person meeting on

the commuter train.

Simple data is data (or a process) coming from a

single source. Complex or conﬂated data consists of

interwoven simple data objects coming from multi-

ple sources. Solutions to BSS of complex data re-

quire vector respresentable inputs, but it is not ap-

parent that this is broadly possible for general sepa-

ration tasks. Rather than vector representations, we

therefore propose to represent simple data as a state

machine (Schneider, 1990) and complex data as state

machine synthesis (Ginsburg, 1959).

Our state machine representation for conﬂated

data is presented in Figure 3. We claim that an ob-

served event sequence (i.e., complex data) is the syn-

thesis of an unknown multiplicity of simple data ob-

jects. The Data Deconﬂation Problem is a general-

ization of the BSS Problem (Section 2.1): given an

observed event sequence, which simple data objects

are responsible for speciﬁc observed events? Further-

more, many separation solutions assume some a pri-

ori knowledge–whether a language spoken, some un-

derlying processes, beginning and ending parameters,

etc.–so we would like to be able to deconﬂate com-

plex data without any assumed background knowl-

edge.

Figure 3: A state machine representation of conﬂated data

objects or processes.

3.2 A General Solution to the Data

Deconﬂation Problem

The approach that we describe below proposes to

solve hard deconﬂation problems by the extension

and application of DRL and GANs. We believe that

a general solution to the deconvolution problem can

be achieved by iterating between estimates of which

signal component or element is contributed by which

process (accomplished by DRL) and ﬁltering done

by using generative networks seeded by small signal

samples.

To illustrate this iterative process, refer back to

Figure 1. We might be estimating a classiﬁcation

based on a small sample portion of the image and

then completing the small portion for that class using

a generative model (e.g., sampling a small part of the

school bus and using that sample to generate a more

complete school bus image). We might then alter-

nately ﬁlter in and out the generated constituent image

to either isolate it and conﬁrm identiﬁcation or elimi-

nate it to allow focusing on other objects. Though we

are speculating about how a human might solve this

particular problem, it is a reasonable starting point for

investigating this difﬁcult problem.

Our approach to the deconﬂation problem takes

place over two phases. In the ﬁrst phase we use GANs

to model potential simple data objects based on ob-

served complex data. Once simple data models have

been generated, we will use DRL to approximate la-

beled complex data training sets by processes of inter-

AI4EIoTs 2021 - Special Session on Artiﬁcial Intelligence for Emerging IoT Systems: Open Challenges and Novel Perspectives

378

Figure 4: Our training process for deconﬂating complex data.

leaving and repetition of models. Once we have cre-

ated an approximation of observed complex data, we

use deep neural networks that have been trained to de-

convolve complex data. Both phases of this approach

are illustrated in Figure 4. This process is analogous

to learning to play a game–the observed complex data

sequence is the game state and label assignments are

the player moves.

4 CONCLUSION

The adoption and emerging ubiquity of Internet-

connected devices is leading us to a digital environ-

ment that is full of complex data streams that must

be correctly deconﬂated in order to conduct mean-

ingful analysis. While much of this data can be ad-

equately separated through traditional BSS solutions,

a non-trivial amount of this complex data is not vector

representable and thus requires new deconﬂation so-

lutions. In this paper we have described complex data

objects that cannot be deconﬂated by current BSS so-

lutions, and for which we have proposed a more gen-

eral data deconﬂation problem. Furthermore, we have

presented our vision for a general solution to the data

deconﬂation problem that extends recent advances in

DRL and GANs.

We are currently working on an initial proof-of-

concept implementation. Other ongoing work on this

effort includes a rigorous generalization of the data

conﬂation process from vector representations to state

machine representation (Section 3.1). We are also

designing experiments to determine the appropriate

structures for recurrent and/or convolutional neural

networks to learn minimal simple data object mod-

els. Once we have demonstrated results with estab-

lished with complex spatio-temporal data, we will ex-

tend our approach to non-spatio-temporal data, such

as semantic conﬂations that might appear in pattern

of life tracking.

ACKNOWLEDGEMENTS

Roger A. Hallman is partially supported by the United

States Department of Defense SMART Scholarship

for Service Program, funded by USD/R&E (The Un-

der Secretary of Defense-Research and Engineering),

National Defense Education Program (NDEP) / BA-

1, Basic Research.

REFERENCES

Berk, V., Chung, W., Crespi, V., Cybenko, G., Gray, R.,

Hernando, D., Jiang, G., Li, H., and Sheng, Y. (2003).

Process query systems for surveillance and awareness.

In In Proc. System. Cyber. Infor.(SCI2003. Citeseer.

The Data Deconﬂation Problem: Moving from Classical to Emerging Solutions

379

Berk, V. and Fox, N. (2005). Process query systems for

network security monitoring. In Sensors, and Com-

mand, Control, Communications, and Intelligence

(C3I) Technologies for Homeland Security and Home-

land Defense IV, volume 5778, pages 520–530. Inter-

national Society for Optics and Photonics.

Cherry, E. C. (1953). Some experiments on the recognition

of speech, with one and with two ears. The Journal of

the acoustical society of America, 25(5):975–979.

Comon, P. and Jutten, C. (2010). Handbook of Blind Source

Separation: Independent component analysis and ap-

plications. Academic press.

Cybenko, G. and Berk, V. H. (2007). Process query sys-

tems. Computer, 40(1):62–70.

Dulac-Arnold, G., Mankowitz, D., and Hester, T. (2019).

Challenges of real-world reinforcement learning.

arXiv preprint arXiv:1904.12901.

Franc¸ois-Lavet, V., Henderson, P., Islam, R., Bellemare,

M. G., and Pineau, J. (2018). An introduction to deep

reinforcement learning.

Giani, A., Berk, V., Cybenko, G., and Hanover, N. (2005).

Covert channel detection using process query systems.

In proceedings of: FLoCon.

Ginsburg, S. (1959). Synthesis of minimal-state machines.

IRE Transactions on Electronic Computers, (4):441–

449.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,

Warde-Farley, D., Ozair, S., Courville, A., and Ben-

gio, Y. (2014a). Generative adversarial nets. In

Ghahramani, Z., Welling, M., Cortes, C., Lawrence,

N., and Weinberger, K. Q., editors, Advances in Neu-

ral Information Processing Systems, volume 27, pages

2672–2680. Curran Associates, Inc.

Goodfellow, I. J., Shlens, J., and Szegedy, C. (2014b). Ex-

plaining and harnessing adversarial examples. arXiv

preprint arXiv:1412.6572.

Haykin, S. and Chen, Z. (2005). The cocktail party prob-

lem. Neural computation, 17(9):1875–1902.

He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., and Os-

tendorf, M. (2016). Deep reinforcement learning with

a natural language action space. In Proceedings of the

54th Annual Meeting of the Association for Compu-

tational Linguistics (Volume 1: Long Papers), pages

1621–1630.

Hughes, K. (2015a). Blind source separa-

tion on images with shogun. (Accessed

via Internet Web Archive) http://shogun-

toolbox.org/static/notebook/current/bss image.html.

Hughes, K. (2015b). Blind source separation

with the shogun machine learning toolbox.

https://nbviewer.jupyter.org/github/kevinhughes27/bs-

s jade/blob/master/bss jade.ipynb.

Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza,

M., Carnevale, F., Ahuja, A., and Wayne, G. (2019).

Optimizing agent behavior over long time scales by

transporting value. Nature communications, 10(1):1–

12.

Koﬁdis, E. (2016). Blind source separation: Fundamentals

and recent advances (a tutorial overview presented at

sbrt-2001). arXiv preprint arXiv:1603.03089.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Im-

agenet classiﬁcation with deep convolutional neural

networks. Advances in neural information processing

systems, 25:1097–1105.

Lee, D., Kim, J., Moon, W.-J., and Ye, J. C. (2019). Colla-

gan: Collaborative gan for missing image data impu-

tation. In Proceedings of the IEEE/CVF Conference

on Computer Vision and Pattern Recognition, pages

2487–2496.

Luo, Y., Cai, X., Zhang, Y., Xu, J., and Yuan, X. (2018).

Multivariate time series imputation with generative

adversarial networks. In Proceedings of the 32nd In-

ternational Conference on Neural Information Pro-

cessing Systems, pages 1603–1614.

MathWorks (2020). Alexnet convolutional neural network.

https://www.mathworks.com/help/deeplearning/ref/a-

lexnet.html.

O’grady, P. D., Pearlmutter, B. A., and Rickard, S. T.

(2005). Survey of sparse and non-sparse methods in

source separation. International Journal of Imaging

Systems and Technology, 15(1):18–33.

Qian, Y.-m., Weng, C., Chang, X.-k., Wang, S., and Yu,

D. (2018). Past review, current progress, and chal-

lenges ahead on the cocktail party problem. Frontiers

of Information Technology & Electronic Engineering,

19(1):40–63.

Roblee, C., Berk, V., and Cybenko, G. (2005). Implement-

ing large-scale autonomic server monitoring using

process query systems. In Second International Con-

ference on Autonomic Computing (ICAC’05), pages

123–133. IEEE.

Schneider, F. B. (1990). The state machine approach: A

tutorial. Fault-tolerant distributed computing, pages

18–41.

Shapiro, K. L., Caldwell, J., and Sorensen, R. E. (1997).

Personal names and the attentional blink: A vi-

sual” cocktail party” effect. Journal of Experimen-

tal Psychology: Human Perception and Performance,

23(2):504.

Sutton, R. S. and Barto, A. G. (2018). Reinforcement learn-

ing: An introduction. MIT press.

Tariq, S., Lee, S., Kim, H., Shin, Y., and Woo, S. S. (2018).

Detecting both machine and human created fake face

images in the wild. In Proceedings of the 2nd interna-

tional workshop on multimedia privacy and security,

pages 81–87.

Wan, P., Hao, B., Li, Z., Zhou, L., and Zhang, M. (2016).

Time differences of arrival estimation of mixed inter-

ference signals using blind source separation based

on wireless sensor networks. IET Signal Processing,

10(8):924–929.

Zhang, Y., Zhou, B., Cai, X., Guo, W., Ding, X., and Yuan,

X. (2021). Missing value imputation in multivariate

time series with end-to-end generative adversarial net-

works. Information Sciences, 551:67–82.

Zhao, T. and Eskenazi, M. (2016). Towards end-to-end

learning for dialog state tracking and management us-

ing deep reinforcement learning. In Proceedings of

the 17th Annual Meeting of the Special Interest Group

on Discourse and Dialogue, pages 1–10.

AI4EIoTs 2021 - Special Session on Artiﬁcial Intelligence for Emerging IoT Systems: Open Challenges and Novel Perspectives

380