Towards an Ensemble Approach for Sensor Data Sensemaking

Athanasios Tsitsipas

Institute of Information Resource Management, Ulm University, Ulm, Germany

Keywords:

Sensor Data Sensemaking, Hybrid, Stream Reasoning, Abduction, Shapeoids.

Abstract:

In a world of uncertainty and incompleteness, one must “make sense” of found observations. Cyber-physical

systems output large quantities of data, opening massive opportunities and challenges for scalable techniques

to gain exciting insights. One intriguing challenge is the process of Sensor Data Sensemaking. The research

presents an approach to handle this process by bringing together the strands of data and knowledge in a single

architecture in an interpretable and expressive way. Differently from other works, the use of interpretable

patterns from streaming data is in the spotlight. In addition, background knowledge over these patterns gasps

the intention to give meaning to these patterns with several possible explanations. A hybrid implementation

realises the approach following big data processing models.

1 INTRODUCTION

Imagine an infrastructure with various sensors de-

ployed, witnessing a ﬂux in the data from a speciﬁc

sensor. There could be multiple applications and use

cases treating this “phenomenon” as an anomaly or

just the typical operation of this sensor. The impor-

tance of the ﬂux is interchangeable in any of the two

cases. A malfunction, or a complete failure of this

sensor, can lead to internal issues or shortcomings for

the system in place. Indirect Sensing (IS) is a com-

pelling approach to treating such cases. Indirect Sens-

ing (IS) is where single or different sensor composi-

tions deliver the same information when the property

in need is no longer attainable to direct sense. There-

fore, other alternative sources may continuously pro-

vide streaming data that might contain the missing in-

formation implicitly. However, making sense (Cook,

2007) of sensor data to tackle this uncertainty and

incompleteness in the environment is necessary for

gaining exciting insights.

One approach that arises naturally for such use

cases is classiﬁcation as part of pattern recognition

assigning observations to various classes. For ex-

ample, in the context of IS, the authors in (Laput

et al., 2017) use the streaming data as an input to

a supervised learning algorithm and thus create a

model that approximates the real-world’s “image” to

an understandable format for a machine. To enhance

the robustness of such a model requires a substan-

https://orcid.org/0000-0001-9122-6408

tial amount of training data transforming fundamen-

tal values that need to be categorised into features

to calculate distances between them later. Such ef-

forts fall under catch-phrases like Machine Learning

or Deep Learning. However, both follow the same in-

tuition to explore the input data for prediction, retro-

diction, and imputation tasks. While this approach

is prodigious for making comparisons between such

items (i. e., data) and clustering them accordingly,

they share to a degree the natural incentive of “mak-

ing sense”. However, recent studies have shown that

making sense of sensor data should be more than just

a classiﬁcation task (Teijeiro and Félix, 2018; Evans

et al., 2021).

Sensor Data Sensemaking (SDS) is a process in

Human-Computer Interaction (HCI). Participants in

ﬁeld studies interpret and understand their environ-

ments and the behaviour of complex systems by

“reading off” historical data using situated knowl-

edge (Kurze et al., 2020). The human brain can distin-

guish or observe similarities in features from data as

patterns (Neisser, 2014); use them as evidence which

ought to explain by guessing the underlying process

that caused such observations. Hence, the main re-

search question is how to handle IS under a machine-

based SDS process in a human-centred manner. The

research assumes that making sense of sensor data in

IS requires a theory under explicit symbol manipula-

tion, most commonly referred to and understood as

methods based on formal logic.

Additionally, it is similar to current trends in

Tsitsipas, A.

Towards an Ensemble Approach for Sensor Data Sensemaking.

DOI: 10.5220/0010898800003122

In Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods (ICPRAM 2022), pages 323-329

ISBN: 978-989-758-549-4; ISSN: 2184-4313

323

Figure 1: The ensemble approach bridges data-driven

human-centred representations from streaming data and

symbolic abductive reasoning.

modern Artiﬁcial Intelligence (AI), under its subﬁeld

Neuro-Symbolic AI, which focuses on merging the

neural and symbolic regimes in AI research (Saker

et al., 2021). The process of SDS is following mod-

ern applications of data work (Fischer et al., 2016;

Fischer et al., 2017), which consists of three essen-

tial aspects laying down the proposed approach: (i) as

part of human cognition requires a representation of

objects, actions, numbers and space (Spelke and Kin-

zler, 2007), which persist and evolve, (ii) a key com-

ponent in human commonsense is the construction of

an explanation for the observed phenomenon (Marcus

and Davis, 2019; Lake et al., 2017), (iii) ﬁnally, the

research postulates that making sense of sensor data

series is possible for someone to fathom it by spec-

ulating the presence of an unobserved cause, which

accounts for and characterises the view of the sen-

sor data (McCarthy, 2006; Inoue et al., 2009; Teijeiro

and Félix, 2018). The occurred phenomenon con-

sists of unobserved entities (i. e., environmental prop-

erties) that causally interact with the mechanics of

the sensors to produce a dimensional footprint trans-

lated to the real-valued sensor reaction received as in-

put. Thus, the approach to tackle the problem of IS

using low-level processing to extract representations

and high-level processes as innate abductive reason-

ing (Schurz, 2016) becomes ubiquitous to the under-

lying research areas in Structural Pattern Recognition,

Logic and Stream Reasoning, following the ﬁeld of

Knowledge Representation and Reasoning (KRR).

The novelty and advance of the approach weights

in an amalgamation of the following contributions.

The ﬁrst contribution is to develop and extract human-

centred representations for mapping the time series

sensor data to symbolic lexical constructs that con-

stitute the primary objects for the following step.

The second contribution is a declarative reasoning

method via abduction, employing commonsense and

domain knowledge to exploit the above lexical con-

structs from the sensor data as explanations for their

occurrence. The approach uses probabilistic graph-

ical models, which seamlessly bridge the worlds of

probability and logic-based programming. Finally, ar-

chitecture and implementation are imperative for an

integrated stream reasoning framework for the SDS

process facilitating the IS approach, forming a sensor

fusion and a reasoning pipeline. Figure 1 illustrates

a high-level view of the hybrid approach for the SDS

process.

Section 2 presents a brief overview of the related

work. Next, an enumeration of the focused research

questions and the approach addressing them reside in

Section 3 and Section 4, respectively. Finally, a brief

overview of the main contributions in Section 5 and a

short conclusion in Section 6 ﬁnalise the paper.

2 RELATED WORK

Sensemaking is an active area in HCI and has seen

many studies on how people perceive and understand

complex organisations of information (Fischer et al.,

2016; Kurze et al., 2020). It actively involves the

cognitive state of mind of Situational Awareness to

analyse and assess dynamic problem environments.

Automated methods for sensemaking support have

been surveyed in (Llinas, 2014). The survey in-

cludes conceptual sensemaking models and analytic

tool suites (ATS), primarily visualisation-based oper-

ations. The current research employs various tech-

niques and methods to emulate the sensemaking pro-

cess in HCI as part of an integrated framework. Ob-

serving structural primitives in time series data from a

graph lends from Structural Pattern Recognition (Ol-

szewski, 2001) and examing the hypothesis synthe-

sis as an abductive reasoning process (Glass, 2019)

formulates the ﬁnal approach. In addition, the re-

search includes a realisation of the sensemaking pro-

cess over streaming sensor data. Therefore, each part

contains individual challenges tackled in the respec-

tive research area. The reader has some pointers to

the author’s previous work (Tsitsipas and Schubert,

2021a; Tsitsipas et al., 2021).

Stream reasoning (Della Valle et al., 2009) is

a relatively new ﬁeld from almost the last decade.

It aims to bridge the gap between stream process-

ing and reasoning. Different communities cover as-

pects of it when the scale moves solely on each side.

Data Stream Management Systems (DSMSs) (Go-

lab and Özsu, 2003) cover the element of man-

aging low-level, high-throughput data using con-

tinuous queries, but hardly ever they consider in-

complete information. Complex Event Processing

(CEP) (Cugola and Margara, 2012) approaches ex-

ICPRAM 2022 - 11th International Conference on Pattern Recognition Applications and Methods

324

press high-level events in compositions of single in-

coming events. They use probabilistic methods (Tran

and Davis, 2008; Wasserkrug et al., 2010; Skarlatidis

et al., 2015) to handle uncertainty in incoming data.

The research follows these systems, especially where

Markov Logic Networks has a central role. However,

these approaches usually operate over a static dataset

and do not realise a dynamic scenario for inference

over dynamic data. The issue is mainly because a

Markov Logic Network requires all the available ev-

idence before the inference. The research work ap-

plies a partial solution using the method of soft evi-

dence (Jain and Beetz, 2010). Recently, a survey ex-

amined how Complex Event Recognition could jump

to the Big Data era (Giatrakos et al., 2020). Although,

in line with the work, their interests are around event

forecasting and inductive logic programming (ILP).

Furthermore, in KRR, novel methods and ap-

proaches for representation and reasoning over

streaming input exist but lack the scalability over

the velocity of data. KRR yields the research be-

hind non-monotonic logic and abductive reasoning as

guiding methods for commonsense reasoning (Davis,

2017). Many authors are working on visual under-

standing, utilising commonsense knowledge for ab-

ductive high-level explanations (Suchan et al., 2019;

Le-Phuoc et al., 2021). However, although they use

neural models for ﬁnding the required representa-

tions, they do not use domain-independent semantics

on the model-based processing or the data process-

ing steps. Therefore, generality is not guaranteed by

their approaches. Moreover, the authors in (Kate and

Mooney, 2009) examine the application of abduction

in Markov Logic Networks, with a recent application

to root cause analysis in IT infrastructures (Schoen-

ﬁsch et al., 2018). However, the application does not

foresee a dynamic scenario; instead, it includes stati-

cally generated evidence for the failure events of the

IT components.

Finally, an architecture for bridging the sense-

reasoning gap for stream reasoning developed in the

context of unmanned aerial vehicles (UAVs), named

DyKnow(Heintz et al., 2010) and extended in a robot

operating system for adaptively applied stream pro-

cessing (De Leng and Heintz, 2016). The main sepa-

ration factor between this pioneering work in the area

and the current research work is the absence of gen-

erality expressing the SDS process. DyKnow oper-

ates on raw data using threshold-based static queries

to extract the low-level abstractions for high-level

cognitive functions for agent systems. There is a

lack of human-centred data representations and the

general concept of sensemaking (including reasoning

processes, e. g., abduction).

Figure 2: A diagram of the signiﬁcant dataﬂows in the hy-

brid architecture realising the SDS process.

3 RESEARCH QUESTIONS

The work to alleviate the challenges formulated in the

introduction poses the following research questions to

separate the overall approach logically:

RQ1: Which symbolic descriptions (human-centred)

from low-level sensor data the SDS process re-

quires, and how to extract them?

RQ2: How to employ commonsense and domain

knowledge to exploit the sensor data dimensional

footprint of an unobserved cause as an explana-

tion for its occurrence?

RQ3: How to scale the SDS process in an advanced

stream reasoning framework?

4 RESEARCH APPROACH

The approach is driven by developing an expressive

SDS artefact within the realm of stream reasoning.

As such, it explores the interplay between a data-

driven human-centred primitive pattern extraction and

a knowledge-driven inference method. They support

both a quantitative view on symbolic representations

from streaming data and a qualitative view on com-

plex interrelations of properties and features of an IS

task. A principal approach conciliates the gap be-

tween sensing and reasoning in a streaming setting in

research work. Given the magnitude and complexity

of the problem, addressing each issue and challenge

in the respective ﬁeld is essential to ﬁnally move to-

wards the hybrid architecture of the approach.

Lexical Sensor Observation (LSO). An investiga-

tion of techniques and methods from time series rep-

resentation and more concretely focus on symbolic

representations accommodates the approach. The

study of subsymbolic and neural processes are not

foreseeable in work. Whereas they contain power-

Towards an Ensemble Approach for Sensor Data Sensemaking

325

ful tools for exploring large amounts of data, the re-

sulting feature space is in a numerical representation

that needs further interpretation to translate to human-

centred encodings. The emphasis in work is on how

a human can observe and describe a time series se-

quence by locking to some points of interest to per-

ceive a general shape or pattern (Agrawal et al., 1995;

Bakshi and Stephanopoulos, 1995). The exciting area

of shape-based extraction (Siddiqui et al., 2020) acts

as an inspiration to describe a time series of sensor

perturbations in natural language constructs. The ex-

tracted low-level representations from the sensor data

series act as interpretable LSOs and can compose

more complex shapes expressing their relationships

via sequential operators.

Declarative Hypothesis Model (DHM). Formal

logic is the only epistemically sufﬁcient representa-

tion (Davis, 2017) to encode commonsense and do-

main knowledge in a declarative and expressive man-

ner. The process of sensemaking is naturally an

abductive reasoning process (non-monotonic). Ini-

tially, there is inherent uncertainty in the sensor data,

the pattern extraction, and the different conjectures

(i. e., DHM) created during the SDS process. A

prompt hypothesis should be retracted when more

data is available or a non-perfect logical rule as a hy-

pothesis. In addition, a formalism to guide the process

of reasoning is required, as the semantics and the de-

notations of logic needed for the hypothesis encoding

should be domain-independent.

From Hypotheses to Explanations (HE). The re-

search approach uses as commonsense knowledge the

various encoded hypotheses qua alternatives for an IS

task. They are expressed adequately in the form of

logical language. The evaluation of the hypotheses

rules is implemented as an approximately valid infer-

ence concerning the logic. The inference techniques

in the respective logic are necessary to evaluate the

correctness of the SDS process.

A Hybrid Architecture. As a ﬁnal step, the re-

search work provides a realisation of the approach

to streaming sensor data. A novel and uniﬁed data

processing programming model implements the SDS

process capable of high-level abduction over stream-

ing sensor data. A qualitative evaluation of the over-

all research outcome against a set of requirements for

principal approaches attempting to bridge the gap be-

tween sensing and reasoning (Heintz, 2009) is pre-

sumptive for its resilience.

5 CONTRIBUTIONS

The following contributions realise the approach and

thus provide evidence to counter the constituted ques-

tions. Figure 2 acts as an accompanying illustration,

with more elaborate descriptions of the contribution’s

internal steps.

5.1 Human-centred Representation for

Data-driven Patterns

The process of SDS requires low-level symbolic rep-

resentations of sensor data as patterns that are human-

understandable (e. g., in natural language). The ap-

proach offers the time series representation in a

“morphable” manner by proposing lexical shape-like

primitives, named shapeoids (Tsitsipas and Schubert,

2021b). It utilises the Symbolic Aggregate Approx-

imation (SAX) (Lin et al., 2003) method. On top of

SAX, developing a string-based algorithm for extract-

ing this set of primitive data-driven lexical constructs

constitutes the creation of the shapeoids. A novel

framework named SCOTTY (Tsitsipas et al., 2021) en-

compasses this functionality implemented in Java and

based on an open-source implementation

of SAX.

For the sake of conciseness, the reader may regard

the shapeoids: (i) An ANGLE is a gradual and con-

tinuous increase or decrease, (ii) A HOP describes

a distinctive phase shift leading to an overall visible

change, (iii) The HORN is a temporary effect which

fades quickly, as the end of the pattern returns almost

to the initial point, and (iv) The FLAT is an almost

sturdy line with small variations in the curve. The

shapeoids offer a comprehensive view of the time se-

ries data. SCOTTY can process around a million sin-

gle data points, in not more than six seconds, with a

time and space complexity of O(N).

Additionally, its evaluation has shown interesting

capabilities in time series representation in pattern

recognition. During the evaluation, a declarative clas-

siﬁer was built, based on SCOTTY, competing with

state-of-the-art algorithms in the area. The evalua-

tion uses the Cylinder-Bell-Funnel dataset from the

UCR archive (Dau and Keogh et al., 2018). For the

SDS process, SCOTTY should run over a sliding win-

dow with a parameterised conﬁguration on the win-

dow size to overcome the limitation of the underlying

method of SAX, which performs in an internal step a

statistical computation, requiring bounded data.

https://github.com/jMotif/SAX

ICPRAM 2022 - 11th International Conference on Pattern Recognition Applications and Methods

326

5.2 Expressive High-level Abductive

Reasoning

Rooted in Markov Logic Networks (MLN) (Richard-

son and Domingos, 2006), the framework named

ATOPO (Tsitsipas and Schubert, 2021a) is designed

for integration as a reasoning engine for the hybrid ar-

chitecture of the SDS. It leverages the available com-

monsense knowledge to express the alternative hy-

potheses for the task of IS. It employs axioms from

the Event Calculus (EC) (Kowalski and Sergot, 1989;

Mueller, 2014) to enforce the persistence of objects

whose value change over time by keeping them as

hard-constrained rules in the resulting Markov Net-

work. MLN combines the expressiveness of ﬁrst-

order logic and uses concepts from probability the-

ory to tackle uncertainty. The knowledge engineer-

ing process for the SDS consists of deﬁnite clauses

in ﬁrst-order logic with assigned weights in front of

them indicating their probability degree. The man-

ual assignment of weight is part of the sensemaking

process. It is an aspect of the commonsense reason-

ing stating if they support (positive weight) or pe-

nalise (negative weight) worlds in which the corre-

sponding hypotheses are satisﬁed instead of classical

logic, where all the statements are hard-constrained.

The EC inertia laws must always remain as hard con-

straints, because otherwise during the inference pro-

cess, the hypotheses rules’ probabilities will eventu-

ally converge to be equiprobable (Skarlatidis et al.,

2015).

The implementation of commonsense reasoning is

a valid inference (Davis, 2017) within MLN, calculat-

ing the joint probability distribution of the grounded

Markov Network, using approximation techniques

because the direct computation of the formula is in-

tractable for large networks. ATOPO uses MC-SAT

to output the marginal probabilities. As mentioned

above, the knowledge base contains factual predicates

about the ground truth and the hypotheses rules en-

coded with domain-independent semantics. A hy-

pothesis is a clausal rule with the EC predicate Initi-

atedAt as the head. Its body contains Happens predi-

cates with the recognised shapeoid from a sensor and

other contextual constraints (e. g., sensor type, suit-

able location). The queried ﬂuent in the EC predi-

cate HoldsAt, is a possible quantiﬁcation over veri-

fying an explanation for LSO ﬁndings from the raw

sensor data. Finally, ATOPO uses an open-source

implementation of Markov Logic Networks, named

LoMRF (Skarlatidis and Michelioudakis, 2014) and

implemented in Scala, which is also its development

language. To support the non-monotonic semantics of

EC, LoMRF performs circumscription via predicate

completion (Skarlatidis et al., 2015).

The evaluation of ATOPO in a situational

awareness use case with IS, utilises a real-world

dataset (Birnbach et al., 2019). The authors collected

various sensor data while performing different activ-

ities. In the context of its evaluation ATOPO ran in a

static setting, requiring manual encodings for the in-

ternal representations in the knowledge base and the

evidence for recognising an opened window from the

data using ﬁve in total surrounding sensors (e. g., tem-

perature, air quality). The evaluation showed promis-

ing results even if selecting a rigorous setting to calcu-

late the performance measures, referencing the times-

tamps of the ground truth (i. e., when the window’s

status is open or closed, based on a contact sensor).

While engaging the SDS process, the visible dimen-

sional footprint of the action (e. g., opening the win-

dow) eventuates with a delay as it takes some time un-

til the opened window affects, for example, the tem-

perature sensor sufﬁciently.

5.3 An Integrated Framework for SDS

The ﬁnal contribution is the integrated framework

implemented on top of Apache Beam

. It provides

a uniﬁed programming model, offering common el-

ements of data processing frameworks (e. g., win-

dowing, transform functions), supporting batch and

stream processing and running on many execution

engines. Hence, the solution’s potential is scalable

to clustered data processing platforms (e. g., Apache

Flink, Apache Spark). The architecture enables the

SDS in a setting under streaming use cases. Apache

Beam allows SCOTTY to run over multivariate sensor

data in a streaming setting. Additionally, it retroﬁts

ATOPO in a reasoning pipeline, executing determin-

istic runs of inference on the grounded Markov Net-

works over a sliding window. It supports incremental

reasoning, as the previous states and inference results

propagate as evidence and maximum a posteriori ele-

ments, respectively, to the following sliding window.

Finally, the research work provides evidence for the

compliance of the hybrid implementation by comply-

ing with the requirements of a knowledge processing

middleware (Heintz, 2009).

6 CONCLUSION

The paper presents an “ensemble” approach for SDS.

It extracts human-centred data-driven primitives from

time series data, representing people’s mental mod-

https://beam.apache.org/

Towards an Ensemble Approach for Sensor Data Sensemaking

327

els in an SDS process. A knowledge-driven method

based on theoretical formalism supports the reason-

ing steps for expressing alternative hypotheses that

explain the extracted observations in an IS setting.

The individual parts of the work have been pub-

lished in various conferences on Artiﬁcial Intelli-

gence, Big Data and Pattern Recognition. In addi-

tion, the solution includes a prototype implementation

with a hybrid architecture under modern data process-

ing frameworks. SDS constitutes a signiﬁcant chal-

lenge to solve, as many state-of-the-art research ﬁelds

have the same incentive under different settings: how

to bridge data with knowledge. The outcome of the

work comes to the fore to explain the occurrence of

primitive structures in time series data by assuming

an underlying reality that triggered them.

Furthermore, when submitting this paper to the

conference, the open position of evaluating the prim-

itive structures for the SDS process in other domains,

where a sensor is not physical but software-based vir-

tual (Martin et al., 2021), was submitted and accepted

in parallel to another conference (Tsitsipas et al.,

2022). Finally, the current research work opens the

ﬁeld on the sparsity of other different primitive data

structures for domain-dependent scenarios and how

the SDS process can be realised in a neuro-symbolic

approach while maintaining a human-centred manner.

ACKNOWLEDGEMENTS

Germany’s Federal Ministry of Education and Re-

search (BMBF) has partially funded the research un-

der the project HorME.

REFERENCES

Agrawal, R., Psaila, G., Wimmers, E. L., and Zaït, M.

(1995). Querying shapes of histories. In Proceedings

of the 21th International Conference on Very Large

Data Bases, pages 502–514.

Bakshi, B. R. and Stephanopoulos, G. (1995). Reasoning in

time: Modeling, analysis, and pattern recognition of

temporal process trends. Advances in Chemical Engi-

neering, 22(C):485–548.

Birnbach, S., Eberz, S., and Martinovic, I. (2019). Peeves:

Physical event veriﬁcation in smart homes. In Pro-

ceedings of the 2019 ACM Conference on Computer

and Communications Security. ACM.

Cook, D. J. (2007). Making sense of sensor data. IEEE

Pervasive Computing, 6(2):105–108.

Cugola, G. and Margara, A. (2012). Processing ﬂows of

information: From data stream to complex event pro-

cessing. ACM Computing Surveys (CSUR), 44(3):1–

62.

Dau, H. A. and Keogh et al., E. (2018). The ucr time series

classiﬁcation archive.

Davis, E. (2017). Logical formalizations of commonsense

reasoning: a survey. Journal of Artiﬁcial Intelligence

Research, 59:651–723.

De Leng, D. and Heintz, F. (2016). Dyknow: A dynam-

ically reconﬁgurable stream reasoning framework as

an extension to the robot operating system. In 2016

IEEE International Conference on Simulation, Model-

ing, and Programming for Autonomous Robots (SIM-

PAR), pages 55–60. IEEE.

Della Valle, E., Ceri, S., Van Harmelen, F., and Fensel,

D. (2009). It’s a streaming world! reasoning upon

rapidly changing information. IEEE Intelligent Sys-

tems, 24(6):83–89.

Evans, R., Hernández-Orallo, J., Welbl, J., Kohli, P., and

Sergot, M. (2021). Making sense of sensory input.

Artiﬁcial Intelligence, 293:103438.

Fischer, J. E., Crabtree, A., Colley, J. A., Rodden, T., and

Costanza, E. (2017). Data work: how energy advisors

and clients make iot data accountable. Computer Sup-

ported Cooperative Work (CSCW), 26(4):597–626.

Fischer, J. E., Crabtree, A., Rodden, T., Colley, J. A.,

Costanza, E., Jewell, M. O., and Ramchurn, S. D.

(2016). " just whack it on until it gets hot" working

with iot data in the home. In Proceedings of the 2016

CHI Conference on Human Factors in Computing Sys-

tems, pages 5933–5944.

Giatrakos, N., Alevizos, E., Artikis, A., Deligiannakis, A.,

and Garofalakis, M. (2020). Complex event recogni-

tion in the big data era: a survey. The VLDB Journal,

29(1):313–352.

Glass, D. H. (2019). Competing hypotheses and abductive

inference. Annals of Mathematics and Artiﬁcial Intel-

ligence, pages 1–18.

Golab, L. and Özsu, M. T. (2003). Issues in data stream

management. ACM Sigmod Record, 32(2):5–14.

Heintz, F. (2009). DyKnow: A stream-based knowl-

edge processing middleware framework. PhD thesis,

Linköping University Electronic Press.

Heintz, F., Kvarnström, J., and Doherty, P. (2010). Bridg-

ing the sense-reasoning gap: Dyknow–stream-based

middleware for knowledge processing. Advanced En-

gineering Informatics, 24(1):14–26.

Inoue, K., Furukawa, K., Kobayashi, I., and Nabeshima, H.

(2009). Discovering rules by meta-level abduction.

In International Conference on Inductive Logic Pro-

gramming, pages 49–64. Springer.

Jain, D. and Beetz, M. (2010). Soft evidential update

via markov chain monte carlo inference. In Annual

Conference on Artiﬁcial Intelligence, pages 280–290.

Springer.

Kate, R. J. and Mooney, R. J. (2009). Probabilistic abduc-

tion using markov logic networks. In In: IJCAI-09

Workshop on Plan, Activity, and Intent Recognition.

Kowalski, R. and Sergot, M. (1989). A logic-based calculus

of events. In Foundations of knowledge base manage-

ment, pages 23–55. Springer.

ICPRAM 2022 - 11th International Conference on Pattern Recognition Applications and Methods

328

Kurze, A., Bischof, A., Totzauer, S., Storz, M., Eibl, M.,

Brereton, M., and Berger, A. (2020). Guess the data:

Data work to understand how people make sense of

and use simple sensor data from homes. In Proceed-

ings of the 2020 CHI Conference on Human Factors

in Computing Systems, pages 1–12.

Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh-

man, S. J. (2017). Building machines that learn and

think like people. Behavioral and brain sciences, 40.

Laput, G., Zhang, Y., and Harrison, C. (2017). Synthetic

sensors: Towards general-purpose sensing. In Pro-

ceedings of the 2017 CHI Conference on Human Fac-

tors in Computing Systems, pages 3986–3999.

Le-Phuoc, D., Eiter, T., and Le-Tuan, A. (2021). A scalable

reasoning and learning approach for neural-symbolic

stream fusion. In Proceedings of the AAAI Conference

on Artiﬁcial Intelligence, volume 35, pages 4996–

5005.

Lin, J., Keogh, E., Lonardi, S., and Chiu, B. (2003). A sym-

bolic representation of time series, with implications

for streaming algorithms. In Proceedings of the 8th

ACM SIGMOD workshop on Research issues in data

mining and knowledge discovery, pages 2–11.

Llinas, J. (2014). A survey of automated methods for sense-

making support. In Next-Generation Analyst II, vol-

ume 9122, page 912206. International Society for Op-

tics and Photonics.

Marcus, G. and Davis, E. (2019). Rebooting AI: Building

artiﬁcial intelligence we can trust. Vintage.

Martin, D., Kühl, N., and Satzger, G. (2021). Virtual sen-

sors. Business & Information Systems Engineering,

63(3):315–323.

McCarthy, J. (2006). Challenges to machine learning: Rela-

tions between reality and appearance. In International

Conference on Inductive Logic Programming, pages

2–9. Springer.

Mueller, E. T. (2014). Commonsense reasoning: an event

calculus based approach. Morgan Kaufmann.

Neisser, U. (2014). Cognitive psychology: Classic edition.

Psychology Press.

Olszewski, R. T. (2001). Generalized feature extraction

for structural pattern recognition in time-series data.

Carnegie Mellon University.

Richardson, M. and Domingos, P. (2006). Markov logic

networks. Machine learning, 62(1-2):107–136.

Saker, M. K., Zhou, L., Eberhart, A., and Hitzler, P. (2021).

Neuro-symbolic artiﬁcial intelligence. AI Communi-

cations, pages 1–13.

Schoenﬁsch, J., Meilicke, C., von Stülpnagel, J., Ortmann,

J., and Stuckenschmidt, H. (2018). Root cause anal-

ysis in it infrastructures using ontologies and abduc-

tion in markov logic networks. Information Systems,

74:103–116.

Schurz, G. (2016). Common cause abduction: The for-

mation of theoretical concepts and models in science.

Logic Journal of the IGPL, 24(4):494–509.

Siddiqui, T., Luh, P., Wang, Z., Karahalios, K., and

Parameswaran, A. (2020). Shapesearch: A ﬂexible

and efﬁcient system for shape-based exploration of

trendlines. In Proceedings of the 2020 ACM SIG-

MOD International Conference on Management of

Data, pages 51–65.

Skarlatidis, A. and Michelioudakis, E. (2014). Logical

Markov Random Fields (LoMRF): an open-source

implementation of Markov Logic Networks.

Skarlatidis, A., Paliouras, G., Artikis, A., and Vouros, G. A.

(2015). Probabilistic event calculus for event recog-

nition. ACM Transactions on Computational Logic

(TOCL), 16(2):1–37.

Spelke, E. S. and Kinzler, K. D. (2007). Core knowledge.

Developmental science, 10(1):89–96.

Suchan, J., Bhatt, M., and Varadarajan, S. (2019). Out of

sight but not out of mind: An answer set programming

based online abduction framework for visual sense-

making in autonomous driving. In Proceedings of the

Twenty-Eighth International Joint Conference on Arti-

ﬁcial Intelligence, IJCAI 2019, Macao, China, August

10-16, 2019, pages 1879–1885. ijcai.org.

Teijeiro, T. and Félix, P. (2018). On the adoption of abduc-

tive reasoning for time series interpretation. Artiﬁcial

Intelligence, 262:163–188.

Tran, S. D. and Davis, L. S. (2008). Event modeling and

recognition using markov logic networks. In Euro-

pean Conference on Computer Vision, pages 610–623.

Springer.

Tsitsipas, A., Eisenhart, G., Seybold, D., and Wesner, S.

(2022). Scalable shapeoid recognition on multivariate

data streams with apache beam. In Proceedings of the

Computing Conference 2022, forthcoming. Springer.

Tsitsipas, A., Schiessle, P., and Schubert, L. (2021). Scotty:

Fast a priori structure-based extraction from time se-

ries. In 2021 IEEE International Conference on Big

Data (BigData 2021), forthcoming.

Tsitsipas, A. and Schubert, L. (2021a). Modelling and

reasoning for indirect sensing over discrete-time via

markov logic networks. In Proceedings of the Twelfth

International Workshop Modelling and Reasoning in

Context (MRC 2021). CEUR-WS.org.

Tsitsipas, A. and Schubert, L. (2021b). On group theory and

interpretable time series primitives. In Proceedings

of the Seventeenth International Conference on Ad-

vanced Data Mining and Applications (ADMA 2021),

forthcoming. Springer.

Wasserkrug, S., Gal, A., Etzion, O., and Turchin, Y. (2010).

Efﬁcient processing of uncertain events in rule-based

systems. IEEE Transactions on Knowledge and Data

Engineering, 24(1):45–58.

Towards an Ensemble Approach for Sensor Data Sensemaking

329