Using Functional near Infrared Spectroscopy to Assess Cognitive

Performance of UAV Sensor Operators during Route Scanning

Jazsmine Armstrong

, Kurtulus Izzetoglu

and Dale Richards

School of Biomedical Engineering, Science and Health Systems, Drexel University,

3141 Chestnut Street, Philadelphia, PA, U.S.A.

School of Mechanical, Aerospace and Automotive Engineering, Coventry University, Coventry, U.K.

Keywords: Unmanned Aerial Vehicles, UAV, Sensor Operators, Cognitive Workload, Functional near Infrared

Spectroscopy, fNIRS.

Abstract: The composition of UAV (Unmanned Aerial Vehicle) crew will sometimes define roles specific to tasks

associated with the Ground Control Station (GCS). The sensor operator task is specific to both the type of

platform and GCS they are operating, but in many instances the role of this operator is critical in determining

mission success. In order to assess mission effectiveness we applied human performance measures that

focussed on neurological brain imaging techniques and other physiological biomarkers in conjunction with

behavioral data acquired from the sensor operator task. In the execution of the experiment, this included such

tasks as route scanning, target detection and positive identification, and the tracking of identified targets.

Within the scope of this paper, we reported the preliminary results for the route scanning task. Over the

duration of three trials brain activity measures from the prefrontal cortex region were acquired via functional

near infrared spectroscopy (fNIRS) in this research study. As the trials progressed, there was a significant

difference between low and high performers on the route scanning task as determined by specific biomarkers,

namely oxygenated haemoglobin. These findings support previous studies and indicates the benefits of

applying neurophysiological measures in order to gain further objective insight into human cognitive

performance. The use of fNIRS in this context is also discussed in terms of providing a key benefit in

dynamically evaluating human performance in parallel with personalized training for UAV operators.

1 INTRODUCTION

A great deal of research is currently underway that

focuses efforts on integrating routine flights of UAVs

(Unmanned Aerial Vehicle) into the national airspace

system (NAS). Of course this is not as straightforward

as simply allowing such operations in current air

traffic operations (utilising the same traffic

management infrastructure), but requires a fuller

understanding of not only the nature of UAV

operations from this perspective but also the role of

the human who is tasked to control such platforms.

Thus, herein lies a unique problem. Apart from the

obvious differences between operating a manned

platform versus an unmanned platform, the

composition and defined roles of a UAV flight crew

is somewhat more dependent on the platform and

nature of operations. In some instances several roles

within the UAV crew may be shared across several

crew members, alternate between them, or in some

instances be carried out by the same individual

(Wickens et al., 2005). This presents the operator as a

focal point for ensuring not only the safe flying of the

UAV, but also the operational effectiveness

associated with the mission. By examining the way in

which UAV missions are conducted within the

defence realm, it is possible to use this as a means by

which we can assess how the operator (in this instance

the operator directing the sensor, as opposed to the

pilot) may be assessed in terms of his/her

effectiveness.

Previous evidence has suggested that nearly 70%

of all UAV incidents may cite causal factors that

would suggest the role of human factors as a

contributing factor (Williams 2004). The sensor

operator (SO) adopts a role that dictates a number of

specific tasks. At some points these tasks may be to

assist the pilot in command, perform other tasks not

specific to the mission but pertinent to the safe

operation of the flight (e.g. liaise with Air Traffic

286

Armstrong, J., Izzetoglu, K. and Richards, D.

Using Functional near Infrared Spectroscopy to Assess Cognitive Performance of UAV Sensor Operators during Route Scanning.

DOI: 10.5220/0006731502860293

In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 4: BIOSIGNALS, pages 286-293

ISBN: 978-989-758-279-0

Control). However, we will focus on the SO tasks that

are related to the mission context. Primarily this

involves a great deal of sensor manipulation, either

self-directed or instructed via a Mission Commander

or third party. In many instances the sorts of tasks this

would involve include scanning areas along the flight

path both on the ground and in the air, searching and

classifying tracks (and possible threats/targets);

whilst also focussing on the operational requirements

for the specific mission which may include other

requirements of the SO, such as gathering intelligence

in relation to building Pattern of Life (POL) and any

behaviour that may stand-out from the norm within

such POL (Kenner & Wolf, 2003).

Therefore the development of an efficient SO

must therefore be entrenched within an effective

training programme that allows the individual to

develop appropriate cognitive styles (Kirton 2003)

that best suit SO- specific tasks. In some instances

these may present as a completely new skill set that

must be effectively conveyed to trainees who may

have prior knowledge/experience of SO tasks or even

piloting manned aircraft. An appropriate training

methodology must be therefore designed that will

help to reinforce SO-specific cognitive styles that will

best suit the SO tasks. This will not only provide a

more effective means of conveying training to the

individual, but also assist in effectiveness as related

to the SO task; such as improving SO performance.

This includes improved search behaviour and track

classification and reducing incidents of false

identification (Kowalski-Trakofler & Barrett, 2003).

Apart from taking into account the different

cognitive styles of operators during training, it is

important to consider the role of attention during the

SO task. To some extent an individual's cognitive

style (or aptitude for the task) will determine how

they guide and control attentional resource and

demand during the course of a task. However, it is

beneficial to train attentional focus through the use of

realistic scenarios that most accurately mimic those

found in real-life (Wolfe et al., 2005). Prior studies

have shown that repetitive visual search training does

indeed help to transfer the search task from an overly

active one to a task where it is nearly automatic. This

can lead to improved target detection, and quicker

response time in regards to identifying a track.

(Treisman, Viera & Hayes, 1992). However, the

ability to use training for facilitating preattentive

processing (in terms of developing an instinctual

pattern for attentional processing) remains elusive.

In order for the operator of the UAV to reach a

particular level of operational competence, it is

expected that the operator will present significant

cognitive effort and activity associated with the

region of the brain associated with performing those

tasks. As with other muscle masses in the body, when

effort is required then we can expect a great deal of

metabolic activity, in order for the muscle to produce

an output. This may be comparative to cortical

activity when confronted with a task that has a level

of cognitive demand on resources. In order to meet

the metabolic demand associated with a task that

requires cognitive effort we can observe increases in

blood flow to the area of the brain that is asscoaited

with the task. In essence this is the blood carrying the

oxygen to the parts of the brain that require 'feeding'

in order to address the task being considered.

Traditionally these metabolic changes in brain states

have been measured by MRI, fMRI, and EEG. While

these established measures have allowed us to better

understand the physiological mechanisms for

cognitive activity they also have large degree of

constraints that do not allow us to conduct real-time

studies of participants in naturalistic settings (and at a

reasonable cost).

However, the use of different brain-imaging

techniques have allowed us to appreciate which areas

of the brain are clossely associated with cognitive

functioning. Of particular interest are the higher level

cognitive functioninig that include such tasks as

decision-making, problem-solving and attentional

focus. Advances in optical brain imaging techniques,

and in particular functional near infrared

spectroscopy (fNIRS), allow us to monitor the

hemodynamic changes of the participant as they

progress through different tasks associated with SO

role.

1.1 Functional near Infrared

Spectroscopy

Functional near infrared spectroscopy is a

neuroimaging modality that exploits the optical

properties of biological tissues and hemoglobin

chromophores. fNIRS deploys wavelength in the

range between 700 to 900 nm. At this wavelength, the

majority of biological tissues, including neural

tissues, are transparent while the chromophores of

oxygenated and de-oxygenated hemoglobin (HbO2

and HB, respectively) are found to be the main

absorbers. By examining the manner in which light

passes through cortical tissue (utilising the modified

Beer Lambert Law), concentrations of oxygenated

and deoxygenated hemoglobin can be calculated

(Jobsis, 1977; Cope, 1988). The changes in

oxygenated and deoxygenated hemoglobin are

directly associated with changes in brain activity

Using Functional near Infrared Spectroscopy to Assess Cognitive Performance of UAV Sensor Operators during Route Scanning

287

changes (Izzetoglu et al., 2004, Villringer et al.,

1997).

Figure 1: 16-Channel fNIRS System.

The current fNIRS system (as shown in Fig.1)

used in this study is proven to be a safe, non-invasive

optical method that can be utilized to monitor activity

within the prefrontal cortex of the brain (Obrig et al.,

1997, Villringer et al., 1997). Because of its

portability and ability to capture continuous measures

of the hemodynamic response while allowing

measures in natural settings, fNIRS seems a suitable

neuroimaging modality for assessment of pilot

performance in high fidelity simulation as well as

field study conditions.

1.2 UAV Training Simulator

In order to accurately translate the results of SO skill

acquisition in the field, it would follow logically that

the training apparatus must not only resemble the

work environment of the SO, but also present a high

fidelity representation of the task. (Cooke & Shope,

2004). To address this, we utilized Simlat’s C-STAR

simulator in this proof of concept study. The C-STAR

system consists of Performance Analysis &

Evaluation module (PANEL) that collects and

processes simulation data, whilst producing

comprehensive reports of trainees’ performance in

various tasks during a mission. The simulator has the

capability to transfer views between sensor operator

and pilot, as well as a realistic landscape, targets, and

accurate representations of UAV operator controls.

The software allows for two trainees and one

instructor to operate the generic tactical unmanned

vehicle (G-TAC UAV) simultaneously and in

designated roles, as well as the capability of the

instructor to manually or automatically preset

‘emergency’ situations that the pilot(s) might

encounter such as cloud cover, precipitation, and

equipment failure. This robust system is ideal for real

world training of both the sensory operator and pilot

roles of the UAV (Fig. 2).

Figure 2: UAV Simulator: C-STAR system.

2 METHOD

2.1 Participants

Fifteen participants between the ages of 19 to 40 ( =

23.8; SD =5.3) participated in the Institutional

Review Board (IRB) approved study. Out of 15

participants, there was only one participant excluded

due to an incomplete session. All participants (no

prior UAV piloting experience) fulfilled inclusion

and exclusion criteria of the IRB; they had either

normal or corrected to normal vision, and were

verified as right handed via use of the Edinburgh

Handedness assessment.

2.2 Experiment Protocol

The experimental protocol incorporated scan and

target search tasks. The generic tactical unmanned

vehicle (G-TAC UAV) was utilized to automatically

follow a pre-determined route. The trainee screen was

separated into a GPS screen to the left and a sensor

payload screen to the right (Fig. 3). The map screen

was intended to show the location of the UAV and the

route that the UAV has travelled along. The map

screen was locked to the UAV position, so the SO

could see the UAV move in conjunction with the

map. However, the trainee was provided with the

option to zoom in and out of the map. This was

intentionally designed to rule out any confounding

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

288

factors that may have occurred regarding the position

of the vehicle on the map and consequent loss of

situational awareness of the SO as they actively

searched using the sensor screen to the right.

Figure 3: Experimental design view: Payload screen.

The sensor screen displayed the simulated model

of the landscape of Mallorca, Spain. Within the

sensor display a crosshair and zoom level gauge were

utilised by the operator in order to complete their

tasks. The simulated time and the duration of each

session was provided to the operator and displayed

above this display (Fig. 3). Other instruments were

provided to the operator that displayed primary flight

data, but were not able to be changed by the operator.

Figure 4: The route where the search task was performed.

At the start of each session the UAV was at 2500

feet with a 64 degree heading angle, and set to travel

along a designated flight path at a fixed speed of 70

knots (80.55 miles per hour). Between each sub-area

there was a length of 'dead space' at which time the

instructor could reposition the camera angle to the

widest zoom angle and positioned facing the nose of

the UAV (Fig. 4).

As the vehicle proceeded along the waypoints,

participants were instructed to scan the designated

route. A successful scan was determined to be a scan

at a zoom level lower than 15 degrees. Each sub-area

consisted of one target that needed to be classified as

a threat (in this instance a red civilian bus). To track

and identify a target, the participant was instructed to

zoom in as close as possible in the payload screen

using the hat switch on the joystick. Once the target

was located, the participant was instructed to lock

onto the target when it was positioned in the camera’s

crosshairs. In order to ascertain successful

identification, the participant was advised to track the

target for three seconds before moving the sensor off

the target and continuing to scan the surrounding area.

Each trial lasted for approximately 17.5 minutes and

was repeated 3 times, resulting in 52 minutes of total

flight time in the session. The route and scanning area

of the map were identical for all three trials, however

location of the target component was changed each

time. Clearly, this approach was implemented to

prevent a participant from simply recalling the

location of the target from the previous mission. The

C-STAR simulator system recorded percentage of the

area scanned, the duration of each session, and the

time at which a target was identified.

2.3 Data Analysis

The following task protocol data was analysed within

the scope of this paper:

The Scan Task: Participants were separated into

low and high performers based on their behavioural

performance data to conduct comparisons between

fNIRS measurements amongst the trials. The

simulator software provided behavioural data that

determined the percentage of the designated area that

was properly scanned, over scanned, or not scanned.

The percentage of properly scanned areas and the

camera field of view was used as a direct metric for

determining these performance levels (as shown in

Fig. 5).

fNIRS Data Analysis: Continuous wave and 16-

channel, covering left and right hemispheres, fNIRS

system was used in this study. Sampling rate was

2Hz. For the fNIRS data analysis, a low pass filter

with a finite impulse response and linear phase was

applied to the raw light intensity data for each

wavelength at each channel to tease out high

frequency noise, respiration and cardiac cycle effects

(Izzetoglu et al., 2005). Then, modified Beer-Lambert

Law (MBLL) was used to calculate the oxygenated

Using Functional near Infrared Spectroscopy to Assess Cognitive Performance of UAV Sensor Operators during Route Scanning

289

Figure 5: Example of performance analysis for scanning

task.

and deoxygenated haemoglobin changes at each

channel (Cope & Delpy, 1988; Villringer & Chance,

1988).

3 RESULTS

Based on the behavioural performance measures, the

trainees were classified into two groups within the

scope of the preliminary study reported here. Those

who increased their scan percentage within certain

field of view were placed in the high performers

group (Fig. 6a). If a participant performed worse

between initial trial and final trial, they were

classified as the low performer (Fig. 6b).

A significant difference in scanning task was

observed between initial and final trials, Trial 1 and

Trial 3, respectively. These results were also

confirmed by per participant analysis (Fig. 7). High

Figure 6: Scanned area measures (n=14 participants) versus

trials for a. high performers (F (2, 107) = 7.419, p=.001), b.

low performers, (F (2,143) =3.501; p= .033).

performers (n=6) revealed the expected task

performance progress, whereas low performers’

(n=8) task performance did not improve between

initial and final trials.

The preliminary analysis for the fNIRS

measures was to investigate the measures from the

prefrontal cortex (PFC) region area associated with

attention. We calculated oxygenation changes for low

and high performers using MBLL. We hypothesized

that the high performers would have higher

oxygenation than low performers’ levels. Figure 9

depicts oxygenation changes from Optode 11 located

over the middle frontal gyrus of the right hemisphere

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

290

Figure 7: Performance measures by participant.

that was previously reported for the attention task

studies (Izzetoglu M, 2007). The oxygenations

changes were higher for the high performers as their

scanning performance improved over 3 trials.

On the other hand, low performers oxygenation

changes remained low, which would be expected

when we observe their scanning task performance.

However, a low oxygenation does not always mean a

lack of cognitive effort. For example, we found that

the high performers’ oxygenation levels decreased as

they became more proficient over time whilst

performing the scanning task. A finding that has

previously been reported using fNIRS when assessing

Figure 8: Approximate area of activation region related to

the task reported here- fNIRS oxygenation changes.

Figure 9: fNIRS results for the low and high performers.

(F= 3.095, p=.083 for high performers between initial trial

and final trial. For low performers, F<1).

unmanned and manned pilots while they were

acquiring new skills (Ayaz et al., 2012; Hernandez, et

al., 2015; Izzetoglu, et al 2014; Menda, et al., 2011).

Thus a similar trend was observed here for the high

performers. That is, while you become familiar with

the task, the oxygenation levels at the PFC region

decreased. This was not seen in the low performers

during this study. There was no significant

differences between final and initial trials for this

group.

Right

Left

Using Functional near Infrared Spectroscopy to Assess Cognitive Performance of UAV Sensor Operators during Route Scanning

291

4 DISCUSSION

The ability of a UAV crew to conduct effective

operations is based to a large extent on their training.

The role of the SO is critical in determining the

success of a mission and rapid decision making must

often be made in a timely manner, and sometimes

calls upon greater demand of attentional resources.

This is particularly true of missions that require both

rapid response to a changing environment, and also

missions where vigilance may result in mental fatigue

(and increased likelihood of human error). It is

essential therefore that training regimes for operators

takes these factors into account and in many cases this

will often rely upon high fidelity simulation and

scenario-based mission tasks. However, the

evaluation techniques currently being used to assess

such training tends to focus on behaviour markers

related to task, rather than the cognitive ability of the

operator. This study builds on the knowledge that we

have already gathered by using other functional

neurological imaging techniques (such as MRI, EEG)

and harnessed the utilisation of fNIRS within a field-

based study to demonstrate the benefits this form of

measurement may have when assessing operator

cognitive state. Further, there are vast amounts of

research reported fNIRS studies in the aerospace

domain. The training effect and expertise

development for manned aircraft pilots was studied

with the fNIRS and reported oxygenation decrease on

PFC (Hernandez, et al., 2015). Cognitive workload of

air traffic controllers was measured by using fNIRS

and explored assessment of working memory from

the PFC (Ayaz, et al., 2012). Further, we studied

UAV pilots on landing and approach task while

measuring the expertise development via fNIRS

(Izzetoglu, et al., 2014; Ayaz, et al., 2012).

The current study identified that as participants

acquired knowledge and gained new skills we are

able to observe how they draw on oxygenation to

increase their cognitive effort associated with

different elements localised within the PFC. On

closer inspection this localised oxygenation change is

associated with parts of the PFC closely aligned with

the middle frontal gyrus of the right hemisphere

associated with attention.

The experimental protocol reported here was very

complex task. Although these results, behavioural

and neuro-physiological measures, are in line with

previous reports, one should conduct further analysis

for all the sub-tasks for the payload operators and

investigate fNIRS measures acquired from all the

PFC regions for each sub-task and overall task

performance.

This study has demonstrated the benefit of

utilising fNIRS as a biomarker for cognitive function

of participants employed in conducting UAV sensor

operator tasks. While it has highlighted the nature of

cognitive function within the PFC, it also can be used

as an evaluation of expertise during multiple training

sessions.

ACKNOWLEDGEMENTS

The authors express their gratitude to Shahar Kosti

for his valuable review and support for scenario

design, UAV simulator setup and behavioral data.

The authors also would like to thank Simlat, Inc for

providing access to their UAV training simulator.

REFERENCES

Ayaz H, Shewokis PA, Bunce S, Izzetoglu K, Willems B,

Onaral B. (2012). Optical brain monitoring for operator

training and mental workload assessment. Neuroimage,

59(1):36-47. PMID: 21722738.

Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic

task environment. In Schiflett, S. G.

Cope, M., and Delpy, D. T. (1988). System for long-term

measurement of cerebral blood flow and tissue

oxygenation on newborn infants by infra-red

transillumination. Med. Biol. Eng. Comput., 26: 289–

294.

Hernandez-Meza, G., Slason, L., Ayaz, H., Craven, P.,

Oden, K., & Izzetoglu, K. (2015). Investigation of

Functional Near Infrared Spectroscopy in Evaluation

of Pilot Expertise Acquisition. In D. D. Schmorrow &

C. M. Fidopiastis (Eds.), Foundations of Augmented

Cognition (Vol. 9183, pp. 232-243): Springer

International Publishing.

Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K.,

Chance, B (2004) Functional optical brain imaging

using near-infrared during cognitive tasks.

International Journal of Human-Computer Interaction

17, 211-227 (2004)

Izzetoglu, K., Ayaz, H., Hing, J., Shewokis, P., Bunce, S.,

Oh, P., & Onaral, B. (2014). UAV Operators Workload

Assessment by Optical Brain Imaging Technology

(fNIR). In K. P. Valavanis & G. J. Vachtsevanos (Eds.),

Handbook of Unmanned Aerial Vehicles (pp. 2475-

2500): Springer Netherlands

Izzetoglu M, Bunce S, Izzetoglu K, Onaral B, Pourrezaei K

(2007). Functional Brain Imaging Using Near Infrared

Technology for Augmented Cognition. IEEE Eng. in

Med. and Biol. Mag., Special issue on The Role of

Optical Imaging in Augmented Cognition, 26(4):38 –

46.

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

292

Jobsis, F.F (1997) Noninvasive, infrared monitoring of

cerebral and myocardial oxygen sufficiency and

circulatory parameters. Science 198, 1264-1267

Kenner, N., & Wolfe, J. M. (2003). An exact picture of

your target guides visual search better than any other

representation. Journal of Vision, 3, 230a.

Kirton, M.J. (2003) Adaptation and innovation in the

context of diversity and change Routledge, London

Kowalski-Trakofler, K. M., & Barrett, E. A. (2003). The

concept of degraded images applied to hazard

recognition training in mining for reduction of lost-time

injuries. Journal of Safety Research, 34, 515-525.

Menda, J., Hing, J.T., Ayaz, H., Shewokis, P.A., Izzetoglu,

K., Onaral, B., Oh, P. (2011) Optical Brain Imaging to

Enhance UAV Operator Training, Evaluation, and

Interface Development. J. Intell. Robotics Syst. 61,

423-443 (2011)

Obrig, H., Villringer, A (1997) Near-infrared spectroscopy

in functional activation studies. Can NIRS demonstrate

cortical activation? Advances in Experimental Medicine

and Biology 413, 113-127

Treisman, A., Vieira, A., & Hayes, A. (1992).

Automaticity and Preattentive Processing. The

American Journal of Psychology,105(2), 341-362.

Villringer A, Chance B (1997). Non-invasive optical

spectroscopy and imaging of human brain function.

Trends in Neuroscience, 20:435-442.

Wickens, C.D., Dixon, S., Goh, J., & Hammer, B.

(2005). Pilot dependence on imperfect diagnostic

automation in simulated UAV flights: An attentional

visual scanning analysis. In Proceedings of the 13th

International Symposium on Aviation Psychology (pp.

919-923). Columbus, OH: Association of Aviation

Psychology

Williams, K.W (2004) A summary of unmanned aircraft

accident/incident data: human factors implications.

U. S. Department of Transportation Report, No.

DOT/FAA/AM—04/24 (2004)

Wolfe, J. M., Horowitz, T. S., & Kenner, N. M. (2005).Rare

items often missed in visual searches. Nature, 435, 439-

440.

Using Functional near Infrared Spectroscopy to Assess Cognitive Performance of UAV Sensor Operators during Route Scanning

293