Detection and Identiﬁcation of Threat Potential of Ships using Satellite

Images and AIS Data

Akash Kumar, Aayush Sugandhi and Yamuna Prasad

Indian Institute of Technology Jammu, India

Keywords:

Automatic Identiﬁcation System (AIS), VGG16, Faster RCNN, MMSI Number, Draught Weight, Blind

Period, Port Call.

Abstract:

This paper addresses the issue of vessel tracking using Automatic Identiﬁcation Systems (AIS) and imagery

data. In general, we depend on AIS data for the accurate tracking of the vessels, but there is often a gap

between two consecutive AIS instances of any vessel. This is called as blind period or the inactivity period.

In this period, we can not be sure about the location of the ship. The duration of inactivity period is quite

variable due to various factors like weather, satellite connectivity and manual turn off. This makes tracking

and identiﬁcation of any threat difﬁcult. In this paper, we propose a two-fold approach for tracking and

identifying the potential threat using deep learning models and AIS data. In the ﬁrst fold, the ships out of

satellite imaging are identiﬁed while in the second fold, the corresponding AIS data is analysed to discover

any potential threat or suspicious activity.

1 INTRODUCTION

The problem of surveillance and threat monitoring

is utmost important for the security in coastal areas.

This problem is very challenging and many concerns

with national security. There were many attempts

to monitor the suspicious activities in sea using im-

agery and signal data (JH et al., 2018); (Lane et al.,

2010); (Chang, 2003); (Garagic et al., 2009). In gen-

eral, ocean-going vessels communicates their posi-

tions and route informations among each other using

AIS to avoid collisions. AIS can also be used to mon-

itor the vessels remotely. There are many gaps in AIS

transmissions due to high vessel density, poor qual-

ity transmission and jamming/disabling of transmit-

ters etc. This leads to broken monitoring which is

a critical problem in vessel monitoring and collision

avoidance. In (JH et al., 2018), the authors proposed

a model to identify the high risks gaps occured due

to intentional disabling of the transmitter using prob-

abilistic models in Arafura Sea vessels. Further in

(Lane et al., 2010), ﬁve anomalies such as deviation

from standard routes, unexpected AIS activity, unex-

pected port arrival, close approach, and zone entry are

explored and the risk is computed using bayesian net-

work.

https://orcid.org/0000-0002-3709-7956

In (Rhammell, 2018), satellite images of ships in

bay areas are produced in 2018 at Kaggle website.

There were many attempts to do segmentation and

identify the ships in satellite images (Swamidason

et al., 2020); (Xie et al., 2020). The methods avail-

able in the literature as outlined above does not ex-

ploit the use of AIS data. In this paper, we attempt to

exploit the AIS data alongwith the satellite imagery.

for surveillance and threat monitoring caused due to

the inconsistency in data. In this work, we assumed

to monitor any particular area at a time. The ﬁrst step

is to identify any boats in the speciﬁed location using

the satellite images. In the next step, we need to locate

all the vessels in the image and ﬁnd their exact GPS

coordinates. Once we have the latitude and longitude

of all the vessels, we can look up for them in AIS data

to identify the vessels or the vessels that should be

present on the basis of their last transmission. In the

ﬁrst step, we present the results for identiﬁcation of

the vessels in any vicinity.

In the next step, we identify the threat potential

of all the vessels using the AIS data. In AIS data, we

can get certain details about the vessel like- MMSI ID,

Vessel type, Cargo number, SOG, transmission times-

tamp, Course, heading, maximum speed and Draught

weight of the vessel. We can use the MMSI id and

the Cargo details to check whether the vessels are au-

thorized or not. Further, even if authorized, vessels

Kumar, A., Sugandhi, A. and Prasad, Y.

Detection and Identiﬁcation of Threat Potential of Ships using Satellite Images and AIS Data.

DOI: 10.5220/0010914600003124

In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 4: VISAPP, pages

691-698

ISBN: 978-989-758-555-5; ISSN: 2184-4321

691

may pose threat, so we can run all the AIS transmis-

sions by that vessel over a few days (or weeks) on a

deep learning model and try to identify the pattern in

the transmission in order to ﬁnd any suspicious ac-

tivity that could have been performed by the vessel.

It should be noted that the current state-of-the-art ap-

proaches try to analyse the threat entirely on the basis

of AIS data, where, the threat is identiﬁed only af-

ter the transmission is back online. In this case, the

information about the blind period is not available.

The paper is organised as follows: In the Section

2, we describe the available literature for ship identi-

ﬁcation and suspicious activity monitoring task. Sec-

tion 3, presents the proposed methodology and exper-

iments. The results are discussed in Section 4. Also,

in Section 4, Further directions for the improvements

are also discussed in Section 4. We conclude our work

in Section 5.

2 RELATED WORK

In this Section, we present the related work done on

Ship data available in (Rhammell, 2018). The sample

dataset is available for illustration in Figure 9.

2.1 Generative Additive Models (GAM)

(JH et al., 2018)

In this model, a GAM based model is proposed to

compute the spatial and temporal probability of a suc-

cessful transmission in a speciﬁc time window and

geolocation. This model calculates the expected fre-

quency of the transmissions using AIS polls received

from the vessels on daily basis. This expected fre-

quency is affected by terrestrial receiver availabil-

ity, satellite coverage, trafﬁc density, and vessel den-

sity. This paper focuses on identifying intention-

ally disabled transmissions that arises due to longer

runs of non-transmission or much lesser transmis-

sion frequencies. The deviations in expected trans-

mission frequencies are computed in order to identify

the intentionally disabled transmissions. This can also

identify the malfunctioned AIS transmitters.

2.2 Bayesian Method (Lane et al., 2010)

In this approach, authors describes the ﬁve anomalies

such as deviation from standard routes, unexpected

AIS activity, unexpected port arrival, close approach,

and zone entry. The authors presented various proba-

bility distribution models to address the above issues

as follows:

• Gaussian mixture model (GMM) for Identifying

deviations from standard routes.

• Bayesian Estimates for identifying unexpected

AIS activity.

• Markov models for identifying unexpected port

arrival.

• Spatial indexing for identifying close approach.

• Gaussian distribution for identifying zone entry.

Further, these ﬁve anomalies are modelled using

Bayesian Network to infer the potential threat prob-

ability.

In our proposed work, we would like to combine

the satellite imagery and AIS data to monitor the ves-

sels for any potential threats.

3 PROPOSED METHODOLOGY

AND EXPERIMENT

As our ﬁrst step is to identify ships in a satellite image,

we have used a dataset of cropped satellite images of

ships (1000) and non-ships (3000), downloaded from

Kaggle-datasets. As the data was imbalanced, we

augmented the data to get 1:1 ratio of ships to non-

ship images. Then we trained a VGG16 based archi-

tecture on the dataset. Here, instead of running clas-

siﬁcation sequentially over the total image, a selec-

tive search segmentation method is used which dis-

tinguishes main region of interests (ROIs) and then

the saved VGG16 classiﬁcation model is run on that.

Once the Ships are identiﬁed, we are supposed to

locate them on the image and then ﬁnd their coordi-

nates. Now, as we have to only pinpoint the ships in a

predeﬁned proposed regions (due to selective search

in the previous step), we used Faster RCNN model

to make bounding box around all the ships in the im-

age. Then we used an open-source API to pinpoint

the coordinates of the vessels using the location of the

nearest port.

Using the coordinates and matching the details

with the AIS database, we will all the necessary de-

tails about the ship. Now, in case the ship can not

be traced back to the AIS data, it should be put under

suspicion. Otherwise, we use the Cargo number of the

vessel and apply the government formula to it to ﬁnd

the type of cargo and the threat that cargo poses. The

MMSI number is then used to see if the vessel is au-

thorised. Unauthorised vessels should be considered

suspicious.

Lastly, we check the past record of the authorised

vessel for any suspicious activity. For this, we use all

of its last AIS transmissions over past few days. We

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

692

used features like Coordinates, Timestamp, Draught

weight and maximum speed. Using the Coordinates,

we computed the distance of the vessel to the near-

est port at every time it transmitted. As there are dif-

ferent sources for the datasets, we have chosen the

commonly available features for analysing the data.

Now using the time stamp we calculated the time gap

between every two transmissions. There is no stan-

dard threshold for identifying the blind period, we

have used statistical method between two continuous

transmissions to mark blind period. For this, we com-

puted the average time gap over 10 continuous trans-

missions and the standard deviation for the same. If

the time gap between any two transmissions is more

than the sum of average and the standard deviation,

we marked it as ’Blind period’. We also computed

the difference in draught weight between every two

transmissions. We then developed a deep learning

model and trained in on a dataset with features like

time gap, change in latitude and longitude, maximum

speed, SOG, distance from port before and after and

the target class as the suspicion level.

In addition, we are using rule based application

of the model. We are only applying the model when

there is a possibility of something suspicious, i.e. ei-

ther a blind period or a change in draught weight. We

then used this model on the features extracted from

the data of the vessel that we are tracking. The model

provides the suspicion level of the vessel, and also the

type of suspicious activity it could have been doing

like unauthorised port call, illegal ship to ship trading

or illegal ﬁshing. The proposed approach is illustrated

in Figure 4.

In this work, we have proposed a model for ves-

sel tracking and threat prediction using deep learn-

ing framework by employing satellite images and syn-

chronised AIS data. The primary results are motivat-

ing for the deployment of the our proposed model in

real-time.

4 RESULTS

4.1 Dataset

We have used a dataset of cropped satellite images of

ships (1000) and non-ships (3000), downloaded from

Kaggle datasets https://www.kaggle.com/rhammell/

ships-in-satellite-imagery (Rhammell, 2018). This

dataset is imbalanced. In order balance the datasets,

ofﬂine image augmentation is performed using Hor-

izontal ﬂips, random crops, strengthening and weak-

ening of brightness as well as contrasts, and applying

Figure 1: Ship Detection.

(a) Scene 1: ship (b) Scene 1: ship

Figure 2: These ﬁgure represents the scenes containing

ships (a and b) and no ships (c and d).

afﬁne transformations. After the augmentation, the

data has 1:1 ratio of ships to non-ship images.

In order to identify the suspicion level from AIS

data, we used dataset from IEEE Dataport (Hakola,

2020). The AIS dataset contains the information

about timestamp, mmsi, lat, lon, speed (meters per

second), course (degrees), heading (degrees), turn-

rate (degrees per minute), breadth (meters), ves-

sel type, vessel max speed (meters per second), draft

(meters), power, dwt (tons) and ice-class. Out of

Detection and Identiﬁcation of Threat Potential of Ships using Satellite Images and AIS Data

693

Figure 3: Ship Detection Accuracy.

Figure 4: Proposed Approach.

Figure 5: AIS threat detection (confusion Matrix) for

Dataset1.

these many features, the seven features were derived

from the dataset. These features are time difference,

longitude difference, latitude difference, max speed,

draught weight difference, distance from shore for

Figure 6: AIS threat detection (confusion Matrix) for

Dataset2.

ﬁrst instance and distance from shore for second in-

stance. All these features are derived from a pair of

consecutive AIS transmissions.

As the dataset didn’t have any threat/suspicion la-

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

694

Figure 7: AIS threat detection loss and accuracy for Dataset1.

Figure 8: AIS threat detection loss and accuracy for Dataset2.

bel, we created the labels using statistical approach.

It lacked certain features like ‘Draught Weight’, so

we added the same on our own. The ﬁrst step was to

identify Blind Periods, now if the blind period was

less than an hour, we assumed it to be some tech-

nical issue and marked it safe (label 0), if the blind

period was over an hour and less than 6 hours, and

there is a difference in draught weight, we assume

it to be suspicious (label 1) of doing some illegal

ship to ship trading, illegal garbage dumping or ille-

gal ﬁshing. Now in case, the blind period is over six

hours and less than 24 hours, and if the total distance

of the ship from the nearest port, before and after

the blind period is less than max(speedo f vessel) ×

(blind period–2hours), we assumed it to be a port call

(label 2). And if the Blind period was over 24 hours,

we assumed it to be in a dockyard (label 3). Rest all

are assumed to be safe i.e. label 0. We name this

dataset as dataset1 in our experiments.

Further, we have used the same dataset but instead

of going for a staring forward statistical labelling, we

introduced randomness varying from 5% to 40% for

different labels while keeping the same approach as

the previous one. This dataset is named as dataset2 in

out experiments. Both the datasets are used to detect

the threat.

4.2 Discussion

The satellite images were used to ﬁnd the ship using

Faster RCNN ((Ren et al., 2017)) with VGG-16 based

CNN model ((Liu and Deng, 2015))

. The Faster

RCNN created the bounding boxes around the vessels

while VGG-16 framework (a CNN model) detects the

vessels with an accuracy of 98% (Figure 1 and Figure

Once the vessels (ships) are identiﬁed, the cor-

responding synchronised AIS data is captured for

threat detection. In order to develop a threat model

the synchronised dataset1 and dataset2 is used for

threat detection. A simple 4-layer dense neural net-

work (DNN) model with relu activations has been

used to train the AIS data for threat detection. The

-layer represents the multi-class classiﬁcation out-

put, therefore, we have used softmax activation with

cross-entropy as the loss function. The proposed

DNN model achieves a validation accuracy of around

98.37% for detecting the possible suspicious activi-

ties (Note: the labels were created on the basis of

statistical approach (dataset1; Figure 7)). The same

model gave an accuracy of 86.33% when some ran-

domness was introduced in the dataset (dataset2; Fig-

ure 8). Figure 2, presents the scenes with ships (a and

b) and without ships (c and d). It should be noted that,

Figure 1 and Figure 3 illustrates the ship detection and

accuracy results while Figures 7 - 8 presents the loss

We have choosen Faster RCNN due to its lower com-

plexity over YOLO and RCNN

we tried multiple models like VGG-16, VGG-19, In-

ception, Simple Convolutional models and we found that

VGG-16 outperformed VGG-19, Inception and Simple

Convolutional models for this data.

Detection and Identiﬁcation of Threat Potential of Ships using Satellite Images and AIS Data

695

(a) Scene 1

(b) Scene 2

(d) Scene 4

Figure 9: These ﬁgure represents ships at different locations.

and accuracies for threat detection for dataset1 and

dataset2 respectively. We would like to demonstrate

the proposed tool during the conference if accepted

4.3 Future Work

In addition to the current work, there are two more ap-

proaches that can be used in order to determine threat

more accurately.

4.3.1 Like for Like Comparison

Using the AIS data of ships that travelled through that

region, we can predict if the transmission gap/ blind

period is due to some network issue or the AIS trans-

mitter is turned off manually. Additionally if multiple

ships go dark in the same vicinity and in the same

timeframe, we can determine illegal ship to ship trad-

ing more accurately by processing the AIS data for all

The code is available at https://github.com/

akash-iitjammu/AIS-threat-monitoring.

those ships simultaneously.

4.3.2 Path Prediction

We can use RNNs to predict the possible path of a ves-

sel in its blind period using its trajectory before and

after the dark period, time difference and maximum

speed. Knowing the possible trajectory can help us

determine illegal port calls, ship to ship trading with

more precision.

4.3.3 Density based Clustering

We can use Density based clustering algorithms to

ﬁnd out regions with minimum to no transmissions at

all by all the ships in a particular timeframe. This will

help us identify regions with poor network connectiv-

ity, and then if a vessel went dark in one cluster and

then re-appears in another, we can take it as it passed

through a low connectivity zone and so can’t transmit,

and we can mark it safe i.e. not suspicious.

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

696

5 CONCLUSIONS

There are very few literatures available in this ﬁeld to

identify the potential threats during vessels movement

in ocean. There is a huge scope of developments and

improvements to identify the threat potential of the

Vessels. In most of the works, only AIS data is con-

sidered for monitoring the ships.

In this work, we have proposed a deep learning

based approach where, we are not just tracking the

vessels by its AIS data but also using satellite imaging

to detect the vessels. Satellite imaging gives us an

added advantage, as for in any region, we can know

the ships that passed through even if their AIS beacon

was turned off. This increases the reliability of our

approach. The results obtained show the applicability

of our proposed model in real-time.

REFERENCES

Chang, S. (2003). Vessel identiﬁcation and monitoring sys-

tems for maritime security. In IEEE 37th Annual 2003

International Carnahan Conference on Security Tech-

nology.

Chawla, N., Bowyer, K., Hall, L., and Kegelmeyer, W.

(2002). Smote: Synthetic minority over-sampling

technique. Journal of Artiﬁcial Intelligence Research,

16:321–357.

Connor, S. and Taghi, M. (2019). A survey on image data

augmentation for deep learning. Journal of Big Data,

6(60).

Garagic, D., Rhodes, B. J., Bomberger, N. A., and

Zandipour, M. (2009). Adaptive mixture-based neu-

ral network approach for higher-level fusion and au-

tomated behavior monitoring. In NATO Workshop on

Data Fusion and Anomaly Detection for Maritime Sit-

uational Awareness, La Spezia, Italy.

Hakola, V. (2020). Vessel tracking (ais), vessel metadata

and dirway datasets. In IEEE Dataport.

He, K., Gkioxari, G., Doll

ar, P., and Girshick, R. (2017).

Mask r-cnn. In IEEE International Conference on

Computer Vision (ICCV), pages 2980–2988.

I., H. (2018). Data augmentation by pairing samples for

images classiﬁcation. ArXiv e-prints.

JH, F., D, P., D, K., BD, H., U, R., and C, W. (2018). De-

tecting suspicious activities at sea based on anomalies

in automatic identiﬁcation systems transmissions. In

PLoS ONE 13(8): e0201640.

Karen, S. and Andrew, Z. (2015). Very deep convolutional

networks for large-scale image recognition. In Inter-

national Conference on Learning Representations.

Lane, R. O., Nevell, D. A., Hayward, S. D., and Beaney,

T. W. (2010). Maritime anomaly detection and threat

assessment. In 13th International Conference on In-

formation Fusion.

Liu, S. and Deng, W. (2015). Very deep convolutional

neural network based image classiﬁcation using small

training sample size. In 2015 3rd IAPR Asian Confer-

ence on Pattern Recognition (ACPR), pages 730–734.

Luis, P. and Jason, W. (2017). The effectiveness of data aug-

mentation in image classiﬁcation using deep learning.

Stanford University research report.

Ren, S., He, K., Girshick, R., and Sun, J. (2017). Faster r-

cnn: Towards real-time object detection with region

proposal networks. IEEE Transactions on Pattern

Analysis and Machine Intelligence, 39(6):1137–1149.

Rhammell (2018). Ships in satellite imagery https://www.

kaggle.com/rhammell/ships-in-satellite-imagery.

Swamidason, Joseph, I. T., Sasikala, J., and Juliet, S.

(2020). Detection of ship from satellite images us-

ing deep convolutional neural networks with improved

median ﬁlter. In Artiﬁcial Intelligence Techniques for

Satellite Image Analysis. Remote Sensing and Digital

Image Processing, vol 24. Springer.

Xie, X., Li, B., and Wei, X. (2020). Ship detection in multi-

spectral satellite images under complex environment.

In Remote Sens. 12, 792.

APPENDIX

The proposed model for ship detection based on

VGG-16 with detailed layer architecture is presented

in the ﬁgure (10) below. After detecting the ship,

Faster RCNN method is applied to get the bounding

box around the ships.

Detection and Identiﬁcation of Threat Potential of Ships using Satellite Images and AIS Data

697

Figure 10: Ship Detection Model Based on VGG-16.

VISAPP 2022 - 17th International Conference on Computer Vision Theory and Applications

698