Use of a Virtual Twin for Dynamic Storage Space Monitoring in a

Port Terminal

Andreas Höpfner, Olaf Poenicke, Christian Blobner and André Winge

Fraunhofer IFF, Sandtorstr. 22, 39106 Magdeburg, Germany

Keywords: Virtual Twin, VR Port Model, LiDAR 3D Data, IoT, Port of the Future.

Abstract: The paper describes research and development for dynamic 3D models and Virtual Reality (VR) applications

in the context of port processes. The work described is carried out in the currently ongoing EU-funded project

PortForward. The paper addresses the use of VR technologies in the context of industrial applications and

describes how dynamic sensor data can be integrated into a 3D model of a Port. In addition to tracking data

of moving assets, the sensor data mainly comprises 3D measurement data from LiDAR sensors. These sensors

are installed in the port infrastructure to automatically record the current occupancy status of storage areas.

The measurement data from the LiDAR sensors are dynamically integrated into the VR model in an abstracted

form together process related meta-data to reflect the current process status in the port terminal. Based on that

approach in the sense of a Virtual Twin, process flows and storage space management can be optimized. The

use of VR technologies is crucial in this context in order to depict the complex spatial and dynamic process

situation in an intuitive way.

1 INTRODUCTION

Digitization and Industry 4.0 are buzzwords that pose

new challenges for companies and infrastructure

operators. The ability to process and store digital data

and to make decisions on the basis of digital data is

becoming an elementary component of companies to

ensure their competitiveness. This is all the more

important in logistics, which operates at the hubs of

the economy.

In the course of digitization, ports are faced with

the challenge of intuitively and perceptibly mapping

a wide variety of data and information on widely

distributed infrastructures, superstructures, operating

resources as well as freight and stored goods in an

integrated form. The PortForward project (2018-

2021), which is funded by the EU Commission within

the framework of Horizon 2020 and its specific pro-

gram Port of the Future, addresses these points and

challenges with regard to the digitization of port

infrastructure. The Fraunhofer IFF leads the project

and works with twelve European partners to develop

solutions and technologies for small and medium-

sized ports in Europe.

In the PortForward project, the IFF relies

primarily on its long-standing cooperation with the

Port of Magdeburg, which is the biggest inland port

in central Germany with an annual handling volume

of about four million tons. Here, preliminary projects

have already produced virtual models of the port area

for strategic infrastructure planning. Against this

background, the Fraunhofer IFF is developing a so-

called Virtual Twin (Adler and Masik, 2020) of the

Magdeburg inland port as part of PortForward in

order to integrate real-time information and

interaction possibilities into the virtual model.

A central use case in the project focussing on the

consistent use of 3D data is the so-called Dynamic

Storage Space Monitoring. For this purpose, an

approach is being developed that links the spatial

model of a port terminal with the dynamic

movements of logistics objects and the changing

space occupancy states. These are automatically and

cyclically recorded by LiDAR sensors integrated into

the port infrastructure. Thus real-time information

about the occupancy status of dedicated storage areas

and the shapes of stored objects will be available in

the Virtual Twin model to enable efficient yard

management for a multi-purpose use terminal.

Section 2 of the paper describes the Virtual Twin

of the port that is used as a basis for the use case.

Based on a static 3D model initially developed, it is

briefly described how information from sensor

sources are integrated to generate a dynamic 3D

model.

116

Höpfner, A., Poenicke, O., Blobner, C. and Winge, A.

Use of a Virtual Twin for Dynamic Storage Space Monitoring in a Port Terminal.

DOI: 10.5220/0010676800003062

In Proceedings of the 2nd International Conference on Innovative Intelligent Industrial Production and Logistics (IN4PL 2021), pages 116-122

ISBN: 978-989-758-535-7

Section 3 gives an overview on the specific

requirements in the use case in a multi-purpose use

terminal and describes the overall approach for the

development of the Dynamic Storage Space

Monitoring.

In section 4 the approach for 3D data capture

based on LiDAR sensors is described. The paper

closes with a short summary and outlook in section

2 VIRTUAL TWIN OF THE PORT

2.1 Objectives and Vision

The Virtual Twin approach is based on the concept of

the Digital Twin, which was first introduced in 2010

(Shafto et al., 2010). With the advent of new and

powerful Virtual Reality (VR) technologies, the focus

of this concept is also shifting to the virtual and

interactive representations of Digital Twin solutions

(see Schroeder et al., 2016).

The Fraunhofer IFF creates Virtual Twin

solutions by integrating different system models. For

example, Höpfner et al. describe the combination of a

spatial model with an energy model in a VR

application for strategic decision making in industrial

parks (Hoepfner et al., 2017). For the application in

the port environment, the focus is on the combination

of the spatial model with a logistics model of the port

to enable tactical and operational decision support for

the management of port processes. To allow

operational support, real-time information from the

physical port environment need to be integrated into

the 3D model – thus, making it a dynamic 3D model.

This dynamic Virtual Twin is also used for other use

cases along the Dynamic Storage Space Monitoring

within the project PortForward.

The use of 3D models and corresponding VR user

interfaces is seen as a central possibility to intuitively

display complex processes and states in their spatial

and partly also temporal context. In general, the

intuitive grasp of complex information and

knowledge transfer are seen as central advantages of

employing VR and AR solutions (see Adler and

Masik, 2020 and Reder, 2019). In addition, the

consistent use of 3D models in different life cycle

phases from planning to operational use in terms of

Digital Engineering opens up significant efficiency

potentials (Adler et al., 2015).

2.2 Development of the Basic Static

Model

As a basis for the dynamic Virtual Twin a static 3D

model of the Port of Magdeburg was generated using

basic geo data (digital terrain model DGM02 and

digital ortho photos DOP20) from the State Office for

Surveying and Geoinformatics LVermGeo Saxony-

Anhalt. The special requirements of the later model

application made further processing of this model

necessary. The resulting terrain geometry of the

previously regular DGM02 was transferred into a

volume and performance optimized irregular

triangular network TIN (Triangulated Irregular

Network). The terminal areas of the Port of

Magdeburg were manually integrated on the basis of

digital plans. Port infrastructure, such as quay walls

and locks, were integrated into the terrain model

using as-built plans of the Port of Magdeburg and on-

site photographs. Selected LoD 02 buildings of

special relevance were modeled with façade textures

(see figure 1). The result was a virtual 3D terrain

model of the Port of Magdeburg, which is suitable for

interactive work in real time.

Figure 1: 3D Terrain model TIN geometries (top); 3D

building models with LoD 2 (down-left); relevant buildings

with façade textures (down-right).

In a final modelling step, cityscape-defining

vegetation and secondary objects were manually

integrated into the virtual model using a 3D object

library. As shown in Figure 2, this also includes

logistics-related objects such as handling equipment

Use of a Virtual Twin for Dynamic Storage Space Monitoring in a Port Terminal

117

(e.g. cranes) or stored goods (e.g. ISO containers,

machinery components).

Figure 2: 3D model of the Hanse-Terminal with logistics

objects and marking of areas with different storage purpose

(right).

As also shown in Figure 2, individual sub-areas

are differentiated as these offer different capabilities

for handling and storage of goods (e.g. in terms of

loads and storage capacity or regulations for storable

types of goods). With such meta information assigned

to storage areas, the model is currently already used

operationally by the port for the administration of

storage area permits.

2.3 Development of the Dynamic Model

To establish a dynamic real-time model, additional

object information provided by heterogeneous

software systems are necessary. Collecting these

information via different special interfaces to the

software systems are not effective. Therefore, an

approach of using standard interface technologies and

protocols, like RESTful API and MQTT was

developed. In context to the Use Case of Dynamic

Storage Space Monitoring, as described in section 3,

an interface definition was developed that allows

classified logistic objects (containers, storage goods

etc.) and its meta-information to be defined and

transferred. In order to map these additional

information on objects in the 3D overall model, all

classified objects need a unique identifier. In addition,

an initial alignment of the coordinate systems

between the model world and reality must be

performed, e.g. to determine the reference point for

positioning the objects.

There are different strategies to update the virtual

model. On the one hand, the 3D model is able to pull

new information periodically. But this is not always

performant. On the other hand it is possible to trigger

the virtual model when information are provided. The

first prototype was implemented using an MQTT

based publish-subscribe mechanism. With a central

broker several input sources can be connected to the

3D model. The model itself fetches the relevant

dynamic object data referring to the currently used

features of the Virtual Twin (e.g. only fetch container

related data).

Figure 3: 3D model prototype with on-click annotations of

container data and search function.

The project uses different visualization

technologies to display the meta-information in the

scene. For example, status information such as

incoming logistics objects are coloured green. Other

information such as cargo number, content, sensor

data (temperature, humidity) etc. can be visualized as

on-click annotation on the objects. The developed

prototype of the dynamic 3D model also provides

features for searching objects – e.g. container by

container ID (figure 3).

The main advantage of using this dynamic real-

time model is the compact, clear, object-based

representation of meta-information from different

software systems in one scene.

3 DYNAMIC STORAGE SPACE

MONITORING

3.1 Use Case Requirements

In the Port of Magdeburg, individual port terminals

are used for the handling and storage of a wide variety

of goods (e.g. mixed use with palletised goods,

containers, heavy goods, etc.). As a result, it is not

possible to define fixed storage locations and mark

them in the terminal area, as is done, for example, in

pure container terminals with marking of storage

aisles and locations. This of course results in a high

IN4PL 2021 - 2nd International Conference on Innovative Intelligent Industrial Production and Logistics

118

complexity for documentation of storage locations

and high efforts for the yard management.

To enable a more efficient yard management a use

case was developed to implement and test a Dynamic

Storage Space Monitoring. The requirements towards

this use case were:

• to develop a technical approach to

automatically capture the position of stored

goods (without technical devices attached to

the goods) and

• to integrate the good’s information (meta

data and actual storage location) into the 3D

model of the Virtual Twin

Based on these requirements a technical approach

was developed, is currently implemented and will be

tested in a dedicated terminal of the port.

3.2 Approach of the Virtual Storage

Location Grid and Data Capture

To make use of the Virtual Twin the approach for a

Virtual Storage Location Grid was developed, based

on the spatial model of the port. With a grid

dimension of 2x2 meters, stored goods can be

assigned to individual virtual storage locations. Thus

occupied storage locations can be marked in the

dynamic 3D model (see figure 4).

Figure 4: 3D model of the Hanse-Terminal with Virtual

Storage Location Grid (occupied locations marked red).

This assignment requires the acquisition of

information about the goods and their location during

storage. In the project, this recording of meta data

shall be carried out with mobile devices with an

integrated GPS. For a dynamic storage space

management, however, this GPS-based position

acquisition is too imprecise, as there is always an

offset between the mobile device used by personnel

and the stored good itself.

In the focused Hanse-Terminal of the Port of

Magdeburg, a camera infrastructure for research

purposes already exists with which a so-called Virtual

Bird Eye View (Borstell et al., 2012) can be

generated. Based on differential image analyses, this

allows to record the occupancy of the individual

storage location grids. However, this camera based

capturing is only giving 2D images and no 3D

information to be integrated into the Virtual Twin.

In order to obtain 3D information on the current

occupancy of storage spaces for the Virtual Twin, the

project is working on an expansion of the sensor

infrastructure. Using LiDAR sensors installed in the

Hanse-Terminal, real contours can be recorded and

integrated into the 3D model of the port in an

abstracted form. Figure 5 summarizes the concept of

the Virtual Storage Location Grid and its status

changes based on sensor data and provided meta data

of stored freight.

The occupancy of individual grid locations shall

be detected by LiDAR sensors and a referring

processing of the sensor data. Based on the current

occupancy, it is thus possible to check where a

sufficiently large free storage area is available in the

terminal for a new cargo to be stored. During storage,

the identification of the freight and the metadata of

the freight are to be entered on site via a mobile

device (e.g. tablet computer).

Figure 5: Concept for the Dynamic Storage Space

Monitoring using a Virtual Grid.

In this way, the storage can be roughly localized

via GPS. Once the storage has been confirmed, the

data from the LiDAR system are be used to check

whether there has been a corresponding change in the

space allocation. Based on the size and position and

orientation of the novel detected object the position

of the stored freight can be corrected and the status of

the correspondingly occupied grid areas is changed in

the Virtual Storage Location Grid.

Use of a Virtual Twin for Dynamic Storage Space Monitoring in a Port Terminal

119

4 DATA CAPTURE IN BIGGER

YARDS

4.1 Related Work

The central aspect described in this paper is the

development of a novel approach for continuous

recording of 3D information in a logistics specific

environment using LiDAR sensors.

A classic approach of scanning 3D contours in

port environments is the use of industrial line lasers

(see Chun et al., 2004). Today there are also

commercial solutions and applications available to

automatically scan container stacks in automated or

semi-automated terminals (e.g. LASE, 2021). They

are creating the 3D information only in connection

with the movement of cranes – thus other handling

operations (e.g. freight handled by forklifts or

reachstackers) are not directly captured. Newly stored

freight is only scanned “by chance”. For that reason

such solutions are not suitable for the specific

requirements of a multi-purpose terminal as described

above.

Other fields of research and development are

focussing on continuous scanning of storage yards

with regard to the storage of bulk material (see Ou et

al., 2012). But these approaches using

videogrammetry and projector-contour scanning also

rely on moveable sensor infrastructure.

Based on the specific experience with the Virtual

Bird Eye View in the Hanse-Terminal of the Port of

Magdeburg, the focus for a sensing solution was

again laid on fixed sensor positions. For the purpose

of the use cases described above, LiDAR sensors

were identified as the most suitable sensor type to

continuously generate 3D information of a wide area

with no requirement of additional sensor movement.

4.2 Data Capture using LiDAR Sensors

Light Detection and Ranging (LiDAR) sensors

belong to the group of time-of-flight sensors that

determine distances by emitting light and measure

distances by measuring the time of flight of light

reflected from surfaces back to the sensor. Because

the LiDAR sensor emits a laser beam in a previously

known direction, it is possible to determine a 3D

coordinate in space relative to the position of the

sensor. Over time, this results in 3D point clouds that

can be used to create spatial images and to detect and

localize movements. LiDAR sensors are mainly used

in the context of autonomous driving because they

offer high reading ranges of >>100m and provide the

3D data comparatively quickly. First models are

available on the market even in the low-cost range of

<1,000€, which opens up new fields of application.

For the target application described above, the

MID-40 model available from the company Livox

offers a very good price-performance ratio. The

performance parameters analysed by Ortiz Arteaga et

al. show a very high accuracy of the sensor at ranges

up to more than 200m (compare Ortiz et al., 2019).

The special feature of the sensor is a non-repetitive

scanning pattern, which allows a higher coverage of

the Field of View (FOV) compared to line-based

LiDAR sensors (Livox, 2021a), which is especially

useful for the acquisition of 3D objects in port

terminals over a longer integration time.

Given the possibilities to install such sensors on

the light posts in the Hanse-Terminal and the required

coverage of the dedicated storage area the MID-40

sensor will be complemented by another LiDAR

model of Livox, offering a wider field of view (Livox,

2021b). Figure 6 shows exemplary test images taken

with the MID-40 LiDAR sensor in the Hanse-

Terminal of the Port of Magdeburg.

Figure 6: Test recordings with MID-40 LiDAR sensor –

enriched with RGB data (left) and with height coding

(right).

On the basis of the generated point clouds,

standard shapes and sizes (e.g. for ISO containers or

swap bodies) can be classified using AI methods.

Fraunhofer IFF already developed AI based

methodologies to classify objects based on dynamic

point cloud data in other industrial contexts. For point

cloud segmentation, object detection and tracking and

object classification several AI methodologies are

described in the literature (see Guo et al., 2020). For

the use case several approaches can be evaluated now

with the sensor installations completed.

Abstracted information of classified objects in the

port environment will be directly integrated into the

3D model of the port. Freight and containers with

shapes that cannot be directly classified are

transferred using a bounding box. For the transfer of

data on object type, size, position and orientation

from the measuring system to the 3D model the

IN4PL 2021 - 2nd International Conference on Innovative Intelligent Industrial Production and Logistics

120

Machine2Machine communication protocol MQTT

will be used as described above in section 2.3.

A simplified 3D model of the Hanse-Terminal is

furthermore used for planning the LiDAR

deployment. In particular, it can be used to check

which viewing and detection ranges are available for

the LiDAR sensors at different installation points and

orientations. Currently the pilot installation of

LiDAR sensors on the light posts in the Hanse-

Terminal was finalised. With that installation, a

defined storage area of approximately 2,000 m² is

covered. The installations were planned in such a way

that the storage area is detected from several sides to

be able to detect the stored objects from as many sides

as possible by matching the point clouds of the

individual LiDARs.

Figure 7: VR-based planning of sensor positions for the

dedicated storage area.

The VR-based planning of LiDAR positions

(figure 7) shows the potential for the consistent use of

3D models, since the 3D model data can be used both

for planning purposes and later for the operational use

of the sensor systems and corresponding evaluation

of the 3D measurement data. Furthermore the high

detail 3D model of the Virtual Twin can be used for a

sensor simulation of the LiDAR sensors to derive

synthetic 3D point clouds for virtual training of the

AI methods for object classification. After the

installation of the LiDAR sensors in the port

environment it will be evaluated whether such virtual

training can support the further development and

implementation of the object classification.

5 SUMMARY AND OUTLOOK

The PortForward project has so far developed the

technical concepts for several services, that can be

integrated into Virtual Twin applications. One of

these services is the Dynamic Storage Space

Monitoring based on real-time 3D information. The

use of the Virtual Storage Location Grid to document

the space occupancy at the Hanse-Terminal, which is

used for a wide variety of goods, can be used to tap

into potential for optimizing the management of the

storage areas. Furthermore, the use case shows the

potential of the underlying VR technologies for the

visualization and intuitive comprehension of complex

spatial and process relationships.

In the further course of the project, the technical

installations in the port will be carried out and the

functions for transferring the scan data into the 3D

model will be further developed and tested. On the

basis of a robust recording of the storage situation and

corresponding reproduction in the Virtual Twin,

targeted applications can subsequently be developed

which support the operative business, e.g. with regard

to storage area management or optimised storage

strategies.

ACKNOWLEDGEMENTS

The PortForward project is funded by the EU under

project number 769267 as part of the "Ports of the

Future" program: https://www.portforward-project.eu/

REFERENCES

Adler, S., Kernchen, A., Reipsch, T., Bayrhammer, E.,

Schmucker, U. (2015): Mobile Assistenzsysteme für

sicheren Betrieb und Wartung von Maschinen und

Anlagen. In: Gausemeier, J., Grafe, M., Meyer auf der

Heide, F. (eds.): Augmented & Virtual Reality in der

Produktentstehung, p 199. Hans Gieselmann Druck und

Medienhaus, Bielefeld.

Adler, S., Masik, S. (2020): Der digitale Zwilling für

virtuelle Fabrikplanung und -betrieb. In: Orsolits, H.,

Lackner, M. (eds.): Virtual Reality und Augmented

Reality in der Digitalen Produktion, pp. 191-215.

Springer Gabler.

Borstell, H., Plate, C., Richter, K. (2012): Virtuelle

Draufsicht für die bildbasierte Situationsanalyse.

Tagungsband InnoSecure 2012. VDE-Verlag, Berlin,

Offenbach.

Chun, T.-W., Kim K.-M., Lee, H.-G., Nho, E.-C. (2003):

Fast scanning method for container stacking profile

with one laser sensor. Industrial Electronics Society,

2003. IECON '03. The 29th Annual Conference of the

IEEE.

Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., Bennamoun,

M. (2020): Deep Learning for 3D Point Clouds: A

Survey. IEEE Transactions on Pattern Analysis and

Machine Intelligence.

Hoepfner, A., Mencke, N., Lombardi, P., Franke, R.,

Komarnicki, P. (2017): A Virtual Reality Platform that

supports integrated Design of Energy and Land-Use

Use of a Virtual Twin for Dynamic Storage Space Monitoring in a Port Terminal

121

Plans in Brownfield Industrial Parks. The Seventh

International Symposium on Energy. Manchester.

LASE (2021): Automatic Yard Crane, https://lase-

solutions.com/products/ports/automatic-yard-crane/

last accessed 2021/06/22.

Livox Technology (2021a): MID-40 lidar sensor.

https://www.livoxtech.com/mid-40-and-mid-100, last

accessed 2021/06/22.

Livox Technology (2021b): Horizon lidar sensor.

https://www.livoxtech.com/horizon, last accessed

2020/07/04.

Ortiz Arteaga, A., Scott, D., Boehm, J. (2019): Initial

Investigation of a low-cost automotive LiDAR system.

In: The International Archives of the Photogrammetry,

Remote Sensing and Spatial Information Sciences,

Volume XLII-2/W17.

Ou, J., Zhou, J., Zhu, X., Yuan, Y., Shang, Y., Zhang, X.

(2012): Large stack-yard three-dimensional

measurement based on videogrammetry and projected-

contour scanning. In: Optical Engineering, 51(6).

Springer.

Reder, B. (2019): Vom Science-Fiction-Film in die Realität.

In: IDG Research Services (eds.): Studie Virtual

Reality/ Augmented Reality 2019, p 14. IDG Business

Media, München (2019).

Schroeder, G. N., Steinmetz, C., Pereira, C. E., Espindola,

D. B. (2016): Digital Twin Data Modeling with

Automation ML and a Communication Methodology for

Data Exchange. In: IFAC-PapersOnLine – Volume 49,

Issue 30 (pp. 12-17). Laxenburg, Austria: International

Federation of Automatic Control IFAC.

Shafto, M., Conroy, M. E. G., Kemp, C., Le Moigne, J.,

Wang, L. (2010): Draft modelling, simulation,

information technology & processing roadmap,

Technology Area 11. Washington, DC: National

Aeronautics and Space Administration.

IN4PL 2021 - 2nd International Conference on Innovative Intelligent Industrial Production and Logistics

122