Miniature Autonomy as One Important Testing Means in the

Development of Machine Learning Methods for Autonomous Driving:

How ML-based Autonomous Driving could be Realized on a 1:87 Scale

Tim Tiedemann, Jonas Fuhrmann, Sebastian Paulsen, Thorben Schnirpel, Nils Sch

onherr,

Bettina Buth and Stephan Pareigis

Department CS, University of Applied Sciences Hamburg, Berliner Tor 7, Hamburg, Germany

Keywords: Autonomy, Autonomous Driving, System Level Tests, Machine Learning, Mobile Robots.

Abstract:

In the current state of autonomous driving machine learning methods are dominating, especially for the envi-

ronment recognition. For such solutions, the reliability and the robustness is a critical question.

A “miniature autonomy” with model vehicles at a small scale could be beneﬁcial for different reasons. Ex-

amples are (1) the testability of dangerous and close-to-crash edge cases, (2) the possibility to test potentially

dangerous concepts as end-to-end learning or combined inference and learning phases, (3) the need to opti-

mize algorithms thoroughly, and (4) a potential reduction of test mile counts.

Presented is the motivation for miniature autonomy and a discussion of testing of machine learning meth-

ods. Finally, two currently set up platforms including one with an FPGA-based TPU for ML acceleration are

described.

1 INTRODUCTION

Autonomous driving (AD) is a research topic for

several decades now. The dominating methods for

the subtask of environment recognition changed from

classical pattern recognition to methods using ma-

chine learning (ML) and in the past years speciﬁcally

to proposals based on deep learning (DL). Especially

with such methods it is often not clear how robust

they will work if rolled out on the streets. New so-

lutions/methods can be tested at ﬁrst in simulations

and/or with recorded data, followed by ﬁrst real world

tests on non-public test tracks, and ﬁnally on pub-

lic streets. This can be done component-wise (soft-

ware, electronics, mechanical components), for a set

of components, or for a whole system.

However, AD is a complex problem, especially as

the environmental variations are huge. The test of

new methods needs to cover a large number of dif-

ferent trafﬁc situations to ensure proper functioning

when brought to market. Therefore, often malfunc-

tioning is recognized after thousands of test miles in

public trafﬁc. For a safety critical system that is not

acceptable according to the relevant standards, e.g.

ISO 26262. Thus, further means to ensure the overall

system safety are required.

One further testing means in addition to simula-

tions and real-world tests are tests using model vehi-

cles. Such tests are also carried out in competitions,

e.g., the Carolo Cup for vehicles on a 1:10 scale (Zug

et al., 2014). We propose an even more miniaturized

setup with vehicles on a scale of about 1:87. This

has some advantages: One point is that at this scale

there are several off-the-shelf components available

for vehicles and for the environment, as it is used for

model railways. Moreover, existing model railway se-

tups can be used for testing. For example, we use

the Miniatur Wunderland in Germany for tests (Wun-

derland, 2019). Another important advantage are the

high demands of a more miniaturized vehicle: by

this additional challenge not only electronics and me-

chanics need to be optimized but also the algorithms

used for advanced driver assistance systems (ADAS)

and autonomous driving (AD). Especially ML-based

methods which might be used on standard PC hard-

ware without thinking about, need to be checked care-

fully and revised when run on a miniature autonomy

setup.

The option of using model vehicles instead of real

cars or simulations gives some interesting advantages:

• First physical tests can be carried out before ﬁn-

ishing or even starting vehicle design.

• Unusual or even dangerous i.e. close-to-crash sit-

Tiedemann, T., Fuhrmann, J., Paulsen, S., Schnirpel, T., Schönherr, N., Buth, B. and Pareigis, S.

Miniature Autonomy as One Important Testing Means in the Development of Machine Learning Methods for Autonomous Driving: How ML-based Autonomous Driving could be Realized on

a 1:87 Scale.

DOI: 10.5220/0007955704830488

In Proceedings of the 16th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2019), pages 483-488

ISBN: 978-989-758-380-3

483

uations can be tested (edge cases).

• The differences between model setup and model

environment on the one hand and the real vehi-

cle, sensors, and environment on the other hand

are actually not an issue. This is, because the

lower level sensor processing (object detection,

range measurement, etc.) needs to be tested sep-

arately, anyways. One more critical problem in

many close-to-crash situations or accidents is the

higher layer where, e.g., sensor readings of differ-

ent modalities are fused, and/or sensor readings

are interpreted. Tests of these higher levels are

supposed to be performable on a miniature scale,

too.

• To test ML-based solutions that combine learning

and inference/recall in the application on the vehi-

cle, miniature models are preferable. The effect of

learning on a vehicle, after roll-out is hard to esti-

mate and even if combined with additional safety

means this can be a dangerous step on real vehi-

cles.

• Finally, running many miniature model vehicles

in a miniature environment 24/7 might be a means

to reduce the number of test miles needed later on

the real vehicles.

This paper presents ﬁrst ideas for such a testing

concept. The following section discusses general as-

pects of testing autonomous driving solutions using

ML-based algorithms with regard to currently valid

standards. The currently employed hardware plat-

forms are described in section 3. Speciﬁc aspects re-

lating to the FPGA-based ML acceleration are pre-

sented in section 4. The paper closes with considera-

tions concerning the next steps and a short summary.

2 TESTING OF MACHINE

LEARNING BASED

AUTONOMOUS DRIVING

METHODS

As, e.g., Koopman et al. point out “it is more impor-

tant than ever to understand the gaps between theory

and practice in automotive computer-based system

safety” (Koopman, 2018). According to ISO 26262

(see, e.g., (Gebhardt et al., 2013)) design, veriﬁca-

tion and validation activities should relate to the Auto-

motive Safety Integrity Level (ASIL). Typical valida-

tion acitivities comprise hardware-software integra-

tion tests, system integration tests, system tests. All

of these levels need to take into account the respec-

tive functional safety requirements as well as func-

tional correctness and general dependability require-

ments. Typically, test approaches include analysis

techniques as well as dynamic tests. Complex sys-

tems require a well-deﬁned test concept, which of-

ten includes Hardware-In-The-Loop (HIL) as well as

Software-In-the-Loop (SIL) phases. A general deci-

sion is whether dynamic tests should be performed on

the target computer or whether an emulator or simu-

lator provides a sufﬁciently realistic environment for

the components to be tested. The use of emulators

or simulators may require additional qualiﬁcation of

the software tools employed. For each requirement,

it needs to be decided whether the tests performed in

the simulation environment need to be rerun on the ﬁ-

nal target hardware (see, e.g., (Rierson, 2013) for tool

qualiﬁcation in the aviation context of DO-178C). It

is generally acknowledged that some types of errors

can not be detected in a simulation.

For autonomous driving, the usual assumption that

low-level components are individually tested, may not

hold. AI-based learning methods are not suitable

for traditional testing, e.g., typical requirements relat-

ing to code coverage or even requirements coverage

are not easily applicable for the resulting neural net-

works.

Typical test approaches e.g. for obstacle detection

or in general picture recognition divide the available

input data into training and test sets. The training set

is used to learn the respective data and the pictures

from the test set are then used to validate the learn-

ing success. The data acquisition thus is a critical part

for the validity of the resulting algorithm. There re-

mains an uncertainty with regard to situations not rep-

resented in the original training or test data.

System level tests on the other hand may not be

feasible in a realistic setting – i.e. on the road –

due to general restrictions for non-certiﬁed vehicles.

The current approach comprises prototypical situa-

tions due to driving in non-critical areas for as long

as possible.

We propose a testing concept which exceeds sim-

ple simulation, but allows for a far more controlled

environment conﬁguration than the ﬁnal prototype by

using miniature vehicles. This test level should be ad-

ditional to the usual tests of components, integration

and systems tests and should focus on critical situ-

ations, which are not easily reproducible in realistic

settings. Similar to the use of emulators or simulators

we expect additional efforts for the validation of the

miniature vehicles. The beneﬁt for this additional ef-

fort should be a very controlled setting especially for

robustness tests and fault injections, which may not

be possible for the ﬁnal system itself. This approach

requires a thorough risk analysis approach especially

ICINCO 2019 - 16th International Conference on Informatics in Control, Automation and Robotics

484

for those scenarios relating to safety critical situations

due to autonomous decisions. Ideally this analysis

would also provide additional test cases for compo-

nent tests - but this is not the primary objective.

3 MINIATURE VEHICLES

In the computer science department of HAW Ham-

burg various autonomous model vehicles have been

developed and built for use in research and teaching

in the last 10 to 15 years. In particular, students of

HAW have been participating in students competi-

tion Carolo-Cup which is held each February at TU

Braunschweig (see for example (Nikolov, 2009) and

(Braunschweig, 2019)). Model vehicles of scale 1:10

had to be constructed for this competiton. The minia-

ture vehicles (scale smaller than 1:10) described in

this article have been developed based on the follow-

ing preliminary requirements:

• explore miniaturization of autonomous systems

• develop small and cheap autonomous systems for

teaching and further research

• be able to use Miniatur Wunderland Hamburg as

autonomous driving test area (Wunderland, 2019)

• exploit and study miniaturization of machine

learning hardware

In this article two different miniature autonomous ve-

hicles shall be de described.

3.1 Autonomous Vehicle Type 1: Sedan

Car

The type 1 vehicle is designed as a sedan car on a

scale of 1:63 (see Figure 1, left). It is equipped with

2 degrees of freedom (parallelogram steering), a cam-

era, an ESP32 micro controller board and a battery

with charging circuit.

A custom design of the vehicle frame, chassis and

parallel steering was made and printed out on a 3D

printer. As a result, the development of a precise

steering using a linear servo and a precise drive using

a micro transmission was possible. This overcame the

downside of electric toy cars with their for this pur-

pose insufﬁcient mechanics.

The current control algorithms cover lane detec-

tion, lane following, and obstacle detection in a clas-

sical, image based non-ML-based fashion. A wide

angle lens for the camera is used to enlarge the visi-

ble area of the camera. Images are transformed into

world coordinates using a dynamic region of inter-

est and ﬁxed point calculation to reduce the necessary

computational power. Lane detection is performed in

the transformed image using scan lines and intelligent

search algorithms. Obstacle detection is included in

the method. Lane following is achieved using a pure

pursuit algorithm (Coulter, 1992) (Nikolov, 2009).

The vehicle drives autonomously on a parcours simi-

lar to the one used at the student competition for au-

tonomous model cars in scale 1:10 Carolo-Cup at TU

Braunschweig (Braunschweig, 2019). The roadway

consists of a black ﬂoor which has white road mark-

ing attached. Battery lasts for about 9 minutes.

A telemetry application has been developed which

is used for easy remote access via WLAN to the vehi-

cle. In particular this is currently used for parameter

tuning and observation of the camera image.

Experiments in Miniatur Wunderland have not yet

been made at this point.

In the future, image data could be collected via the

telemetry application to feed off-site ML algorithms.

It is planned to recall the object recognition obtained

in this way on the miniature truck (vehicle type 2)

which contains an FPGA-based ML acceleration for

this purpose.

3.2 Autonomous Vehicle Type 2: Truck

For the computation of ML-based algorithms hard-

ware acceleration and, thus, more space is needed.

For that reason a second platform (“type 2”) is de-

signed which corresponds to a truck with trailer on

a scale of 1:87 (see Figure 1, right). Also for this

platform, as testing environment the commercial 1:87

scale model railroad Miniatur Wunderland (1,490 m

layout size, see Figure 2) is planned to be used (Wun-

derland, 2019).

The “truck” platform consists of an off-the-shelf

1:87-scale truck chassis and a 3D-printed body. It will

be equipped with a custom printed circuit board carry-

ing a Xilinx Zynq 7030 FPGA. This FPGA provides

enough computational power to implement machine

learning methods, at least in a limited way (see sec-

tion 4).

Planned sensors on the truck platform 2 are a cam-

era, multiple distance sensors Sharp GP2Y03E and an

Inertial Measurement Unit (IMU) MPU-9255. To be

able to relate sensor performances on the miniature

vehicle to typical sensor performances on real-world

vehicles and to estimate performances on real vehi-

cles, comparisons and calibrations are needed. For

the IMU a method proposed by Tedaldi et al. is used

and compared with other approaches (Tedaldi et al.,

2014). The small IMU in the vehicle is compared with

a XSens MTi-300 AHRS.

Miniature Autonomy as One Important Testing Means in the Development of Machine Learning Methods for Autonomous Driving: How

ML-based Autonomous Driving could be Realized on a 1:87 Scale

485

Figure 1: Left: Autonomous miniature vehicle type 1 (“sedan”, small), right: Autonomous miniature vehicle type 2 (“truck”,

large). The latter is designed to carry an FPGA board while the former is controlled by an ESP32 micro controller.

Figure 2: Sample view on the Miniatur Wunderland model

railroad on a scale of 1:87. Parts of the 1,490 m

lay-

out could be used for tests (Wunderland, 2019; Tiedemann,

2019, under CC-BY 4.0).

Figure 3: On the miniature “truck” vehicle a custom FPGA

board with a Xilinx Zynq 7030 will be used to enable the

application of ML methods on the vehicle. The board is

currently planned with a size of approximately 38 mm ×

84 mm. Off-the-shelf boards can not be used because of

space limitations. While the ﬁrst ML tests did not use

FPGA-external memory (see section 4) the custom board

will include DDR memory (size not ﬁxed yet).

4 FPGA-BASED ML

ACCELERATION

For the computation of ML-based algorithms on mo-

bile compact and power critical hardware an opti-

mized hardware design is needed. While standard

micro controllers have a very low power consump-

tion and a high single-thread performance, their com-

putational performance on parallel/SIMD tasks like

needed in machine learning is rather poor (Schenck

et al., 2017). A low-power FPGA with an optimized

design can lead to a much better computational per-

formance with an acceptable power consumption.

To support different types of ML methods a gen-

eral design is advantageous. One proposal was pub-

lished by Google and called “Tensor Processing Unit”

(TPU) (Jouppi et al., 2017). A comparable de-

sign on a Zynq FPGA was proposed by Fuhrmann

(Fuhrmann, 2018) and tested on a commercial off-the-

shelf board.

The test application for the ﬁrst implementation

was the MNIST data set of handwritten digits (Le-

Cun et al., 1998; Lecun et al., ). The computation

of small 14×14 matrices on the FPGA (without using

FPGA-external memory) was about ﬁve times com-

pared to a computation on a Intel Core i5-5287U –

however, with reduced precision (Fuhrmann, 2018).

If the resulting additional error is acceptable is highly

dependent on the application and needs to be studied.

Therefore, the next step is a test setup that uses the

FPGA-based ML accelerator to classify street signs.

This is the milestone for a ﬁrst ML application on a

1:87 miniature vehicle.

ICINCO 2019 - 16th International Conference on Informatics in Control, Automation and Robotics

486

5 CURRENT STATE AND NEXT

STEPS

Currently, the sedan setup is ﬁnalized and for the

truck a custom FPGA board is designed. The sen-

sors are being calibrated, starting with the IMU and

with the method proposed by Tedaldi et al. (Tedaldi

et al., 2014). Figure 4 shows the IMU calibration

setup including data logging system and power sup-

ply to be able to carry out different movements with-

out the need for any external cables. For the Sharp

distance sensors an evaluation with a Lidar-generated

map is prepared.

Figure 4: IMU test frame. The orange block is the XSens

reference IMU, the MPU-9255 device under test IMU is

mounted on the small blue printed circuit board.

When the setup has been sufﬁciently completed, a

feasibility study for using the FPGA-based ML ac-

celerator in combination with a vehicle is planned.

This will include further considerations regarding the

test of trafﬁc sign detection both for a hard-wired net

which has been trained beforehand and a version ac-

tively learning during and improving during the test

phase. This phase should provide insight, e.g., in the

ability of the setup to react to exceptional situations

not provided during the learning phase as well as in

the beneﬁts of capturing exceptional situations during

the driving phase in comparison to simulation-based

generated scenarios. We assume that even in a minia-

ture setup, e.g., the manipulation of light, contrast,

partially hidden signs will be much easier than in sim-

ulations. Furtheron, we will consider if the setup

allows for an automatic optimization of driving be-

haviour under aggrevated circumstances such as take-

over or driving curves in different weather conditions.

6 CONCLUSIONS

There is indication that the use of miniature model

vehicles is a helpful testing means for ML-based au-

tonomous driving solutions – in addition to real-world

tests and simulations. Using miniature model vehi-

cles on a 1:87 scale has some speciﬁc advantages over

other common scales, either directly (using off-the-

shelf components) or indirectly (by the need to opti-

mize methods thoroughly to ﬁt to the demanding con-

straints).

First steps were taken to set up a ﬂeet of minia-

ture vehicles that could serve as testing platform for

– speciﬁcally ML-based – methods to facilitate au-

tonomous driving. Further implementation and tests

will be carried out to evaluate if miniature autonomy

can be realized and where it is applicable.

REFERENCES

Braunschweig, T. (2019). https://wiki.ifr.ing.tu-

bs.de/carolocup/.

Coulter, R. C. (1992). Implementation of the Pure Pur-

suit Tracking Algorithm. Robotics Institute, Carnegie

Mellon University.

Fuhrmann, J. (2018). Implementierung einer Tensor Pro-

cessing Unit mit dem Fokus auf Embedded Systems

und das Internet of Things. Bachelor Thesis.

Gebhardt, V., Rieger, G. M., Mottok, J., and Gießelbach, C.

(2013). Funktionale Sicherheit nach ISO 26262 - ein

Praxisleitfaden zur Umsetzung. dpunkt.verlag.

Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal,

G., Bajwa, R., Bates, S., Bhatia, S., Boden, N.,

Borchers, A., Boyle, R., Cantin, P., Chao, C., Clark,

C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb,

B., Ghaemmaghami, T. V., Gottipati, R., Gulland, W.,

Hagmann, R., Ho, C. R., Hogberg, D., Hu, J., Hundt,

R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Ka-

plan, A., Khaitan, H., Killebrew, D., Koch, A., Ku-

mar, N., Lacy, S., Laudon, J., Law, J., Le, D., Leary,

C., Liu, Z., Lucke, K., Lundin, A., MacKean, G.,

Maggiore, A., Mahony, M., Miller, K., Nagarajan, R.,

Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omer-

nick, M., Penukonda, N., Phelps, A., Ross, J., Ross,

M., Salek, A., Samadiani, E., Severn, C., Sizikov, G.,

Snelham, M., Souter, J., Steinberg, D., Swing, A.,

Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E.,

Vasudevan, V., Walter, R., Wang, W., Wilcox, E., and

Yoon, D. H. (2017). In-datacenter performance anal-

ysis of a tensor processing unit. In 2017 ACM/IEEE

44th Annual International Symposium on Computer

Architecture (ISCA), pages 1–12.

Koopman, P. (2018). Practical experience report: Automo-

tive safety practices vs. accepted principles. In Gal-

lina, B., Skavhaug, A., and Bitsch, F., editors, Com-

puter Safety, Reliability, and Security, pages 3–11,

Cham. Springer International Publishing.

Miniature Autonomy as One Important Testing Means in the Development of Machine Learning Methods for Autonomous Driving: How

ML-based Autonomous Driving could be Realized on a 1:87 Scale

487

LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.

(1998). Gradient-based Learning Applied to Doc-

ument Recognition. Proceedings of the IEEE,

86(11):2278–2324.

Lecun, Y., Cortes, C., and Burges, C. J. C. THE

MNIST DATA-BASE of handwritten digits. website,

http://yann.lecun.com/exdb/mnist/.

Nikolov, I. (2009). Verfahren zur Fahrbahnverfolgung

eines autonomen Fahrzeugs mittels Pure Pursuit und

Follow-the-carrot. Bachelor Thesis.

Rierson, L. (2013). Developing Safety-critical Software - A

Practical Guide for Aviation Software and DO-178C

Compliance. CRC Press.

Schenck, W., Horst, M., Tiedemann, T., Gaulik, S., and

oller, R. (2017). Comparing parallel hardware ar-

chitectures for visually guided robot navigation. Con-

currency and Computation: Practice and Experience,

29(4):e3833. e3833 cpe.3833.

Tedaldi, D., Pretto, A., and Menegatti, E. (2014). A Ro-

bust and Easy to Implement Method for IMU Cali-

bration without External Equipments. In 2014 IEEE

International Conference on Robotics and Automation

(ICRA), pages 3042–3049.

Tiedemann, T. (2019). Part of the Miniatur

Wunderland Hamburg. ﬁgshare. doi:

https://doi.org/10.6084/m9.ﬁgshare.8231198.

Wunderland, M. (2019). Miniatur wunderland. (website),

https://www.miniatur-wunderland.com.

Zug, S., Steup, C., Scholle, J., Berger, C., Landsiedel,

O., Schuldt, F., Rieken, J., Matthaei, R., and Form,

T. (2014). Technical Evaluation of the Carolo-Cup

2014 - A Competition for Self-Driving Miniature

Cars. In 2014 IEEE International Symposium on

Robotic and Sensors Environments (ROSE) Proceed-

ings, pages 100–105.

ICINCO 2019 - 16th International Conference on Informatics in Control, Automation and Robotics

488