value we could expect to be the hardware limit,
about 102 MiB/s. Below this block size, we observe
a progressive drop of performance, which can be
explained as the duration of the hardware I/O
operation which takes a fixed amount of time
independently of the amount of data being
transferred.
At the same time, we notice that the buffering
operation, with bigger data blocks, incurs a penalty
of about 3 MiB/s, which corresponds to about 4% of
the total bandwidth.
3.3 Experiment 2: Storage/var Data
Size
The second measurement performed on the host
machine “storage” was performed by holding the
block size to a very large value, 1 MiB and varying
the total data size. The purpose of this measurement
was to examine the point at which the result
stabilizes and the effect of CPU caches and
scheduler non-determinism is no longer dominant.
Results are reported in Table 2 and analyzed in
Figure 4.
Table 2: Results for read/write measurement on native
host “storage”, with and without the O_DIRECT flag, with
fixed block size of 1MiB and variable block size.
Size
(B)
Write1MiBblock
Write1MiBblock
withO_DIRECT
Read1MiBblock
withO_DIRECT
Time
(s)
Speed
(MiB/s)
Time
(s)
Speed
(MiB/s)
Time
(s)
Speed
(MiB/s)
1Mi 0,03 37,3 0,03 37,8 0,04 26,8
2Mi 0,03 67,7 0,03 67,8 0,02 83,5
4Mi 0,06 62,0 0,05 83,9 0,04 109,6
8Mi 0,11 73,3 0,10 79,6 0,08 106,0
16Mi 0,21 76,9 0,17 92,0 0,16 102,6
32Mi 0,42 76,4 0,34 93,5 0,30 106,4
64Mi 0,81 78,9 0,67 96,1 0,61 105,5
128Mi 1,57 81,6 1,29 98,9 1,26 101,4
256Mi 2,84 90,0 2,57 99,6 2,51 101,9
512Mi 5,44 94,1 5,08 100,8 5,01 102,2
1Gi 10,53 97,3 10,07 101,7 10,01 102,3
2Gi 20,57 99,6 20,10 101,9 20,03 102,2
4Gi 40,59 100,9 40,30 101,6 40,04 102,3
8Gi 80,92 101,2 80,44 101,8 80,13 102,2
16Gi 162,84 100,6 162,38 100,9 161,44 101,5
It is easy to notice that the noise is big with
smaller amounts of data: this is obviously the effect
of scheduling, context switches, and other
background activities occurring at the operating
system level. The obtained values tend to converge
at bigger amounts of data and the value selected for
the subsequent experiments was 1 GiB, which is a
good compromise between data size and stability of
the measurement.
Figure 4: Chart for read/write measurement on native host
with fixed block size relatively to Table 2.
3.4 Experiment 3: Centos1
After building solid frame of reference for the
expected results, it is time to proceed with
measuring the I/O performance of the
paravirtualized installations. The “centos1” VM was
freshly restarted, which is the one with the file
image storage. The file is saved on the file system of
the host OS, which is running an ext4 partition. The
measurement was executed with read/write
operations on the empty partition /dev/xvda2
using the same dd command as before, with variable
block size and fixed data size of 1 GiB. The results
are reported in Figure 5.
Figure 5: Chart for results of experiment 3, machine
“centos1”, read/write on file image based storage in
comparison with the host OS results from experiment 1.
We note that the buffered write operations reach
the maximum average speed even for small block
sizes (1 KiB), while the read/write operations with
O_DIRECT flag reach fast speed with very large
block sizes (about 1 MiB), but pay a very high
overhead price for each I/O operation.
3.5 Experiment 4: Centos2
After experiment 3, the VM “centos1” was shut
down and the VM “centos2” was started up, and the
same measurement was repeated.
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
80,0
90,0
100,0
110,0
120,0
1Mi 2Mi 4Mi 8Mi 16Mi 32Mi 64Mi 128Mi256Mi512Mi 1Gi 2Gi 4Gi 8Gi 16Gi
Speed(MiB/s)
Filesize (B)
Read1MiBblockwith O_DIRECT
Write1MiBblockwith O_DIRECT
Write1MiBblock
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
80,0
90,0
100,0
110,0
120,0
512 1ki 2ki 4ki 8ki 16ki 32ki 64ki 128ki 256ki 512ki 1Mi
Speed(MiB/s)
Block size (B)
Write1GBwith O_DIRECT
Nativ e write with O_DIRECT
Read1GBwith O_DIRECT
Nativ e write
Write1GB
CLOSER 2011 - International Conference on Cloud Computing and Services Science
660