SSD Performance

#1

Hello all,

I did some testing and found a very poor SSD performance. Just to compare:

Scaleway: Total transferred 1.976Gb (6.7444Mb/sec)
Digital Ocean (512Mb): Total transferred 8.6853Gb (29.644Mb/sec)

I used sysbench for testing with:

sysbench --test=fileio --file-total-size=2G prepare
sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run

Could someone please confirm?

Thanks.

1 Like
#2

My results on Scaleway:

Operations performed:  62898 Read, 41932 Write, 134144 Other = 238974 Total
Read 982.78Mb  Written 655.19Mb  Total transferred 1.5996Gb  (5.4599Mb/sec)
  349.43 Requests/sec executed

on RunAbove (sandbox with local ssd):

perations performed:  49140 Read, 32760 Write, 104772 Other = 186672 Total
Read 767.81Mb  Written 511.88Mb  Total transferred 1.2497Gb  (4.2655Mb/sec)
  272.99 Requests/sec executed
#3

Thanks Maarten!

Other tests:

  1. dd bs=1M count=2048 if=/dev/zero of=test conv=fdatasync

Digital Ocean (512Mb)
2147483648 bytes (2.1 GB) copied, 10.9867 s, 195 MB/s

Scaleway
2147483648 bytes (2.1 GB) copied, 23.7442 s, 90.4 MB/s

2.a) bonnie++ -d /tmp -r 512 -n 10 -u nobody (Digital Ocean 512Mb)
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xxx.itlabs. 1G 465 99 131301 39 106207 43 1010 99 547657 76 +++++ +++
Latency 34128us 298ms 200ms 19720us 21629us 8598us

2.b) bonnie++ -d /tmp -r 2048 -n 10 -u nobody (Scaleway)
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xxx-itlabs 4G 124 99 89372 41 40443 22 448 99 94513 19 5925 162
Latency 64782us 398ms 236ms 19268us 43166us 5866us

Would be interesting if someone from Scaleway could talk about this SSD results.

#4

@itlabs_eu you’re welcome. I seem to have run the RunAbove test on the wrong machine by accident. It was ran on one with distributed network storage. So I re-ran it on the Sandbox plan

Operations performed:  21060 Read, 14040 Write, 44842 Other = 79942 Total
Read 329.06Mb  Written 219.38Mb  Total transferred 548.44Mb  (1.828Mb/sec)
  116.99 Requests/sec executed

For a weird reason it is worse maybe due the fact that this one has no dedicated resources.

EDIT same for the dd test 2147483648 bytes (2.1 GB) copied, 43.043 s, 49.9 MB/s

#5

Maybe this helps: https://www.scaleway.com/faq/server/#-How-does-the-SSD-storage-perform

#6

The SSD storage is still pretty slow.

#7

Theoretically they could be using the Samsung PM1633a SSD which is 16Tb. Divide that up into 50Gb slices and you could have up to 320 C1’s all sharing the same SSD.

For me, dedicated hardware loses pretty much all advantages when the SSD is shared too much.

1 Like
#8

Hi Guys,

On the C1, sequential IOs are limited by the network link. The advantage of SSDs is essentially ramdom IOPS which is the usual bottleneck of servers.
This thread was initially about the C1 SSD performance, the C2 and VPS cloud servers do not have the same performances. :wink:

The VPS is able to go up to 60K IOPS and the C2 servers should have the same kind of performances.

#9

Interesting !

Do you know any tools on GNU/Linux to mesure IOPS ?

#10

Hi @Angristan,

We usually use fio for SSD benchmarking purpose.
The iodepth and numjobs parameter need to be increased to see the full potential of SSD devices.

1 Like
#11

Hi,

I have just spotted this thread after I created an issue on GitHub:

Putting benchmarks apart, in this real life example, you can see that disk performance on VPS is much better than on C2 servers.

Has anyone else experienced this? Can you relate? Is there anything that can be done?

#12

I came here looking for reproducible disk performance benchmarks of C2 - Do you still see these performance differences?

It would be bad if the underlying issue were the deployed SOCs. C1 is not great on IO either…

#13

This is not expected, we will investigate to find the root cause.

Yann

#14

So, I’ve gone and got myself a C2, because… its nice and such, but the disk performance is still severely underwhelming. I did my usual disk performance test, though mind that I only test random access:

  • 1 thread : 7.7707MB/sec
  • 2 threads: 19.265MB/sec
  • 4 threads: 24.604MB/sec

Some numbers for comparison:

  • Scaleway C1 - 1 thread: 5.4528MB/sec
  • Scaleway C1 - 2 threads: 10.22MB/sec
  • Scaleway C1 - 4 threads: 16.801MB/sec
  • Competitor virtual server - 1 thread: 301.91MB/sec
  • Competitor virtual server - 2 threads: 472.8MB/sec (Dual-core VM, so no 4 thread test done)

As you can see, the disk performance is only insignificantly better than C1. It’s really really bad.

This is my test script (you can get sysbench from apt and you should run the script in an empty directory):

#!/bin/bash
FSIZE=2G
OPTS="--test=fileio --file-total-size=$FSIZE --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0"
PREPARE_OPTS="--test=fileio --file-total-size=$FSIZE"

(time sysbench $PREPARE_OPTS prepare) 2>&1 | tee bench.0.log
sysbench $OPTS --num-threads=1 run | tee bench.1.log
(time sysbench $PREPARE_OPTS prepare) 2>&1 | tee -a bench.0.log
sysbench $OPTS --num-threads=2 run | tee bench.2.log
(time sysbench $PREPARE_OPTS prepare) 2>&1 | tee -a bench.0.log
sysbench $OPTS --num-threads=4 run | tee bench.4.log
#15

@yann Any news here? I still see the same behaviour. I did also try to find out if it could stem from the OS configuration (Your default Ubuntu Xenial). I could not find anything bad, though I’m not too well versed in Linux Kernel and network stack internals.

At any rate, this behaviour is basically a blocker for serious usage.

#16

@yann are there any news?

#17

@e14,

Our engineering team is still investigating to understand what’s going on.

Here are our preliminary results:

  • dpkg has a specific IO pattern, it does a lot of fsync (See “Why is dpkg so slow when using new filesystems such as btrfs or ext4?” https://wiki.debian.org/Teams/Dpkg/FAQ)
  • we’ve tested with xfs and we’re not able to reproduce the issue, the issue seems to be triggered with the combination of ext4 + LSSD
  • we’ve analyzed the network IO going to the LSSD when using ext4 and at some point there is absolutely no IO activity during 200s

We’re still trying to understand why at some point the network activity stops.

Yann

1 Like
#18

@yann Maybe we can get OS images with XFS then, instead of ext? I would not really care one way or the other, Linux has to be able to mount it as /, but that’s about as far as my priorities go

#19

Hi,

I also noticed slow disk performance and here are my 2 cents:
I’m running for shorter time and with 32 threads and using ubuntu 16.04 with sysbench 0.4.12:

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrw --init-rng=on --max-time=60 --max-requests=0 --num-threads=32 run

VC1S in par1 zone:
Operations performed: 19727 Read, 13136 Write, 41788 Other = 74651 Total Read 308.23Mb Written 205.25Mb Total transferred 513.48Mb (8.5532Mb/sec) 547.41 Requests/sec executed

VC1S in ams1 zone:
Operations performed: 90711 Read, 60468 Write, 192834 Other = 344013 Total Read 1.3841Gb Written 944.81Mb Total transferred 2.3068Gb (39.366Mb/sec) 2519.40 Requests/sec executed

C2S in ams1 zone:
Operations performed: 84114 Read, 56078 Write, 177135 Other = 317327 Total Read 1.2835Gb Written 876.22Mb Total transferred 2.1392Gb (36.488Mb/sec) 2335.25 Requests/sec executed

Cheapest droplet in Digital Ocean in Amsterdam region:
Operations performed: 436923 Read, 291283 Write, 926615 Other = 1654821 Total Read 6.6669Gb Written 4.4446Gb Total transferred 11.112Gb (189.62Mb/sec) 12135.86 Requests/sec executed

So the conclusion - the disk speed is much slower then for competitors.
And even Scalawag speed in Paris is much slower than in Amsterdam.
I understand that this is just one kind of test sequential reads/writes could be different but nevertheless
it does not feel right.

Could you please provide more information how SSD disks are shared among the servers.

Would appreciate your input on the issue.

Jurgis

#20

I also tested on C2L in ams1 zone:

Operations performed: 61199 Read, 40801 Write, 128652 Other = 230652 Total Read 956.23Mb Written 637.52Mb Total transferred 1.5564Gb (26.562Mb/sec) 1699.96 Requests/sec executed

And Direct SSD:
Operations performed: 191845 Read, 127904 Write, 393750 Other = 713499 Total Read 2.9273Gb Written 1.9517Gb Total transferred 4.879Gb (83.254Mb/sec) 5328.23 Requests/sec executed

May Macbook Pro in comparison:
Operations performed: 1153955 Read, 769317 Write, 2459991 Other = 4383263 Total Read 17.608Gb Written 11.739Gb Total transferred 29.347Gb (500.85Mb/sec) 32054.24 Requests/sec executed