[PERF] DEV1-S + NVMe .... 20-40% IOWait !?!?!?

#1

First try with this instance.
Is it me or these instances are just slower than rotating drive ?
The only thing I’ve done… was creating it, and upgrading ubuntu… still running after 1 hour…

I can stand that I may have nasty noisy neighbour (10-20% steal)… but how the hell ‘some kind of’ SSD can’t even handle a few 100’d KB/sec average and even less tps.

Anyone experienced the same ?

 # iostat 60
Linux 4.4.0-134-generic (<censored>)    03/10/2020      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.40    0.00    6.48   20.22   16.97   50.93

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              33.23       156.90      1399.75     254161    2267489

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.49    0.00    1.24    3.93    5.33   78.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               2.52         0.00       115.11          0       6908

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.40    0.00    3.51    2.94   13.87   51.28

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               8.28         0.60      4878.73         36     292724

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.76    0.00    6.98   15.07   18.05   52.14

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              23.08         5.87      4329.07        352     259744

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          23.45    0.00    8.37    9.13   15.32   43.73

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              19.95       109.87      1198.07       6592      71884

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.41    0.00    7.32   32.31   11.97   43.99

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              46.89        24.66      1040.16       1480      62420

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.17    0.00    2.13   43.44    6.54   46.72

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              21.90         6.20       498.67        372      29920

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    1.00   47.15    3.07   48.28

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              12.72         0.93       195.80         56      11748

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.72    0.00    1.68   45.03    5.20   47.37

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              16.68         1.13       347.00         68      20820

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.76    0.00    9.02   27.18   22.33   38.71

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              49.37         2.87       959.47        172      57568

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.40    0.00    5.70   32.94   16.23   41.73

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              46.52         7.13      1115.27        428      66916

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.40    0.00    4.51   36.65   12.53   43.90

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              53.58         2.67      1376.53        160      82592

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.03    0.00   11.29   25.72   20.90   40.05

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              23.12         1.00       559.93         60      33596

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.81    0.00    6.70   28.98   15.73   41.77

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              22.50         0.73      1869.07         44     112144

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.66    0.00    3.09   40.83    8.04   46.39

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              39.39         2.33       987.27        140      59256

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.66    0.00    2.83   44.23    6.70   44.57

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              22.95         0.80       461.47         48      27688

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.58    0.00    3.52   40.42    9.96   44.53

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              32.02         1.07       748.80         64      44928

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.95    0.00    2.94   40.05    7.98   46.08

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              29.60         2.40       613.33        144      36800

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.71    0.00    1.50   45.10    4.91   47.78

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              30.92         1.93       532.93        116      31976

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.33    0.00    2.87   41.30    9.51   45.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              38.73         3.07       787.47        184      47248

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.75    0.00    2.00   44.14    6.50   46.61

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              29.33         0.93       484.67         56      29080

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.67    0.00    3.32   40.49    9.57   44.95

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              32.42         2.47       727.53        148      43652

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.32    0.00    3.36   41.08   10.00   44.23

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              35.63        10.93       706.73        656      42404

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.44    0.00    1.19   45.61    5.03   47.73

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              24.45         0.87       459.20         52      27552

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.38    0.00    4.00   41.41   12.42   40.79

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              10.62         0.60       705.27         36      42316

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.81    0.00    4.55   38.41   13.16   42.07

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              47.87         6.33      1037.07        380      62224

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.40    0.00    3.08   41.54    8.35   45.63

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              49.83        45.06       741.68       2704      44508

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.18    0.00    3.33   41.52    8.93   45.04

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              36.10        85.33       770.80       5120      46248

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.24    0.00    2.72   43.09    7.01   45.93

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              26.90        59.53       577.60       3572      34656

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.07    0.00   11.63   25.21   31.67   26.42

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              47.75       497.93      3559.40      29876     213564

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.85    0.00   12.77   23.29   35.30   23.78

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              58.80       902.27      3533.40      54136     212004

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.84    0.00   10.79   29.44   33.17   22.76

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              89.55       537.40      3172.33      32244     190340

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.92    0.00    7.05   40.17   19.97   30.90

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             259.90       174.80      2529.53      10488     151772

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.17    0.00    4.52   36.20   13.63   43.48

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              76.27        92.27      1931.53       5536     115892

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.99    0.00    4.11   37.61   10.76   44.53

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              45.40        68.00       629.73       4080      37784

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.70    0.00    6.18   30.73   13.79   43.60

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              40.47       394.87       569.60      23692      34176

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.70    0.00    9.43   23.95   23.55   35.37

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              28.85        91.60       363.93       5496      21836

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.99    0.00   19.02    7.48   42.31   22.20

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              20.22        69.47       109.13       4168       6548

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.67    0.00   15.61   11.99   31.84   32.89

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              18.50       118.07        93.13       7084       5588

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.28    0.00    1.72   17.20    3.92   74.88

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              14.23        52.73       199.47       3164      11968

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.00    0.00    8.12   29.10   17.18   40.60

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              33.58       212.00       380.40      12720      22824

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.17    0.00    2.29   42.50    5.77   47.27

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              49.63       114.13       837.80       6848      50268

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.00    0.00    5.39   35.89   10.43   45.29

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              62.55       505.56       456.00      30333      27360

================================================

For comparison, and OLD START1 with LSSD …

# iostat 10
Linux 4.9.0-8-amd64 (<censored>)        03/10/2020      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          95.59    0.00    1.02    0.00    0.20    3.18

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               0.51         1.58         4.46   43595832  122803020

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.64    0.00    0.61    0.30    0.20   94.25

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               1.11         0.81       608.68          8       6032

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          42.18    0.00   24.82   10.29    0.20   22.50

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             547.83      2936.02     15483.35      29096     153440

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          53.27    0.00   32.43   14.20    0.10    0.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             495.17      4406.85     19211.28      43760     190768

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          61.58    0.00   28.72    9.50    0.20    0.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             520.42      5341.56     19195.96      52828     189848

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          35.51    0.00   54.83    9.46    0.20    0.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             636.52     20750.96      6581.49     206264      65420

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          62.47    0.00   31.19    6.24    0.10    0.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             398.29      3790.74      3835.41      37680      38124

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.98    0.00   20.74   12.05    0.20   44.02

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             725.94      8292.54      6664.35      81184      65244

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.33    0.00    1.11    0.40    0.20   93.95

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               0.50         0.40        16.94          4        168

==========================================

Reboot … install gcc / git / cmake … Just can’t believe the numbers … (iostat 60)

Linux 4.15.0-88-generic (<censored>)  03/10/2020      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.90    0.00    8.16   14.56   22.27   45.11

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             142.85      3532.96      2395.87     231409     156929

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.77    0.00    0.87    2.91    2.93   89.52

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               3.97        46.67      1239.73       2800      74384

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.97    0.00    0.67   15.36    1.81   80.19

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              21.72       427.20        61.80      25632       3708

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.80    0.00    0.08    1.65    0.49   96.98

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               1.63         0.00       617.53          0      37052

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.47    0.00    4.11   32.80   12.76   45.87

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              49.55        22.20      2414.40       1332     144864

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.85    0.00    6.13   28.57   20.51   39.94

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda             108.80         6.73      3057.93        404     183476

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.05    0.00    3.35   39.28   10.70   43.62

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              65.43        28.20      1008.73       1692      60524

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.33    0.00   17.26   10.89   32.85   29.67

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              18.98        52.00       164.93       3120       9896

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.03    0.00   22.14    2.89   42.16   22.77

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda               2.72         0.07        59.20          4       3552
#2

Hello

We use a GP1-XS machine with block storage and are suffering from >50% iowait most of the time. During night it went down below 10% and performance of database was good but now during day time it went back to 50%.

Disk utilization is up to 100% even if data read/write measures are moderate with 5…10 MB/s.

Any ideas?

Thanks
-Silvio