Poor write performance on virtio windows

[bassu]

On Windows VM with virtio drivers, the write speeds are drastically very slow between 1 to 15 MB/s. Sequential reads are over 1GB/s.

Same hardware performed better with general KVM and other hypervisors even when coupled with zfsonlinux.

Global zone is not affected by it.

Prstat and izone during test.

# prstat -mLc -p 546
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
  5465 root     1.2  11 0.0 0.0 0.0 0.0  88 0.7  5K 124 14K   0 qemu-system-/3
  5465 root     5.0 3.8 0.0 0.0 0.0 0.0  90 0.9 10K  99 85K   0 qemu-system-/1
  5465 root     0.8 5.7 0.0 0.0 0.0 0.0  93 0.5  6K   0 13K   0 qemu-system-/4
  5465 root     0.6 1.6 0.0 0.0 0.0 0.0  97 0.9 10K   7 21K 10K qemu-system-/5
  5465 root     0.1 1.5 0.0 0.0 0.0 1.0  97 0.1 957   8 715   0 qemu-system-/80
  5465 root     0.1 0.8 0.0 0.0 0.0  18  81 0.1 873   0 711   0 qemu-system-/82
  5465 root     0.1 0.6 0.0 0.0 0.0  13  86 0.0 423   7 620   0 qemu-system-/77
  5465 root     0.1 0.6 0.0 0.0 0.0  16  83 0.0 413   5 603   0 qemu-system-/79
  5465 root     0.1 0.6 0.0 0.0 0.0  18  81 0.1 404  11 590   0 qemu-system-/78
  5465 root     0.1 0.6 0.0 0.0 0.0  21  78 0.0 385  10 571   0 qemu-system-/81
  5465 root     0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 784   6  1K 778 qemu-system-/2
Total: 1 processes, 11 lwps, load averages: 1.58, 1.69, 1.53

#iostat -xnz 1
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  305.0    0.0 29035.9  0.0  0.3    0.0    1.0   0  28 c0t50014EE058E6348Ed0
    0.0  306.0    0.0 29035.9  0.0  0.3    0.0    1.0   0  29 c0t50014EE0038CD9A4d0
    0.0  319.0    0.0 29579.9  0.0  0.6    0.0    2.0   0  55 c0t50014EE3556D4990d0
    0.0  320.0    0.0 29579.9  0.0  0.4    0.0    1.3   0  36 c0t50014EE3001EB3A4d0
    0.0 1120.0    0.0 117231.7 3171.0  1.7 2831.2    1.5  52  60 zones

[bassu]

Updating for rmustacc

92G RAM, one KVM Windows guest zone. GZ I/O is unaffected.

idle usage

kvmstat 1, iostat -xnz 1, vfsstat -ZzM 1, dtrace iotop

screenshot

dtrace while idle

# dtrace -s zvollat.d 
dtrace: script 'zvollat.d' matched 4 probes
^C

  f9075e84-2b8f-4c14-89ad-d519e8aa89d9                zvol_read                                         
           value  ------------- Distribution ------------- count    
            2048 |                                         0        
            4096 |@                                        998      
            8192 |@@@@@@@@@                                9616     
           16384 |@@@@@@@@@@@@@@                           15301    
           32768 |@@@@@@@@@@@@@                            14023    
           65536 |@@                                       2181     
          131072 |@                                        625      
          262144 |                                         221      
          524288 |                                         259      
         1048576 |                                         80       
         2097152 |                                         2        
         4194304 |                                         2        
         8388608 |                                         2        
        16777216 |                                         0        

  f9075e84-2b8f-4c14-89ad-d519e8aa89d9                zvol_write                                        
           value  ------------- Distribution ------------- count    
          524288 |                                         0        
         1048576 |                                         5        
         2097152 |                                         63       
         4194304 |@@@@@                                    881      
         8388608 |@@@@@@@@@@                               1635     
        16777216 |@@@@@@@@@@@@@@@@@@@@                     3205     
        33554432 |@@@                                      516      
        67108864 |@                                        176      
       134217728 |                                         39       
       268435456 |                                         0  

during a benchmark test on kvm zone running windows with latest virtio

the dtrace while benchmarking

# dtrace -s zvollat.d 
dtrace: script 'zvollat.d' matched 4 probes
^C

  f9075e84-2b8f-4c14-89ad-d519e8aa89d9                zvol_read                                         
           value  ------------- Distribution ------------- count    
            2048 |                                         0        
            4096 |@@@@@@@@@@@                              322965   
            8192 |@@@@@@@@@@@@@@@@@@@@@@                   656247   
           16384 |@@                                       59068    
           32768 |                                         1396     
           65536 |                                         1132     
          131072 |@@@@                                     129598   
          262144 |                                         9393     
          524288 |                                         244      
         1048576 |                                         11       
         2097152 |                                         1        
         4194304 |                                         5        
         8388608 |                                         6        
        16777216 |                                         0        
        33554432 |                                         0        
        67108864 |                                         1        
       134217728 |                                         0        

  f9075e84-2b8f-4c14-89ad-d519e8aa89d9                zvol_write                                        
           value  ------------- Distribution ------------- count    
          524288 |                                         0        
         1048576 |                                         18       
         2097152 |                                         583      
         4194304 |@@                                       3414     
         8388608 |@@@@@@@@@@@                              19516    
        16777216 |@@@@@@@@@@@@@@@@@@@@@@@@                 43214    
        33554432 |@@                                       3298     
        67108864 |@                                        1644     
       134217728 |                                         292      
       268435456 |                                         31       
       536870912 |                                         0 

iostat while benchmarking

# iostat -xnzM 1
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  241.0    0.0   12.9  0.0  0.2    0.0    1.0   0  14 c0t50014EE058E6348Ed0
    0.0  238.0    0.0   12.9  0.0  0.1    0.0    0.4   0   8 c0t50014EE0038CD9A4d0
    0.0  249.0    0.0   13.1  0.0  0.2    0.0    0.8   0  18 c0t50014EE3556D4990d0
    0.0  253.0    0.0   13.1  0.0  0.2    0.0    0.6   0  14 c0t50014EE3001EB3A4d0
    0.0  670.0    0.0   51.9  3.7  0.7    5.5    1.1  14  23 zones
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  327.0    0.0   25.2  0.0  0.3    0.0    0.9   0  23 c0t50014EE058E6348Ed0
    0.0  326.0    0.0   25.1  0.0  0.2    0.0    0.6   0  16 c0t50014EE0038CD9A4d0
    0.0  325.0    0.0   24.0  0.0  0.2    0.0    0.5   0  16 c0t50014EE3556D4990d0
    0.0  285.0    0.0   19.2  0.0  0.2    0.0    0.6   0  16 c0t50014EE3001EB3A4d0
    0.0  949.0    0.0   93.5 2470.0  0.9 2602.7    0.9  13  26 zones
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  359.6    0.0   27.3  0.0  0.6    0.0    1.6   0  47 c0t50014EE058E6348Ed0
    0.0  360.6    0.0   27.4  0.0  0.4    0.0    1.2   0  41 c0t50014EE0038CD9A4d0
    0.0  363.6    0.0   27.7  0.0  0.5    0.0    1.4   0  47 c0t50014EE3556D4990d0
    0.0  407.6    0.0   32.4  0.0  0.5    0.0    1.2   0  49 c0t50014EE3001EB3A4d0
    0.0 1316.6    0.0  114.7 3770.2  2.1 2863.5    1.6  57  70 zones
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  284.0    0.0   17.0  0.0  0.1    0.0    0.3   0   6 c0t50014EE058E6348Ed0
    0.0  283.0    0.0   17.0  0.0  0.1    0.0    0.3   0   5 c0t50014EE0038CD9A4d0
    0.0  284.0    0.0   17.0  0.0  0.1    0.0    0.3   0   6 c0t50014EE3556D4990d0
    0.0  283.0    0.0   17.0  0.0  0.1    0.0    0.3   0   5 c0t50014EE3001EB3A4d0
    0.0  732.0    0.0   68.0  0.0  0.4    0.0    0.5   0   7 zones
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  455.0    0.0   41.7  0.0  0.5    0.0    1.0   0  43 c0t50014EE058E6348Ed0
    0.0  456.0    0.0   41.7  0.0  0.4    0.0    0.9   0  40 c0t50014EE0038CD9A4d0
    0.0  468.0    0.0   42.1  0.0  0.6    0.0    1.3   0  55 c0t50014EE3556D4990d0
    0.0  467.0    0.0   42.1  0.0  0.5    0.0    1.1   0  48 c0t50014EE3001EB3A4d0
    0.0 1615.0    0.0  167.5 6348.8  2.1 3931.2    1.3  55  59 zones
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  250.3    0.0   11.2  0.0  0.1    0.0    0.5   0  11 c0t50014EE058E6348Ed0
    0.0  245.2    0.0   11.2  0.0  0.2    0.0    0.8   0  18 c0t50014EE0038CD9A4d0
    0.0  252.3    0.0   11.4  0.0  0.4    0.0    1.4   0  25 c0t50014EE3556D4990d0
    0.0  248.2    0.0   11.4  0.0  0.2    0.0    0.8   0  18 c0t50014EE3001EB3A4d0
    0.0  725.7    0.0   45.2  5.5  0.9    7.6    1.3  22  36 zones
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  272.7    0.0   16.0  0.0  0.1    0.0    0.3   0   5 c0t50014EE058E6348Ed0
    0.0  272.7    0.0   16.0  0.0  0.1    0.0    0.3   0   5 c0t50014EE0038CD9A4d0
    0.0  271.7    0.0   16.4  0.0  0.1    0.0    0.5   0   8 c0t50014EE3556D4990d0
    0.0  273.7    0.0   16.4  0.0  0.1    0.0    0.3   0   5 c0t50014EE3001EB3A4d0
    0.0  703.3    0.0   64.7  0.0  0.4    0.0    0.6   0  10 zones

kvmstat

# kvmstat 1
 22950    0 |   7788 :   1002     84      7    110   2143 |   1047   5213     18
 22950    1 |   5689 :   1080      5     11     64   2161 |   1095   3389      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7824 :   1014     65     12    123   2148 |   1057   5273     18
 22950    1 |   5862 :   1098      6     17     70   2200 |   1126   3489      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7619 :    979     71      7     83   2091 |   1029   5186     18
 22950    1 |   5463 :   1053     12     12     33   2110 |   1071   3260      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7923 :   1003     87      6    104   2195 |   1068   5368      0
 22950    1 |   5818 :   1084     32      9     53   2182 |   1116   3464     18
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7877 :   1007     76      8    128   2140 |   1063   5282     34
 22950    1 |   5918 :   1114      5      7     64   2233 |   1128   3536      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7810 :   1008     81      2     91   2152 |   1058   5277     18
 22950    1 |   5815 :   1097      3     10     71   2201 |   1109   3467      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7544 :    989     51      1     59   2100 |   1023   5158     18
 22950    1 |   5356 :   1039      9      5     37   2080 |   1050   3194      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7789 :   1006     65      5    121   2143 |   1052   5221     18
 22950    1 |   5729 :   1085      0     14     65   2171 |   1104   3414      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7877 :   1012     78     10    139   2154 |   1064   5287     18
 22950    1 |   5876 :   1108     11      9     61   2216 |   1125   3510      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7743 :    983    127      8     48   2132 |   1049   5276     18
 22950    1 |   5555 :   1051     13      4     42   2124 |   1079   3330      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7906 :   1019     93      7    123   2168 |   1066   5292     18
 22950    1 |   5896 :   1104      7     11     72   2218 |   1122   3524      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7777 :   1010     53      5    117   2145 |   1053   5227     18
 22950    1 |   5769 :   1096      3      4     67   2193 |   1106   3447      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7599 :    992     60      1     68   2112 |   1024   5161     18
 22950    1 |   5368 :   1041     10      2     37   2085 |   1049   3207      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7740 :   1001     62      4    107   2132 |   1051   5222     18
 22950    1 |   5651 :   1078      4      8     48   2159 |   1088   3383      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7768 :   1008     69      6    125   2137 |   1051   5211     18
 22950    1 |   5799 :   1100      2     11     69   2202 |   1110   3454      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7698 :    998     57      1     84   2134 |   1033   5202     18
 22950    1 |   5502 :   1058      7      8     47   2121 |   1071   3280      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7712 :    985     71      3     72   2141 |   1034   5224     18
 22950    1 |   5491 :   1056     12      4     41   2119 |   1068   3281      0
   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
 22950    0 |   7871 :   1017     58      4    137   2165 |   1059   5266     18

benchmark results inside the windows zone

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 Shizuku Edition x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :  1098.643 MB/s
          Sequential Write :    24.251 MB/s
         Random Read 512KB :   964.529 MB/s
        Random Write 512KB :    20.833 MB/s
    Random Read 4KB (QD=1) :    23.792 MB/s [  5808.5 IOPS]
   Random Write 4KB (QD=1) :     0.513 MB/s [   125.4 IOPS]
   Random Read 4KB (QD=32) :    96.296 MB/s [ 23509.7 IOPS]
  Random Write 4KB (QD=32) :     5.322 MB/s [  1299.2 IOPS]

  Test : 1000 MB [C: 16.7% (13.3/79.9 GB)] (x5)
  Date : 2013/11/28 0:44:48
    OS : Windows Server 2008 R2 Datacenter (Full installation) SP1 [6.1 Build 7601] (x64)