Disk write latency is high

[siegfriedLH]

Hi,

Since yesterday morning, disk write latency on my Esxi's servers is very high (30 to 120 ms).

Normally the values are between 2 to 10 ms max.

Do you know what can be the cause?

I'm running Vcenter 4.1.0, 348481 with ESXi up to date, my datastore is an IBM DS3400 FC.

Thanks a lot for your help.

[idle-jam]

i could be during that time that some other VM is using alot of the disk resources? or maybe raid rebuilding and etc. but then looks like it's Ok there after?

[siegfriedLH]

Hi,

I don't have raid rubuilding and this configuration had never exceeded 10 ms since 6 months.

I check all VM  and i didn't find any anormal disk activity :smileyconfused:

[idle-jam]

i see, try to see if any of disk intensive schedule job is recently being added to your environment? maybe a backup? anything from the SAN logs?

[siegfriedLH]

I'm going to look at the SAN's logs.

[siegfriedLH]

I find this on my DS3400's logs (can it be the cause?):

Logical Drive name:            DATASTORE_ESX1                                   
                                                                                   
      Logical Drive status:       Optimal                                          
                                                                                   
      Capacity:                   1,0 TB                                           
      Logical Drive ID:           60:0a:0b:80:00:68:bf:73:00:00:01:af:4b:ce:9e:3e  
      Subsystem ID (SSID):        3                                                
      Associated array:           RAID5                                            
      RAID level:                 5                                                
                                                                                   
      Drive type:                 Serial Attached SCSI (SAS)                       
      Enclosure loss protection:  No                                               
                                                                                   
      Preferred owner:            Controller in slot B                             
      Current owner:              Controller in slot B                             


      Segment size:                                       128 KB    
      Capacity reserved for future segment size changes:  Yes       
      Maximum future segment size:                        2 048 KB  
      Modification priority:                              High      


      Read cache:                            Enabled                        
      Write cache:                           Enabled (currently suspended)  
         Write cache without batteries:      Disabled                       
         Write cache with mirroring:         Enabled (currently suspended)  
      Flush write cache after (in seconds):  10.00                          
      Dynamic cache read prefetch:           Enabled          

Battery status:             Battery learning     
         Location:                    Controller A         
         Age:                         246 days             
         Days until replacement:      474 days             
         Last learn cycle:            30 décembre 2010     
         Next learn cycle:            30 juin 2011         
         Weeks between learn cycles:  13                   
         Part number:                 PN 39R6520           
         Serial number:               SN SY02100241SA      
         Vendor:                      VN IBM               
         Date of manufacture:         1 mai 2010           


         Battery status:              Battery learning     
         Location:                    Controller B         
         Age:                         246 days             
         Days until replacement:      474 days             
         Last learn cycle:            30 décembre 2010     
         Next learn cycle:            30 juin 2011         
         Weeks between learn cycles:  13                   
         Part number:                 PN 39R6520           
         Serial number:               SN SY95001038SA      
         Vendor:                      VN IBM               
         Date of manufacture:         1 décembre 2009  

[siegfriedLH]

I've found why the performances are degraded:

http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080951

Thanks for your help

[idle-jam]

glad that you found the root cause and thanks for sharing the link from IBM site.