Slow response for cli commands and storage issues

[pargit]

hi

i have a strange issue with some of my esxi servers.

hardware is HPE Synergy SY480-gen10 (hpe spp 2022.11.01) with esxi 7.0.3, 20328353.

i had few vms that got stunned while the backup software created a snapshot. when i tried to get the vm logs from the datastore i couldn't download them so i connected to the esxi from ssh.

when i ran df -h command, it took about 10 minutes to reply back, and most the datastores where in size 0 (while in the vc the are ok)

[root@esxka38:~] df
Filesystem         Bytes          Used     Available Use% Mounted on
VMFS-6     7696312958976 5847903109120 1848409849856  76% /vmfs/volumes/MIS_KA_DS_SSD_18
VMFS-6      858993459200  556776030208  302217428992  65% /vmfs/volumes/KA_DS_NETAPP_SQLCLDB1-VKW
VMFS-6     7696312958976 5484695257088 2211617701888  71% /vmfs/volumes/MIS_KA_DS_SSD_19
VMFS-6     7696312958976 5312562069504 2383750889472  69% /vmfs/volumes/MIS_KA_DS_SSD_20
VMFS-6     7696312958976 5728353910784 1967959048192  74% /vmfs/volumes/MIS_KA_DS_SSD_21
VMFS-6     7790802239488 6324900331520 1465901907968  81% /vmfs/volumes/KA_DS_NETAPP_BILLING
VMFS-6     7696312958976 5096337309696 2599975649280  66% /vmfs/volumes/KA_DS_NETAPP_USAGE
VMFS-L       23353884672    1704984576   21648900096   7% /vmfs/volumes/LOCKER-63cfd028-de849834-c127-3e3b4b1000e8
VMFS-6                 0             0             0   0% /vmfs/volumes/MIS_KA_DS_SSD_12
VMFS-6                 0             0             0   0% /vmfs/volumes/MIS_KA_DS_SSD_13
VMFS-6                 0             0             0   0% /vmfs/volumes/KA_DS_NETAPP_ALTSQL1-VVW
VMFS-6                 0             0             0   0% /vmfs/volumes/KA_DS_NETAPP_SQL14EXT1-VVW
VMFS-6                 0             0             0   0% /vmfs/volumes/KA_DS_NETAPP_PRECISESQL2-VVW
VMFS-6                 0             0             0   0% /vmfs/volumes/MIS_KA_DS_SSD_08
VMFS-6                 0             0             0   0% /vmfs/volumes/MIS_KA_DS_SSD_07
VMFS-6                 0             0             0   0% /vmfs/volumes/MIS_KA_DS_SSD_15
VMFS-6                 0             0             0   0% /vmfs/volumes/KA_DS_NETAPP_SQL14CAL1-VVW
VMFS-6                 0             0             0   0% /vmfs/volumes/MIS_KA_DS_SSD_09
VMFS-6                 0             0             0   0% /vmfs/volumes/MIS_KA_DS_SSD_01

in another esxi the df command also ran for long time but the reply was correct

[root@esxka39:~] df -h
Filesystem    Size   Used Available Use% Mounted on
VMFS-6      299.8G 172.9G    126.9G  58% /vmfs/volumes/KA_DS_SYSLOG
VMFS-6        1.5T 848.0G    676.0G  56% /vmfs/volumes/KA_DS_NSX
VMFS-6        7.0T   5.3T      1.7T  76% /vmfs/volumes/MIS_KA_DS_SSD_01
VMFS-6        7.0T   4.9T      2.1T  69% /vmfs/volumes/MIS_KA_DS_SSD_02
VMFS-6        7.0T   5.4T      1.6T  78% /vmfs/volumes/MIS_KA_DS_SSD_03
VMFS-6        7.0T   4.6T      2.4T  65% /vmfs/volumes/MIS_KA_DS_SSD_04
VMFS-6        7.0T   4.8T      2.2T  69% /vmfs/volumes/MIS_KA_DS_SSD_05
VMFS-6        4.0T   2.6T      1.4T  66% /vmfs/volumes/KA_DS_NETAPP_MGMT_01
VMFS-6        7.0T   5.5T      1.5T  78% /vmfs/volumes/MIS_KA_DS_SSD_06
VMFS-6        7.0T   5.4T      1.6T  78% /vmfs/volumes/MIS_KA_DS_SSD_07
VMFS-6        7.0T   5.1T      1.9T  72% /vmfs/volumes/MIS_KA_DS_SSD_08
VMFS-6        7.0T   5.3T      1.7T  75% /vmfs/volumes/MIS_KA_DS_SSD_09
VMFS-6        4.5T   2.5T      2.0T  57% /vmfs/volumes/KA_DS_NETAPP_SQL14CAL1-VVW
VMFS-6       12.0T   2.9T      9.1T  24% /vmfs/volumes/KA_DS_GUARD_AGGR

we have about 57 DS's in the environment.

i can't understand what can cause the slow response or the lack of DS information.

any help is appreciated before i open an SR for it 🙂

Thanks

Mordechai

[s_wieland]

Hi Pargit,

this sounds like an issue in your SAN environment. Do you use FibreChannel or NFS to connect the datastores. And is only one host affected or more? We had similar issues with a server with local disks. There it was an issue with the HBA.

If it's NFS please check for IP address conflicts in your network, too. If it's FibreChannel check the SFPs if a optic isn't working fine, anymore. You can do it by checking the error counters on the FC Switch.

Best regards,

Sebastian