Esxi 7.0 U1 Losing config on reboot

[ContactSD]

Hi All,

We are having issues with a host reverting any changes and unregistering VMs when you reboot the host.  

for example if I update the IP address for the host and restart the management network the config changes work but when the host reboots it will go back to using its old IP address, One of the virtual machines will also disappear and I have to select register existing VM and select the .vmx from the datastore again.

The server is a HPE DL360 Gen 10 and ESXi is installed on aHPE 32GB microSD Flash Memory Card 700139-B21

I have tried running /sbin/auto-backup.sh 

Spoiler

[root@localhost:~] /sbin/auto-backup.sh
--- /etc/vmware/hostd/authorization.xml
+++ /tmp/auto-backup.527294//etc/vmware/hostd/authorization.xml
@@ -1,27 +0,0 @@
-<ConfigRoot>
- <ACEData id="10">
- <ACEDataEntity>ha-folder-root</ACEDataEntity>
- <ACEDataId>10</ACEDataId>
- <ACEDataIsGroup>false</ACEDataIsGroup>
- <ACEDataPropagate>true</ACEDataPropagate>
- <ACEDataRoleId>-1</ACEDataRoleId>
- <ACEDataUser>root</ACEDataUser>
- </ACEData>
- <ACEData id="11">
- <ACEDataEntity>ha-folder-root</ACEDataEntity>
- <ACEDataId>11</ACEDataId>
- <ACEDataIsGroup>false</ACEDataIsGroup>
- <ACEDataPropagate>true</ACEDataPropagate>
- <ACEDataRoleId>-1</ACEDataRoleId>
- <ACEDataUser>dcui</ACEDataUser>
- </ACEData>
- <ACEData id="12">
- <ACEDataEntity>ha-folder-root</ACEDataEntity>
- <ACEDataId>12</ACEDataId>
- <ACEDataIsGroup>false</ACEDataIsGroup>
- <ACEDataPropagate>true</ACEDataPropagate>
- <ACEDataRoleId>-1</ACEDataRoleId>
- <ACEDataUser>vpxuser</ACEDataUser>
- </ACEData>
- <NextAceId>13</NextAceId>
-</ConfigRoot>
\ No newline at end of file
Saving current state in /bootbank
Clock updated.
Time: 14:16:25 Date: 12/23/2020 UTC

This is the output from the log

[root@localhost:~] cat /sbin/auto-backup.sh
#!/bin/sh

. /usr/lib/vmware/feature-state/feature-state.sh

export PATH=/bin:/sbin

check_configstore()
{
curr=$(date +%s -r "/etc/vmware/configstore/current-store-1")

backup_cs="${tmp_backup_dir}/var/lib/vmware/configstore/backup/current-store-1"
backup=$(date +%s -r ${backup_cs})

if [ $curr -ge $backup ]; then
logger "ConfigStore has been modified since the last backup"
return 1
fi

return 0
}

decrypt_system_configuration()
{
local work_dir="$1"

# Keep this command in sync with backup.sh
crypto_util="crypto-util ++coreDumpEnabled=false,mem=20"
src="${work_dir}/local.tgz.ve"
dst="${work_dir}/local.tgz"
cmd="${crypto_util} envelope extract ${src} ${dst}"
msg=$(${cmd} 2>&1)
res=$?

# Check if we decrypted successfully and we have non-zero file.
if ! [ ${res} -eq 0 -a -s ${dst} ] ; then
logger "Failed command: ${cmd}"
logger "Error: ${msg}"
return 1
fi

return ${res}
}

main()
{
local files_to_save=

# PXE boots need not be backed-up
if [ ! -d /bootbank ] ; then
esxcfg-init --alert "Bootbank cannot be found at path '/bootbank'"
exit 1
fi

if [ ! -f /bootbank/state.tgz ] ; then
# always backup if we do not yet have a state.tgz
/sbin/backup.sh 0 || logger "state.tgz not found, auto-backup failed"
exit 0
fi

files_to_save=$(find /etc -follow -type f -name '.#*' 2>/dev/null |
sed -e 's,.#\(.*\),\1,g' |
while read name ; do [ -f "${name}" ] && echo "${name}" ; done)

tmp_backup_dir="/tmp/auto-backup.$$"

mkdir -p "${tmp_backup_dir}"

tar xzf /bootbank/state.tgz -C "${tmp_backup_dir}" || {
rm -rf "${tmp_backup_dir}"
logger "Unable to extract system configuration. Are you out of disk space?"
exit 1
}

if isFeatureEnabled ESXConfigEncryption && \
[ -s ${tmp_backup_dir}/local.tgz.ve ] ; then
decrypt_system_configuration "${tmp_backup_dir}" || {
logger "Unable to decrypt encrypted system configuration"
exit 1
}
fi

tar xzf "${tmp_backup_dir}/local.tgz" -C "${tmp_backup_dir}" || {
rm -rf "${tmp_backup_dir}"
logger "Unable to extract system state. Are you out of disk space?"
exit 1
}

check_configstore "${tmp_backup_dir}" || {
/sbin/backup.sh 0 || logger "failed to regenerate system configuration"
rm -rf "${tmp_backup_dir}"
exit 0
}

for file in ${files_to_save} ; do
diff -N "${file}" "${tmp_backup_dir}/${file}" || {
/sbin/backup.sh 0 || logger "failed to regenerate system configuration"
break
}
done

# clean up
rm -rf "${tmp_backup_dir}"
}

main "${@}"

I've reinstalled ESXi and this seem to resolve the issue at first but its come back again, this is vSphere essentials but we only have 1 host so I have not installed vCentre yet.

I'm new to VMware and any help would be greatly appreciated.

[scott28tt]

@ContactSD 

Moderator: I moved your script output to a “spoiler” section to make the thread easier for others to scroll through - this function is the triangle on the editor toolbar with an exclamation mark inside it.

[MDiaa · Accepted solution]

Have a look at https://kb.vmware.com/s/article/2149444