vLCM: Postgres process causes high CPU usage

[mmaurischat]

Hi all,

I'm having here a vLCM appliance (vRealize 8.8.2.3 Build 20080494), which has throughtout a postgres process, that causes a CPU usage between 50% - 70%. It is not the same process (pid), but almost everytime the same statement:

select event0_.vmid as vmid1_10_, event0_.createdBy as createdB2_10_, event0_.createdOn as createdO3_10_, event0_.lastModifiedBy as lastModi4_10_, event0_.lastUpdatedOn as lastUpda5_10_, event0_.tenant as tenant6_10_, event0_.version as version7_10_, event0_.vrn as vrn8_10_, event0_.currentState as currentS9_10_, event0_.engineNodeId as engineN10_10_, event0_.errorCause as errorCa11_10_, event0_.eventArgument as eventAr12_10_, event0_.eventLock as eventLo13_10_, event0_.eventName as eventNa14_10_, event0_.sequence as sequenc15_10_, event0_.stateMachineInstance as stateMa16_10_, event0_.status as status17_10_ from vm_engine_event event0_ where event0_.status=$1 | client backend

select event0_.vmid as vmid1_10_, event0_.createdBy as createdB2_10_, event0_.createdOn as createdO3_10_, event0_.lastModifiedBy as lastModi4_10_, event0_.lastUpdatedOn as lastUpda5_10_, event0_.tenant as tenant6_10_, event0_.version as version7_10_, event0_.vrn as vrn8_10_, event0_.currentState as currentS9_10_, event0_.engineNodeId as engineN10_10_, event0_.errorCause as errorCa11_10_, event0_.eventArgument as eventAr12_10_, event0_.eventLock as eventLo13_10_, event0_.eventName as eventNa14_10_, event0_.sequence as sequenc15_10_, event0_.stateMachineInstance as stateMa16_10_, event0_.status as status17_10_ from vm_engine_event event0_ where event0_.engineNodeId=$1 and event0_.status=$2 | client backend

User: vrlcm
Application: PostgreSQL JDBC Driver
Client: 127.0.0.1

A restart of the LCM, IDMs and vRAs was not the solution 😅 VACCUM magic on the postgres DB too.

Any idea what is happening here or what I can do? My next step would otherwise be upgrading to 8.10.

Regards
Markus

[FredGSanford]

Chronic issue with this product and is still occurring in 8.12.  These commands came from VMware and were tested in a lab lcm 8.12 instance that had high cpu usage. This is not officially supported but did cut the cpu usage in half.  The culprit appears to be old requests in the UI/DB going back 2 years (over 1500) that needed to be deleted.  You can change the interval on the last 3 statements for your Requests retention needs.  Use at your own risk with snapshots.  There is supposed to be new functionality in an upcoming release that you can set the retention for Requests in the UI.  

Take snapshot

su - postgres

cd /opt/vmware/vpostgres/11/bin

./psql -d vrlcm

Run these in order:

// older requests are not significant/indicate other problems in the environment

DELETE FROM vm_engine_user_request;

DELETE FROM vm_rs_request WHERE requestname= 'lcmgenricsetting';

DELETE FROM vm_engine_event WHERE status= 'PROCESSED';

DELETE FROM vm_engine_execution_request WHERE enginestatus in ('SUCCESS','INITIATED'); 

// Note, this will clear the requests from UI and DB and cannot be inspected/viewed afterwards

DELETE from vm_rs_request where state = 'COMPLETED' and TO_TIMESTAMP(createdon/1000) < NOW() - INTERVAL '30 DAYS';

DELETE from vm_rs_request where state = 'FAILED' and TO_TIMESTAMP(createdon/1000) < NOW() - INTERVAL '30 DAYS';

DELETE from vm_rs_request where state = 'CANCELLED' and TO_TIMESTAMP(createdon/1000) < NOW() - INTERVAL '30 DAYS';

Reboot appliance.

[mmaurischat]

Hey, thanks for that answer 🙂

What about requests with state 'DELETED'? After running these commands, two requests with state 'DELETED' appeared in the UI, which are two years old.

[mmaurischat]

Ok, had to revert the snapshot. After running the commands, the environments are gone.

I'm going to open a support case.

[EagleB5]

May be this helps: https://kb.vmware.com/s/article/89962

Unfotunately, not on my 8.12 appliance 😞

[FredGSanford]

After upgrading to 8.14, I have seen a big reduction in CPU usage.