I'm having mailserver problems....again.
I had successfully set up and run three tiles and recently added three more to the testbed. Now I'm seeing no wrf files for the odd-numbered tiles; tile1 had been working before, in the original set of three. I'm seeing that the vmmark2-mailserver-x.xml file is not getting updated in the respective failing clients and the mailservergetstats_error-x file(s) in the result directories contains:
And now when I try and run the original three tiles, tile1 fails to produce a mailserver wrf file. Everything else looks ok. The firewall is turn off on all of the clients and all of the mailservers.
I'm attaching my results folder, less the host tgzs.
Please help.
Thanks,
-Kirk
Are the original 3 mailservers talking to their original 3 clients? IE has client1 has always talked to current mailserver1? It's very odd that a client who was previously able to communicate and drive load for a mailserver cannot after adding tiles, as nothing should have changed for tile1.
I would double check that all of the clients are in their proper domains. I would also double check that your Loadgen installations haven't become corrupted.
I still haven't resolved this issue. I restored client0 to a previously working snapshot and tile0 works fine. I have rebuilt the tile1 client and mailserver from scratch. The databases set up properly. I can run the initialization from client1's loadgen and see activity on mailserver1. When I run the vmmark2-mailserver-1 from client1 it starts doing work then faults out from overload. I ran the simulation for 15 minutes, so I'm not too worried about this. When I run the harness I see mailserver1 initializing the databases and the logs in C: of mailserver1 indicate proper starts, stops and file operations. However when it runs the load there is no activity on mailserver1. The timestamp on the vmmark2-mailserver-1 file coincides with the test run. It's as if the prime client can't get client1 to run loadgen.
I'm going to snoop client1 with wireshark to see if I can capture the command that initiates the workload.
STAF on the prime client can ping client1 and mailserver1. STAF on client1 can ping client0 and mailserver1. STAF on mailserver1 can ping client1 and the prime clent.
Any thoughts?
Thanks,
-Kirk
Here's the log:
20120420-10:10:45 | Starting VMmark2.1 |
20120420-10:10:51 | VMmark2 : Reading Configuration File :: C:\vmmark2\VMMARK2.CONFIG |
20120420-10:10:51 | Default: VMMARK/IMPORTDIR = c:\vmmark2\xml |
20120420-10:10:51 | Enabled: VMMARK/DEBUGFLAG = 1 |
20120420-10:10:51 | Overriding Required Default: VMMARK/RUNTIME = 900 |
20120420-10:10:52 | Default: VMMARK/PARALLELSETUP = 1 |
20120420-10:10:52 | Overriding Default: VMMARK/PARALLELSETUP = 1 |
20120420-10:10:52 | Default: VMMARK/POSTPROCESSCMD = C:\cygwin\bin\perl -f C:\vmmark2\tilescore.pl -t %u |
20120420-10:10:52 | Default: VMMARK/TILEDELAY = 60 |
20120420-10:10:52 | Overriding Required Default: VMMARK/REPORTER = 1 |
20120420-10:10:52 | Default: VMMARK/ScriptName = vSphere-get-config.sh |
20120420-10:10:52 | Default: VMMARK/REPORTERDIR = /cygdrive/c/vmmark2/tools/ |
20120420-10:10:52 | Default: VMMARK/RUNFLAG = 3 |
20120420-10:10:52 | Default: VMMARK/RESCUE = 0 |
20120420-10:10:52 | Default: VMMARK/CLEANUPFLAG = 1 |
20120420-10:10:52 | Default: VMMARK/ERRORIMMEDIATE = 0 |
20120420-10:10:52 | UserSpecified: VMMARK/VCServerUSER = Administrator |
20120420-10:10:52 | UserSpecified: VMMARK/VCServerPASSWORD ***** |
20120420-10:10:52 | UserSpecified: VMMARK/VCServerCLUSTER = QA-Performance |
20120420-10:10:52 | UserSpecified: VMMARK/VCServerHOSTNAME = 192.168.101.81 |
20120420-10:10:52 | Default: VMMARK/DoClusterMapping = 1 |
20120420-10:10:52 | Default: VMMARK/INFRAOPSLIST = Vmotion SVmotion Deploy |
20120420-10:10:52 | Overriding Required Default: VMMARK/WORKLOADLIST = MailServer |
20120420-10:10:52 | Default: VMMARK/CONFIGDIR = c:\vmmark2 |
20120420-10:10:52 | Default: VMMARK/TMPDIR = c:\vmmark2\tmp |
20120420-10:10:52 | Default: VMMARK/CLIENTROOTDIR = C:\\vclient%u |
20120420-10:10:52 | UNIFIED Work List : ['MailServer', 'Vmotion', 'SVmotion', 'Deploy'] |
20120420-10:10:52 | Info: FS Create Directory: c:\vmmark2\results\Results_20120420101045_Two_Tiles Returned: RC = 0, STAFResult = |
20120420-10:10:52 | Info: COPY FILE C:\vmmark2\VMMARK2.CONFIG TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\VMMARK2.CONFIG |
20120420-10:10:52 | Info: FS Copy file: VMMARK2.CONFIG Returned: RC = 0, STAFResult = |
20120420-10:10:52 | Info: Create guestinfofiles subdirectory for guest host information files |
20120420-10:10:52 | Info: FS Create Directory: c:\vmmark2\results\Results_20120420101045_Two_Tiles\guestinfofiles Returned: RC = 0, STAFResult = |
20120420-10:10:52 | VMmark2 : Configuring Workloads |
20120420-10:10:52 | Default: VMMARK/MailServer/SKIPRESTORE = 0 |
20120420-10:10:52 | Default: VMMARK/MailServer/MailServerPassword = ***** |
20120420-10:10:52 | UserSpecified: VMMARK/MailServer/MailDomains = ['Maildomain0', 'Maildomain1'] |
20120420-10:10:52 | UserSpecified: VMMARK/MailServer/MailQualifier = vtg.vce.com |
20120420-10:10:52 | Default: VMMARK/MailServer/clientpath = C:\Progra~1\Exchan~1\ |
20120420-10:10:52 | Default: VMMARK/MailServer/DELAYTIME = 12 |
20120420-10:10:52 | Default: VMMARK/MailServer/dblocation = E:\vmmark\ |
20120420-10:10:52 | Default: VMMARK/MailServer/backuplocation = f:\ |
20120420-10:10:52 | Default: VMMARK/Vmotion/Workloads = OlioDB |
20120420-10:10:52 | Default: VMMARK/SVmotion/Workloads = Standby |
20120420-10:10:52 | UserSpecified: NumSVmotionLUN(s) = '['Performance_Datastore6', 'Performance_Datastore7', 'Performance_Datastore8']' |
20120420-10:10:52 | Default DeployUpgradeTools : 1 |
20120420-10:10:52 | UserSpecified: OScustomizations = '['VMmark_Customization0', 'VMmark_Customization0']' |
20120420-10:10:52 | UserSpecified: DeployLUN(s) = '['Performance_Datastore3', 'Performance_Datastore4', 'Performance_Datastore5']' |
20120420-10:10:52 | UserSpecified: DeployVMs = '['DeployVM0', 'DeployVM1']' |
20120420-10:10:52 | UserSpecified: Deploy Templates = '['DeployTemplate', 'DeployTemplate']' |
20120420-10:10:52 | VMmark2 : Starting Environment Setup |
20120420-10:10:52 | Info: tileindex = 0 |
20120420-10:10:52 | Info: machName = client0 |
20120420-10:10:52 | Info: machName = Mailserver0 |
20120420-10:10:53 | Info: tileindex = 1 |
20120420-10:10:53 | Info: machName = client1 |
20120420-10:10:55 | Info: machName = Mailserver1 |
20120420-10:10:57 | Client client0 : STAF version check : min 3.4.0 : found 3.4.7 |
20120420-10:10:58 | Client client1 : STAF version check : min 3.4.0 : found 3.4.7 |
20120420-10:10:58 | Info: All Client STAF Versions meet minimum requirements |
20120420-10:10:58 | VMmark2 : Checking Client Time Sync |
20120420-10:10:58 | Tile 0 : Prime Client Time 1334941858 : Client client0 Time : 1334941858 :: TimeErrors : 0 :: Clock Skew 0 |
20120420-10:10:59 | Tile 1 : Prime Client Time 1334941859 : Client client1 Time : 1334941859 :: TimeErrors : 0 :: Clock Skew 0 |
20120420-10:10:59 | vCServer : Prime Client Time 1334941859 : vCServer 192.168.101.81 Time : 1334941859 :: TimeErrors : 0 :: Clock Skew 0 |
20120420-10:10:59 | Info: All Clients and vCServer Times Within 60 Second Threshold |
20120420-10:10:59 | InfraOps: Request Cluster ListHosts Info : (QA-Performance, c:\VMmark2\tools\VMmarkPScli.ps1 -f listhosts -vc 192.168.101.81 -c QA-Performance -u Administrator -p 'Acadia123') |
20120420-10:11:24 | VMmark 2 : Hosts In Cluster : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com'] |
20120420-10:11:24 | InfraOps: sVMotion Request Cluster Info : (QA-Performance, c:\VMmark2\tools\VMmarkPScli.ps1 -f listvms -vc 192.168.101.81 -c QA-Performance -u Administrator -p 'Acadia123') |
20120420-10:11:48 | VMs In Cluster : ['Mailserver0', 'DS2WebA0', 'DS2WebB0', 'DS2WebC0', 'Standby0', 'OlioWeb0', 'DS2DB0', 'Standby1', 'DS2DB1', 'DS2WebA1', 'DS2WebB1', 'DS2WebC1', 'OlioWeb1', 'Mailserver1', 'DS2DB2', 'DS2WebA2', 'DS2WebB2', 'DS2WebC2', 'OlioDB2', 'OlioWeb2', 'Mailserver2', 'Standby2', 'OlioDB0', 'OlioDB1', 'DS2WebC3', 'OlioDB3', 'OlioWeb3', 'DS2DB3', 'DS2WebA3', 'DS2WebB3', 'Mailserver3', 'Standby3', 'DS2WebC4', 'OlioDB4', 'OlioWeb4', 'DS2DB4', 'DS2WebA4', 'DS2WebB4', 'Mailserver4', 'Standby4', 'DS2WebB5', 'DS2WebC5', 'OlioDB5', 'OlioWeb5', 'Standby5', 'DS2DB5', 'DS2WebA5', 'Mailserver5'] |
20120420-10:11:48 | Checking VM Mapping And Resources |
20120420-10:13:41 | Warning : Cluster VM Mapping And Resources returned Warnings : InvalidResources 0 : PoweredOnErrors 1 : See c:\vmmark2\results\Results_20120420101045_Two_Tiles\ClusterVMmapping.txt |
20120420-10:13:52 | Info: Start Timer: client0 : start shell command "mstmr" workdir C:\\vclient0\mstmr workload mstmr0 sameconsole |
20120420-10:13:52 | Info: Start Timer: client1 : start shell command "mstmr" workdir C:\\vclient1\mstmr workload mstmr1 sameconsole |
20120420-10:13:52 | VMmark2 : Starting Workload Setup |
20120420-10:13:52 | Info: Tile0: COPY FILE vmmark2-mailserver-0.xml Returned: RC = 0, STAFResult = |
20120420-10:13:52 | Info: Tile0: DELETE ENTRY mailserver_exchange2007*.txt Returned: RC = 0, STAFResult = [] |
20120420-10:13:52 | Tile0 : Starting MailServer Restore Process |
20120420-10:13:53 | Info: Tile1: COPY FILE vmmark2-mailserver-1.xml Returned: RC = 0, STAFResult = |
20120420-10:13:53 | Info: Tile1: DELETE ENTRY mailserver_exchange2007*.txt Returned: RC = 0, STAFResult = [] |
20120420-10:13:53 | Tile1 : Starting MailServer Restore Process |
20120420-10:14:48 | Tile0 : MailServer Finished Restore Process |
20120420-10:15:05 | Tile1 : MailServer Finished Restore Process |
20120420-10:15:05 | ListVMs already done |
20120420-10:15:05 | ListHosts already done |
20120420-10:15:05 | Found ALL needed VMs for VMotion : Powered On VMs : ['Mailserver0', 'DS2WebA0', 'DS2WebB0', 'DS2WebC0', 'Standby0', 'OlioWeb0', 'DS2DB0', 'Standby1', 'DS2DB1', 'DS2WebA1', 'DS2WebB1', 'DS2WebC1', 'OlioWeb1', 'Mailserver1', 'DS2DB2', 'DS2WebA2', 'DS2WebB2', 'DS2WebC2', 'OlioDB2', 'OlioWeb2', 'Mailserver2', 'Standby2', 'OlioDB0', 'OlioDB1', 'DS2WebC3', 'OlioDB3', 'OlioWeb3', 'DS2DB3', 'DS2WebA3', 'DS2WebB3', 'Mailserver3', 'Standby3', 'DS2WebC4', 'OlioDB4', 'OlioWeb4', 'DS2DB4', 'DS2WebA4', 'DS2WebB4', 'Mailserver4', 'Standby4', 'DS2WebB5', 'DS2WebC5', 'OlioDB5', 'OlioWeb5', 'Standby5', 'DS2DB5', 'DS2WebA5', 'Mailserver5'] |
20120420-10:15:05 | vmotion VMs : ['OlioDB0', 'OlioDB1'] |
20120420-10:15:05 | Hosts in cluster : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com'] |
20120420-10:15:05 | gHostNames : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com'] :: NumHosts 6 |
20120420-10:15:05 | Calculated sVMotionBurstQueueSize = 1 |
20120420-10:15:05 | UserSpecified: NumLUNs : 3 : TargetLUN(s) = '['Performance_Datastore6', 'Performance_Datastore7', 'Performance_Datastore8']' |
20120420-10:15:05 | ListVMs already done |
20120420-10:15:05 | Found ALL needed VMs for Storage VMotion : Powered On VMs : ['Mailserver0', 'DS2WebA0', 'DS2WebB0', 'DS2WebC0', 'Standby0', 'OlioWeb0', 'DS2DB0', 'Standby1', 'DS2DB1', 'DS2WebA1', 'DS2WebB1', 'DS2WebC1', 'OlioWeb1', 'Mailserver1', 'DS2DB2', 'DS2WebA2', 'DS2WebB2', 'DS2WebC2', 'OlioDB2', 'OlioWeb2', 'Mailserver2', 'Standby2', 'OlioDB0', 'OlioDB1', 'DS2WebC3', 'OlioDB3', 'OlioWeb3', 'DS2DB3', 'DS2WebA3', 'DS2WebB3', 'Mailserver3', 'Standby3', 'DS2WebC4', 'OlioDB4', 'OlioWeb4', 'DS2DB4', 'DS2WebA4', 'DS2WebB4', 'Mailserver4', 'Standby4', 'DS2WebB5', 'DS2WebC5', 'OlioDB5', 'OlioWeb5', 'Standby5', 'DS2DB5', 'DS2WebA5', 'Mailserver5'] |
20120420-10:15:05 | sVMotion VMs : ['Standby0', 'Standby1'] |
20120420-10:15:37 | Storage vMotion TargetLUN located and all VMs are on different LUNs |
20120420-10:15:37 | ListHosts already done |
20120420-10:15:37 | gHostNames : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com'] :: NumHosts 6 |
20120420-10:15:37 | Calculated DeployBurstQueueSize = 1 |
20120420-10:15:37 | UserSpecified: Deploy NumVMss : 2 : DeployVM(s) = '['DeployVM0', 'DeployVM1']' |
20120420-10:15:37 | UserSpecified: Deploy NumLUNs : 3 : DeployLUN(s) = '['Performance_Datastore3', 'Performance_Datastore4', 'Performance_Datastore5']' |
20120420-10:15:37 | UserSpecified: Deploy NumTemplates : 2 : Deploy Template(s) = '['DeployTemplate', 'DeployTemplate']' |
20120420-10:16:02 | Info: Template 'DeployTemplate' : Located |
20120420-10:16:26 | Info: OS customization 'VMmark_Customization0' : Located |
20120420-10:16:56 | Info: DeployLUN 'Performance_Datastore3' : Located |
20120420-10:17:01 | VMmark2 : Run Starting |
20120420-10:17:01 | InfraOps: Collect Events and Perf |
20120420-10:17:01 | Tile 1 Delay 60000 ms |
20120420-10:17:01 | Tile 0 Delay 0 ms |
20120420-10:17:01 | Info: Request: run mailserver 0 |
20120420-10:17:13 | Tile0: LoadGen Delay 300 |
20120420-10:17:13 | Tile 0: Request: Run mailserver |
20120420-10:18:01 | Info: Request: run mailserver 1 |
20120420-10:18:01 | Info : Calculated VMotionBurstQueueSize = 1 |
20120420-10:18:01 | Info : Calculated sVMotionBurstQueueSize = 1 |
20120420-10:18:01 | Info : Calculated DeployBurstQueueSize = 1 |
20120420-10:18:13 | Tile1: LoadGen Delay 300 |
20120420-10:18:13 | Tile 1: Request: Run mailserver |
20120420-10:20:31 | InfraOps: VMotion0 #0 : OlioDB0 |
20120420-10:20:41 | InfraOps: Request sVMotion0 #1 : VM Standby0 |
20120420-10:20:51 | DeployBurstQueue = ['DeployVM0'], DeployBurstHostQueue ['vxgle2vh0024.vxgle2.vtg.vce.com'] |
20120420-10:20:51 | InfraOps: Request Deploy0 #0 : DeployVM0 onto SUT vxgle2vh0024.vxgle2.vtg.vce.com in LUN Performance_Datastore3 using VMmark_Customization0 |
20120420-10:22:13 | Tile0 : Collecting MailServer Stats |
20120420-10:23:13 | Tile1 : Collecting MailServer Stats |
20120420-10:23:14 | Process: Tile 1: Collecting MailServer Stats failed to start/complete. Returned: RC = 4026531842, STAFResult = None |
20120420-10:24:09 | InfraOps: VMotion0 #1 : OlioDB1 |
20120420-10:27:49 | InfraOps: VMotion0 #2 : OlioDB0 |
20120420-10:28:33 | InfraOps: Thread Reached MinTime for sVMotions |
20120420-10:28:41 | InfraOps: Thread Reached MinTime for Deploy |
20120420-10:28:41 | Info: ping newly Deployed VM = 'DeployVM0' |
20120420-10:28:42 | Deployed VM #0 : Passed Ping Test |
20120420-10:28:42 | InfraOps: Request Destroy Deployed VM #0 : DeployVM0 |
20120420-10:29:11 | VM 'DeployVM0' : Destruction completed successfully |
20120420-10:31:31 | InfraOps: VMotion0 #3 : OlioDB1 |
20120420-10:32:08 | InfraOps : Thread Reached MinTime for VMotions |
20120420-10:35:54 | VMmark2 : Checking Client Time Sync |
20120420-10:35:54 | Tile 0 : Prime Client Time 1334943354 : Client client0 Time : 1334943354 :: TimeErrors : 0 :: Clock Skew 0 |
20120420-10:35:54 | Tile 1 : Prime Client Time 1334943354 : Client client1 Time : 1334943354 :: TimeErrors : 0 :: Clock Skew 0 |
20120420-10:35:54 | vCServer : Prime Client Time 1334943354 : vCServer 192.168.101.81 Time : 1334943354 :: TimeErrors : 0 :: Clock Skew 0 |
20120420-10:35:54 | Info: All Clients and vCServer Times Within 60 Second Threshold |
20120420-10:35:54 | VMmark2 : Collecting Results |
20120420-10:35:54 | Info: COPY FILE C:\\vclient0\mailserver\Mailserver0.wrf TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\Mailserver0.wrf |
20120420-10:35:54 | Info: FS Copy file: Mailserver0.wrf Returned: RC = 0, STAFResult = |
20120420-10:35:54 | Info: COPY DIRECTORY C:\\vclient0\mailserver\ TODIRECTORY c:\vmmark2\results\Results_20120420101045_Two_Tiles\mailserver-debug0 |
20120420-10:35:55 | Info: FS Copy Directory mailserver-debug0.xml Returned: RC = 0, STAFResult = |
20120420-10:35:55 | Info: COPY FILE C:\\vclient1\mailserver\Mailserver1.wrf TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\Mailserver1.wrf |
20120420-10:35:55 | Stafcmd: FS Copy file: Mailserver1.wrf failed with RC = 48, STAFResult = File C:\\vclient1\mailserver\Mailserver1.wrf does not exist |
20120420-10:35:55 | Info: COPY DIRECTORY C:\\vclient1\mailserver\ TODIRECTORY c:\vmmark2\results\Results_20120420101045_Two_Tiles\mailserver-debug1 |
20120420-10:35:59 | Info: FS Copy Directory mailserver-debug1.xml Returned: RC = 0, STAFResult = |
20120420-10:36:02 | Could not Collect all Results for the following 1 Wklds: ['MailServerTile1 failed results collection'] |
20120420-10:36:02 | Warning: VMmark may not have completed successfully. Check for results manually, see documentation for instructions. |
20120420-10:36:02 | Info: Stop Timer: client0 : stop workload mstmr0 using SIGTERM |
20120420-10:36:02 | Info: Stop Timer: client1 : stop workload mstmr1 using SIGTERM |
20120420-10:36:02 | VMmark2 : Reporter Routines Starting |
20120420-10:36:02 | ListHosts already done |
20120420-10:36:02 | Reporter : Generate Cluster Report : QA-Performance |
20120420-10:36:03 | Info: vxgle2vh0020.vxgle2.vtg.vce.com : Starting Reporting Script Routine |
20120420-10:36:03 | Info: vxgle2vh0018.vxgle2.vtg.vce.com : Starting Reporting Script Routine |
20120420-10:36:03 | Info: vxgle2vh0021.vxgle2.vtg.vce.com : Starting Reporting Script Routine |
20120420-10:36:03 | Info: vxgle2vh0023.vxgle2.vtg.vce.com : Starting Reporting Script Routine |
20120420-10:36:03 | Info: vxgle2vh0022.vxgle2.vtg.vce.com : Starting Reporting Script Routine |
20120420-10:36:03 | Info: vxgle2vh0024.vxgle2.vtg.vce.com : Starting Reporting Script Routine |
20120420-10:37:27 | Info: Tile0 : Mailserver0 : Get Windows guest host information |
20120420-10:37:27 | Info: Tile1 : Mailserver1 : Get Windows guest host information |
20120420-10:37:35 | Info: COPY FILE c:\Mailserver1_guestVM.info TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\guestinfofiles\Mailserver1_guestVM.info |
20120420-10:37:35 | Info: COPY FILE c:\Mailserver0_guestVM.info TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\guestinfofiles\Mailserver0_guestVM.info |
20120420-10:37:36 | Info: Create Windows guest host files: 1 Returned: RC = 0, STAFResult = |
20120420-10:37:37 | Info: Create Windows guest host files: 0 Returned: RC = 0, STAFResult = |
20120420-10:37:37 | Info: Tile1 : Mailserver1 : Get Windows guest host information completed |
20120420-10:37:37 | Info: Tile0 : Mailserver0 : Get Windows guest host information completed |
20120420-10:38:49 | scp root@vxgle2vh0021.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0021.vxgle2.vtg.vce.com.txt |
20120420-10:38:49 | scp root@vxgle2vh0021.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore1/reporter/vxgle2vh0021-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/ |
20120420-10:38:51 | scp root@vxgle2vh0022.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0022.vxgle2.vtg.vce.com.txt |
20120420-10:38:52 | scp root@vxgle2vh0022.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0022-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/ |
20120420-10:38:52 | vxgle2vh0021.vxgle2.vtg.vce.com : Removing Reporting Result from Host |
20120420-10:38:54 | vxgle2vh0022.vxgle2.vtg.vce.com : Removing Reporting Result from Host |
20120420-10:38:56 | scp root@vxgle2vh0023.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0023.vxgle2.vtg.vce.com.txt |
20120420-10:38:56 | scp root@vxgle2vh0023.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0023-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/ |
20120420-10:38:58 | vxgle2vh0023.vxgle2.vtg.vce.com : Removing Reporting Result from Host |
20120420-10:39:04 | scp root@vxgle2vh0018.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0018.vxgle2.vtg.vce.com.txt |
20120420-10:39:04 | scp root@vxgle2vh0018.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0018-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/ |
20120420-10:39:07 | vxgle2vh0018.vxgle2.vtg.vce.com : Removing Reporting Result from Host |
20120420-10:39:17 | scp root@vxgle2vh0020.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0020.vxgle2.vtg.vce.com.txt |
20120420-10:39:18 | scp root@vxgle2vh0020.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0020-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/ |
20120420-10:39:20 | vxgle2vh0020.vxgle2.vtg.vce.com : Removing Reporting Result from Host |
20120420-10:42:34 | scp root@vxgle2vh0024.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0024.vxgle2.vtg.vce.com.txt |
20120420-10:42:35 | scp root@vxgle2vh0024.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0024-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/ |
20120420-10:42:37 | vxgle2vh0024.vxgle2.vtg.vce.com : Removing Reporting Result from Host |
20120420-10:42:37 | Info: Cleaning up STAF System Variables |
20120420-10:42:37 | VMmark2 : Ending Normally |
Hi,
Can you zip up the entire results directory for this run? I'm concerned about it faulting due to overload and want to review for task exceptions.
Thanks
-Joshua
Can you confirm the state of the counters under load?
Start a test and then wait for initialization phase to pass.
Then:
1. Log into your client1 as the domain1 administrator.
2. Open a command prompt and type "perfmon".
3. Select "Performance Monitor" from the left pane (Under performance).
4. Click the green add button.
5. You should be able to scroll up to the "Exchange Load Generator Engine" counters.
6. Press the expand + box to the right of the name and then select the "Task Completed" and "Task Exceptions" counters.
7. Press the "add >>" button and then OK.
Let me know what you see.
Joshua,
After I hit the OK button with these counters, I see a graph with nothing but little blips along the bottom which indicate processor time (when I hover over them).
-Kirk
It doesn't look like these counters are getting added. I also tried adding them via the properties; I see them added with colors for the datasets, but nothing is put into the main graph panel.
-Kirk
That sounds like loadgen wasn't actually running. I can see that client1' s vmark-mailserver-1.xml is getting updated so I'd suggest that you run Loadgen manually against that file and see why it's crashing.
The process time counter is already there by default btw.
Joshua
I can see those counters listed in the legend now. Loadgen is showing the tasks completing w/o exceptions, and perfmon shows them at 100% completing. I think the overload I saw in the previous run may have been the result of my monkeying with the constraints of the test run. There's about 5 minutes left and the queue length is 0, I don't think it's going to fault out.
So if Loadgen isn't getting started by Client1 (I assume), what should I be looking at/for?
Thanks,
-Kirk
So are you saying that everything started working fine after you ran loadgen manually with the vmmark2-mailserver-1.xml file?
If so, did you have to change anything beyond time settings?
Based on the timestamps, is Loadgen creating any new files when the harness starts a run?
That is correct, when I didn't alter the parameters on the local execution, it ran correctly w/o errors. That's why I wasn't too concerned earlier, when I had altered them and it overloaded...figured it was the fault of my monkeying with the run parameters.
I'm watching a run right now. When the harness began mailserver1 actions, I saw the timestamp of the vmmark2-mailserver-1 file change to reflect that activity. I see a LoadGenRemoteSvc process running on client1. Client0 shows LoadGenRemoteSvc and LoadGen processes running. Mailserver0 has activity, mailserver1 does not.
-Kirk
On client1, take a look a the C;\Program Files*\Exchange Load Generator folder. See if Loadgen is creating new files in that directory when the harness starts a run. You should be able to match the timestamps and the look in the files to see what's going on.
The newest files are those coincident with the local run of LoadGen, nothing newer matching the latest harness run. In fact, there appears to be three sets of files. The first, I think, is associated with building the database yesterday, on this new client. The second if the 'overloaded' run of LoadGen earlier. And the third is the passing run of LoadGen.
-Kirk
OK, so what happens if you start a command prompt on your client1 and type
"dir c:\Progra~1\Exchan~1\LoadGenCmd.exe"
I get a listing of LoadGen.
OK, so on the prime client start up a cygwin prompt and type the following:
staf client1 process start command "c:\\Progra~1\\Excha~1\\LoadGenCmd.exe /r /c C:\\vclient1\\mailserver\\vmmark2-mailserver-1.xml" returnstdout stderrtostdout wait
I get this:
There was syntax error in the path string (Excha~1 => Exchan~1). Correcting it, I got:
The problem is that the user that is running staf on client1 is not part of the Domain Admin group. Make sure the staf process is started by the domain administrator. The local administrator can work also, but I have seen some issues with it. Usually the local administrator account is automatically added to the Domain Admin group, but sometimes it doesn't seem to work. If you verify that staf is started by the domain administrator and it is still not working try rejoining the domain on that client.