I am new to VIO and am deploying it as a part of my master' project. I have the VIO vApp running on Vcenter. After logging in to management server I am able to ping Vcenter and Vsphere(esxi) . I can see my openstack extensions in Vcenter. I can successfully ping the Vcenter and Vsphere from my PC but I am not able to ping the management server and hence cant open the webpage. Like wise I cant ping the management server from esxi or vcenter ?
Can anyone provide some inputs on this please
icmp echo is disabled on OMS, so you won't get response to ping. Try ssh to the management server.
I am able to SSH to the server. But, without my PC able to ping the server how do I open the webpage ? I need that so I can run the below resolution to fix the plugin issue
For VMware Integrated OpenStack versions 1.0.2 and later, a new interface is available for managing vCenter Server registration if plug-in issues are encountered. Please use this procedure to remedy plug-in issues:
icmp and http are different. If you the http url you mentioned does not work, then it could be different issue.
Can you ssh to OMS and run
sudo service oms status
The response should be
"oms start/running, process xxxx"
May I know the actual issue?
Are you not able to see the VIO web plugin on vcenter?
Did you try logout and login vsphere web client?
viouser@localhost:~$ sudo service oms status
[sudo] password for viouser:
oms stop/waiting
How can I start the oms ?
Yup I logged out and logged in back but was not able to see the plugin. I found the resolution in Vmware docs , but couldn't access the webpage to resolve it ? I tried trouble shooting with ping and found that my PC wont communicate with the management server.
root@localhost:~# service oms start
oms start/running, process 1560
I got it to start
My OMS.log only shows the below error
root@localhost:/var/log/oms# grep failed oms.log
[2017-01-27T16:23:06.213+0000] INFO localhost-startStop-1| org.hibernate.tool.hbm2ddl.TableMetadata: HHH000037: Columns: [error_message, vc_rp_id, rack, vc_datastores, memory, volumes, cpu_number, action_failed, version, moid, node_group_id, action, guest_host_name, id, vm_name, host_name, status, power_status_changed]
This is my boot sequence log
root@localhost:/var/log# more bootsequence.log
Fri Jan 27 12:13:35 UTC 2017 : Running firstboot scripts
Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//00-common
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
Creating SSH2 ED25519 key; this may take some time ...
ssh stop/waiting
ssh start/running, process 1469
Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//02-setup-user
passwd: password expiry information changed.
groupadd: group 'viogrp' already exists
Group viogrp exists!
Generating public/private rsa key pair.
Your identification has been saved in /home/viouser/.ssh/id_rsa.
Your public key has been saved in /home/viouser/.ssh/id_rsa.pub.
The key fingerprint is:
3b:b5:d4:50:ed:c3:89:fd:3d:20:96:05:20:c6:f5:ca root@localhost.localdom
The key's randomart image is:
+--[ RSA 2048]----+
| .o.o..o. |
| ... .. .. |
| ..o= . |
| . .*..* |
| SE+ o .o.|
| + . .o|
| o . .|
| . |
| |
+-----------------+
Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//20-generate-keys
1. Generating keystore for OMS server...
Exporting self-signed certificate...
Certificate stored in file </opt/vmware/vio/etc/oms.crt>
Generating the csr...
Exporting pkcs12 store...
Exporting private key...
MAC verified OK
2. Generating password guard keystore...
generated certificate and private key in /opt/vmware/vio/etc
Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//30-generate-vault-pass
Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//48-configure_rsyslog
Running commands from firstboot script /opt/vmware/vio/sysctl/firstboot.d//50-clean-logs
Fri Jan 27 12:13:45 UTC 2017 : Running everyboot scripts
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp
* Stopping NTP server ntpd
...done.
* Starting NTP server ntpd
...done.
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context
Setting up execution env: export VC information to vc.properties
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete
Sending boot-complete signal to upstart
Fri Jan 27 12:25:45 UTC 2017 : Running everyboot scripts
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp
* Stopping NTP server ntpd
...done.
* Starting NTP server ntpd
...done.
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context
Setting up execution env: export VC information to vc.properties
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete
Sending boot-complete signal to upstart
Fri Jan 27 12:48:07 UTC 2017 : Running everyboot scripts
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp
* Stopping NTP server ntpd
...done.
* Starting NTP server ntpd
...done.
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context
Setting up execution env: export VC information to vc.properties
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete
Sending boot-complete signal to upstart
Fri Jan 27 13:00:01 UTC 2017 : Running everyboot scripts
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp
* Stopping NTP server ntpd
...done.
* Starting NTP server ntpd
...done.
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//71-update-exec-context
Setting up execution env: export VC information to vc.properties
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//75-configure-tc-oms
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//76-configure-tc-osvmw
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//100-boot-complete
Sending boot-complete signal to upstart
Fri Jan 27 13:28:29 UTC 2017 : Running everyboot scripts
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp
Unable to find the ovf environment.
Fri Jan 27 16:06:28 UTC 2017 : Running everyboot scripts
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//00-common
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//55-set-vami-guestinfo
Running commands from everyboot script /opt/vmware/vio/sysctl/everyboot.d//70-configure_ntp
Unable to find the ovf environment.
Did you enter the password in ovf properties while deploying the OVF? Also, I'm not sure if the messages can cause issues to bring up OMS.
After oms service was restarted, please logout and login on vsphere client.
If none of the issues solved, can you please upload full oms.log file?
I see ssl error from the log.
[2017-01-27T16:47:36.905+0000] INFO localhost-startStop-1| com.vmware.aurora.vc.vcservice.VcService: VC login on behalf of {localhost-startStop-1:RegisterPlugin{1}} 112ms
[2017-01-27T16:47:36.936+0000] INFO localhost-startStop-1| com.vmware.openstack.manager.PatchManager: Package: vio-osvmw-plugin Version: 3.0.0.4345506
[2017-01-27T16:47:36.937+0000] INFO localhost-startStop-1| com.vmware.openstack.plugin.NgcRegistrar: Trying to update the old plugin...org.os.vmw.plugin
[2017-01-27T16:47:43.161+0000] INFO localhost-startStop-1| com.vmware.openstack.plugin.NgcRegistrar: org.os.vmw.plugin plugin updation succeeded!
[2017-01-27T16:47:43.174+0000] INFO localhost-startStop-1| com.vmware.openstack.plugin.NgcOsvmwRegistrar: OSVMW NGC plugin auto-installation succeeded!
[2017-01-27T16:47:43.177+0000] ERROR localhost-startStop-1| com.vmware.openstack.security.sso.utils.SecurityUtils: Authentication error :null
[2017-01-27T16:47:43.177+0000] ERROR localhost-startStop-1| com.vmware.openstack.manager.PluginRegisterManager: SSO server is misconfigured
[2017-01-27T16:47:43.181+0000] INFO localhost-startStop-1| com.vmware.openstack.service.resmgmt.impl.ResourceInitializerService: resource initialized? true
[2017-01-27T16:47:43.182+0000] INFO localhost-startStop-1| com.vmware.openstack.manager.PluginRegisterManager: Resource initialization completed
can you try the viocli deployment getlogs command and see if you hit error similar to https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21468...
If yes, then try following the instructions?
If no cert related issue, at least the following steps can be tried (which is subset of the instructions in kb article).
1. Stop oms services (do it as sudo)
service oms stop
service osvmw stop
2. backup /opt/vmware/vio/etc
3. Snapshot OMS
4. Modify /opt/vmware/vio/etc/omjs.properties and set "oms.extension.registered" to false to allow oms to register with VC with new certificate.
5. stop & start VIO vApp
6. Verify and remove snapshot
Also, always logout, give it couple of minutes and login to vsphere client whenever you restart oms.
Thanks i'll try it out, this is what the viocli log shows
root@localhost:/opt/vmware/vio/etc/test/201701281316/vio_mgmt/logs/viocli# more viocli.log
2017-01-28 13:16:34,681 viocli DEBUG ###### Starting execution of /usr/local/bin/viocli deployment getlogs ######
2017-01-28 13:16:34,681 jarvis.cache.file_system DEBUG Checking cache store directory /var/lib/vio/jarvis/cache exists
2017-01-28 13:16:35,065 p=1328 u=root | INFO PLAY [127.0.0.1] **************************************************************
2017-01-28 13:16:35,066 p=1328 u=root | INFO GATHERING FACTS ***************************************************************
2017-01-28 13:16:35,963 p=1328 u=root | INFO ok: [127.0.0.1]
2017-01-28 13:16:35,964 p=1328 u=root | INFO TASK: [support-local | create the logs directory] *****************************
2017-01-28 13:16:36,088 p=1328 u=root | INFO changed: [127.0.0.1]
2017-01-28 13:16:36,089 p=1328 u=root | INFO TASK: [support-local | create the configure directory] ************************
2017-01-28 13:16:36,212 p=1328 u=root | INFO changed: [127.0.0.1]
2017-01-28 13:16:36,213 p=1328 u=root | INFO TASK: [support-local | check log and configuration files existed] *************
2017-01-28 13:16:36,335 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/oms'))
2017-01-28 13:16:36,451 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/jarvis'))
2017-01-28 13:16:36,566 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/viocli'))
2017-01-28 13:16:36,677 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/viopatch'))
2017-01-28 13:16:36,790 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/opt/vmware/var/log/firstboot'))
2017-01-28 13:16:36,902 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/opt/vmware/var/log/subsequentboot'))
2017-01-28 13:16:37,018 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/syslog'))
2017-01-28 13:16:37,133 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/dpkg.log'))
2017-01-28 13:16:37,254 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/bootsequence.log'))
2017-01-28 13:16:37,376 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/opt/vmware/dbg/cores'))
2017-01-28 13:16:37,488 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/osvmw'))
2017-01-28 13:16:37,601 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/viomon'))
2017-01-28 13:16:37,712 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'logs'}, '/var/log/vmware/vapi'))
2017-01-28 13:16:37,824 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'config'}, '/opt/vmware/vio/etc/vc.properties'))
2017-01-28 13:16:37,937 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'config'}, '/opt/vmware/vio/etc/omjs.properties'))
2017-01-28 13:16:38,050 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/ntp.conf'))
2017-01-28 13:16:38,162 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/rsyslog.conf'))
2017-01-28 13:16:38,275 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/rsyslog.d'))
2017-01-28 13:16:38,388 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'config'}, '/var/lib/vio/jarvis'))
2017-01-28 13:16:38,501 p=1328 u=root | INFO ok: [127.0.0.1] => (item=({'dst': 'config'}, '/etc/hosts'))
2017-01-28 13:16:38,502 p=1328 u=root | INFO TASK: [support-local | copying syslog] ****************************************
2017-01-28 13:16:38,944 p=1328 u=root | INFO changed: [127.0.0.1] => (item={'invocation': {'module_name': u'stat', 'module_args': u'path=/var/log/oms'}, 'item': [{
'dst': 'logs'}, '/var/log/oms'], u'stat': {u'islnk': False, u'uid': 998, u'exists': True, u'rgrp': True, u'xoth': False, u'rusr': True, u'woth': False, u'nlink': 2, u'i
ssock': False, u'mtime': 1485534182.4886754, u'xusr': True, u'atime': 1485589948.0653641, u'inode': 1264212, u'isgid': False, u'size': 4096, u'isdir': True, u'ctime': 1
485534182.4886754, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 2054, u'roth': False, u'isreg': False, u'isfifo': False, u'mode': u'0750', u
'pw_name': u'omsapp', u'gid': 4, u'ischr': False, u'wusr': True}, u'changed': False})
2017-01-28 13:16:39,076 p=1328 u=root | INFO changed: [127.0.0.1] => (item={'invocation': {'module_name': u'stat', 'module_args': u'path=/var/log/jarvis'}, 'item':
[{'dst': 'logs'}, '/var/log/jarvis'], u'stat': {u'islnk': False, u'uid': 999, u'exists': True, u'rgrp': True, u'xoth': False, u'rusr': True, u'woth': False, u'nlink':
2, u'issock': False, u'mtime': 1473068110.3191955, u'xusr': True, u'atime': 1473068121.2070587, u'inode': 1288140, u'isgid': False, u'size': 4096, u'isdir': True, u'cti
me': 1473068110.3311954, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 2054, u'roth': False, u'isreg': False, u'isfifo': False, u'mode': u'07
50', u'pw_name': u'jarvis', u'gid': 4, u'ischr': False, u'wusr': True}, u'changed': False})
Also " Go to the vmware integrated openstack plugin and reconnect the OMS " . When I restarted the vAPP the OMS came up and the status was running . Do i have to do another step ? I cant find it in vcenter an option to connect OMS. My implementation is as below
Vsphere version 6.0 has Vcenter(VCSA 6.0) deployed on it and VIO 3.0 on vcenter .
In the plugin manager on vsphere i see nothing . Please find all the snaps attached
One last suggestion. Try unregistering the plugin as mentioned below and restart oms which will register again.
Removing VIO from vCenter Sever manually (2146541) | VMware KB