Hello
I am testing VIO. and I have deployed VIO with NSX, and created network , uploaded the image and created instance.
Most are successful. but unable to create snapshot.
I attach log file(compute, controller0 controller1) and VIO configuration screen shot .
please let me know what i do.
You mention you are using NSX. Would you mind sharing your VMware NSX contact.
What version of the product are you using? 1.0 , 1.0.1 or 1.0.2 ?
I will open a bug to investigate the issue. Will post back if we need more info.
arvind
How big is the instance that you try to snapshot (disk size)? Do you the same problem with smaller instances (e.g. 1GB disk)?
I'm facing the same issue. After a few moments the snapshot status is displayed with "killed".
The only time it worked was when I created a new flavor with the minimum requisites of the ubuntu image (512 RAM and 5 GB disk).
I'd one problem with another openstack installation (not the VIO), where the snapshot took so long that the token expired during the process, but it doesn't seem related here since the process is stopped in a few minutes.
NSX version is 6.1.3
This error is because of nsx??
I tested two instance(ubuntu 14.04 and Win7)
but all failed
Thanks
You can try to workaround the problem by increasing the timeout on each ESX host and each load balancer.
To increase the HTTP timeout on a ESX host:
1. SSH into ESX
2. Open /etc/vmware/rhttpproxy/config.xml for editing
3. Set 'readTimeoutMs' to 300000ms like this:
<vmacore>
...
<http>
...
<readTimeoutMs>300000</readTimeoutMs>
</http>
...
</vmacore>
4. Restart the rhttpproxy process:
# /etc/init.d/rhttpproxy restart
To increase the timeout on a loadbalancer:
1. SSH into loadbalancer
2. Open /etc/haproxy/haproxy.cfg for editing
3. Add the following to the 'glance-api' section:
listen glance-api
timeout client 300000ms
timeout server 300000ms
...
4. Restart the haproxy service:
# /etc/init.d/haproxy restart
Had the same issue here, but with the workaround it works for most images. Thank you rgerganov.
But with the ova from Ubuntu 14.04.2 LTS (Trusty Tahr) Daily Build [20150702] I get the Status "killed". Does this make sense, is this a issue with the image? Can someone confirm this?
I tested as you lead me but result is killed.
In addition to the rhttproxy timeout on the ESX hosts as suggested by rgerganov, you also need to increase the timeout value on the hostd configuration.
Host Configuration Change
1. SSH to each ESXi Host in the environment.
2. Backup both /etc/vmware/rhttpproxy/config.xml and /etc/hostd/config.xml. Edit the original config.xml files with same content changes described earlier i.e
<vmacore>
…
<http>
…
<readTimeoutMs>600000</readTimeoutMs>
…
</http>
…
<ssl>
…
<handshakeTimeoutMs>600000</handshakeTimeoutMs>
…
</ssl>
</vmacore>
2. /etc/init.d/rhttpproxy restart
3. /etc/init.d/hostd restart
Thanks
Sabari, it worked for me.
I'm facing the same problem using NSX-V (Version: 6.2.0 Build 2986609)
I could create snapshot at compute node, but I received 500 error at uploading. Here is the syslog error-log from compute node.
Jan 8 07:40:02 compute01 2016-01-08 07:40:02.074 8039 ERROR oslo_messaging.rpc.dispatcher [req-d6bf535d-0b89-47fb-95d6-c6c6ea6cbe11 c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Exception during message handling: <html>#012 <head>#012 <title>500 Internal Server Error</title>#012 </head>#012 <body>#012 <h1>500 Internal Server Error</h1>#012 Failed to upload image fe9ec08c-63ca-4016-82ad-f4ff54fb4f60<br /><br />#012#012 </body>#012</html> (HTTP 500)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher executor_callback))#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher executor_callback)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6891, in snapshot_instance#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return self.manager.snapshot_instance(ctxt, image_id, instance)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher payload)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 346, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 317, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 374, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 362, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 422, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher instance=instance)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 412, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher *args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3304, in snapshot_instance#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher task_states.IMAGE_SNAPSHOT)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3334, in _snapshot_instance#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher update_task_state)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 527, in snapshot#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher self._vmops.snapshot(context, instance, image_id, update_task_state)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 1067, in snapshot#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher vmdk_size=vmdk.capacity_in_bytes)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/images.py", line 479, in upload_image_stream_optimized#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher IMAGE_API.update(context, image_id, image_metadata, data=read_handle)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 130, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher purge_props=purge_props)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 398, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher _reraise_translated_image_exception(image_id)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 396, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher image_id, **image_meta)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 219, in call#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return getattr(client.images, method)(*args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 329, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher resp, body = self.client.put(url, headers=hdrs, data=image_data)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 265, in put#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return self._request('PUT', url, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", l
And following the error log of Glance server (glance-api.log) when this problem is occurred.
2016-01-08 07:39:59.094 7331 ERROR glance.api.v1.upload_utils [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Failed to upload image fe9ec08c-63ca-4016-82ad-f4ff54fb4f60
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils context=req.context)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 339, in store_add_to_backend
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils context=context)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance_store/capabilities.py", line 226, in op_checker
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils return store_op_fun(store, *args, **kwargs)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/vmware_datastore.py", line 516, in add
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils response = session.put(url, data=data, headers=headers)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 510, in put
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils return self.request('PUT', url, data=data, **kwargs)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils resp = self.send(prep, **send_kwargs)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils r = adapter.send(request, **kwargs)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 389, in send
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils r = low_conn.getresponse()
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils response.begin()
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 415, in begin
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils version, status, reason = self._read_status()
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils raise BadStatusLine(line)
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils BadStatusLine: ''
2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils
2016-01-08 07:39:59.099 7331 DEBUG glance.registry.client.v1.api [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Updating image metadata for image fe9ec08c-63ca-4016-82ad-f4ff54fb4f60... update_image_metadata /usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py:168
2016-01-08 07:39:59.099 7331 DEBUG glance.common.client [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Constructed URL: http://192.168.84.30:9191/images/fe9ec08c-63ca-4016-82ad-f4ff54fb4f60 _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:401
2016-01-08 07:39:59.134 7331 DEBUG glance.registry.client.v1.client [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Registry request PUT /images/fe9ec08c-63ca-4016-82ad-f4ff54fb4f60 HTTP 200 request id req-req-9581d029-e8b2-471e-845c-10647f42db45 do_request /usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py:124
2016-01-08 07:39:59.135 7331 INFO eventlet.wsgi.server [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] 192.168.84.45,192.168.84.42 - - [08/Jan/2016 07:39:59] "PUT /v1/images/fe9ec08c-63ca-4016-82ad-f4ff54fb4f60 HTTP/1.1" 500 430 620.496563