psphere: how to make thread safe

[sajuptpm]

Hi,

psphere: Python interface for the VMware vSphere Web Services SDK

I already developed an app using https://bitbucket.org/jkinred/psphere. Butgetting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that cachingmechanism used by psphere is not thread safe. Could someone please give mesome suggestion to make it thread safe.


=======Test Code ========

import psphere
from psphere.client import Client
from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
client = Client("192.168.0.114", "root", "vmware1") ##vCenter
print "\nSucessfully connected to vCenter.\n"

from threading import Thread

def myfunc(i):
host1 = HostSystem.get(client, name="192.168.0.134")
host2 = HostSystem.get(client, name="192.168.0.113")
print "----i------",i
while True:
#host1.update(properties=["config", "vm"])
#host2.update(properties=["config", "vm"])
c = type(host1.config.network)
v = type(host2.config.network)
for vm in host1.vm:
k = vm.config.template
for vm in host2.vm:
p = vm.config.template


for i in range(10):
t = Thread(target=myfunc, args=(i,))
t.start()


"""
OUTPUT
=======
Sucessfully connected to vCenter.

----i------ 1
----i------ 3
----i------ 5
----i------ 0
----i------ 4
----i------ 2----i------ 7
----i------
9
----i------ 6
----i------ 8
Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "vcenter_test1.py", line 19, in myfunc
k = vm.config.template
File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79,in __get__
value = self.fget(inst)
File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
return self._get_dataobject("config", False)
File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
return self._cache[name][0]
KeyError: 'config'

Exception in thread Thread-6:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "vcenter_test1.py", line 17, in myfunc
v = type(host2.config.network)
AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'



"""

 

[Ramchandra Apte]

Hi,



psphere: Python interface for the VMware vSphere Web Services SDK



I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.





=======Test Code ========



import psphere

from psphere.client import Client

from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource

client = Client("192.168.0.114", "root", "vmware1") ##vCenter

print "\nSucessfully connected to vCenter.\n"



from threading import Thread



def myfunc(i):

host1 = HostSystem.get(client, name="192.168.0.134")

host2 = HostSystem.get(client, name="192.168.0.113")

print "----i------",i

while True:

#host1.update(properties=["config", "vm"])

#host2.update(properties=["config", "vm"])

c = type(host1.config.network)

v = type(host2.config.network)

for vm in host1.vm:

k = vm.config.template

for vm in host2.vm:

p = vm.config.template





for i in range(10):

t = Thread(target=myfunc, args=(i,))

t.start()





"""

OUTPUT

=======

Sucessfully connected to vCenter.



----i------ 1

----i------ 3

----i------ 5

----i------ 0

----i------ 4

----i------ 2----i------ 7

----i------

9

----i------ 6

----i------ 8

Exception in thread Thread-4:

Traceback (most recent call last):

File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

self.run()

File "/usr/lib/python2.7/threading.py", line 504, in run

self.__target(*self.__args, **self.__kwargs)

File "vcenter_test1.py", line 19, in myfunc

k = vm.config.template

File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__

value = self.fget(inst)

File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config

return self._get_dataobject("config", False)

File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject

return self._cache[name][0]

KeyError: 'config'



Exception in thread Thread-6:

Traceback (most recent call last):

File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

self.run()

File "/usr/lib/python2.7/threading.py", line 504, in run

self.__target(*self.__args, **self.__kwargs)

File "vcenter_test1.py", line 17, in myfunc

v = type(host2.config.network)

AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'







"""

use locks please on the cache

 

[jonathan.kinred]

Hi,



psphere: Python interface for the VMware vSphere Web Services SDK



I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.





=======Test Code ========



import psphere

from psphere.client import Client

from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource

client = Client("192.168.0.114", "root", "vmware1") ##vCenter

print "\nSucessfully connected to vCenter.\n"



from threading import Thread



def myfunc(i):

host1 = HostSystem.get(client, name="192.168.0.134")

host2 = HostSystem.get(client, name="192.168.0.113")

print "----i------",i

while True:

#host1.update(properties=["config", "vm"])

#host2.update(properties=["config", "vm"])

c = type(host1.config.network)

v = type(host2.config.network)

for vm in host1.vm:

k = vm.config.template

for vm in host2.vm:

p = vm.config.template





for i in range(10):

t = Thread(target=myfunc, args=(i,))

t.start()





"""

OUTPUT

=======

Sucessfully connected to vCenter.



----i------ 1

----i------ 3

----i------ 5

----i------ 0

----i------ 4

----i------ 2----i------ 7

----i------

9

----i------ 6

----i------ 8

Exception in thread Thread-4:

Traceback (most recent call last):

File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

self.run()

File "/usr/lib/python2.7/threading.py", line 504, in run

self.__target(*self.__args, **self.__kwargs)

File "vcenter_test1.py", line 19, in myfunc

k = vm.config.template

File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__

value = self.fget(inst)

File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config

return self._get_dataobject("config", False)

File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject

return self._cache[name][0]

KeyError: 'config'



Exception in thread Thread-6:

Traceback (most recent call last):

File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

self.run()

File "/usr/lib/python2.7/threading.py", line 504, in run

self.__target(*self.__args, **self.__kwargs)

File "vcenter_test1.py", line 17, in myfunc

v = type(host2.config.network)

AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'







"""

I'm the author of psphere. I'd be really interested to get some experiencedinsight into this as I really don't know the first thing about thread safety.

psphere does use a cache, but I'm confused as to how the cache would be thread unsafe as the cache is just a property of an object.

What I'm suspecting (and this only came to me while writing this post) is that I have a "client" object which is passed into each object and used to retrieve data from the server. See this gist:
https://gist.github.com/3478641

When used in a threaded scenario could the client be used by multiple objects simultaneously and the replies would come in out of order?

A couple of years ago I started psphere to learn Python and I always hoped that I would have an experienced developer join me. I did learn Python but I never got that experienced developer so when it comes to issues like thisI'm definitely out of my depth! Please if someone could spend the time with me I would really appreciate it!

 

[jonathan.kinred]

Hi,
psphere: Python interface for the VMware vSphere Web Services SDK
I already developed an app using https://bitbucket.org/jkinred/psphere.But getting lot of errors since psphere is not thread safe (I think). So iwrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.
=======Test Code ========
import psphere
from psphere.client import Client
from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
client = Client("192.168.0.114", "root", "vmware1") ##vCenter
print "\nSucessfully connected to vCenter.\n"
from threading import Thread
def myfunc(i):
host1 = HostSystem.get(client, name="192.168.0.134")
host2 = HostSystem.get(client, name="192.168.0.113")
print "----i------",i
while True:
#host1.update(properties=["config", "vm"])
#host2.update(properties=["config", "vm"])
c = type(host1.config.network)
v = type(host2.config.network)
for vm in host1.vm:
k = vm.config.template
for vm in host2.vm:
p = vm.config.template
for i in range(10):
t = Thread(target=myfunc, args=(i,))

t.start()

"""

OUTPUT

=======

Sucessfully connected to vCenter.
----i------ 1
----i------ 3
----i------ 5
----i------ 0
----i------ 4
----i------ 2----i------ 7
----i------

9

----i------ 6
----i------ 8
Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

self.run()

File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "vcenter_test1.py", line 19, in myfunc
k = vm.config.template
File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line79, in __get__
value = self.fget(inst)
File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
return self._get_dataobject("config", False)
File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line116, in _get_dataobject
return self._cache[name][0]
KeyError: 'config'
Exception in thread Thread-6:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

self.run()

File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "vcenter_test1.py", line 17, in myfunc
v = type(host2.config.network)
AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'
"""

I'm the author of psphere. I'd be really interested to get some experienced insight into this as I really don't know the first thing about thread safety.



psphere does use a cache, but I'm confused as to how the cache would be thread unsafe as the cache is just a property of an object.



What I'm suspecting (and this only came to me while writing this post) isthat I have a "client" object which is passed into each object and used toretrieve data from the server. See this gist:

https://gist.github.com/3478641



When used in a threaded scenario could the client be used by multiple objects simultaneously and the replies would come in out of order?



A couple of years ago I started psphere to learn Python and I always hoped that I would have an experienced developer join me. I did learn Python but I never got that experienced developer so when it comes to issues like this I'm definitely out of my depth! Please if someone could spend the time with me I would really appreciate it!

Just to clarify, the real Client is here:
https://github.com/jkinred/psphere/blob/master/psphere/client.py#L45