psphere: how to make thread safe

Discussion in 'Python' started by sajuptpm, Aug 22, 2012.

  1. sajuptpm

    sajuptpm Guest

    Hi,

    psphere: Python interface for the VMware vSphere Web Services SDK

    I already developed an app using https://bitbucket.org/jkinred/psphere. Butgetting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that cachingmechanism used by psphere is not thread safe. Could someone please give mesome suggestion to make it thread safe.


    =======Test Code ========

    import psphere
    from psphere.client import Client
    from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
    client = Client("192.168.0.114", "root", "vmware1") ##vCenter
    print "\nSucessfully connected to vCenter.\n"

    from threading import Thread

    def myfunc(i):
    host1 = HostSystem.get(client, name="192.168.0.134")
    host2 = HostSystem.get(client, name="192.168.0.113")
    print "----i------",i
    while True:
    #host1.update(properties=["config", "vm"])
    #host2.update(properties=["config", "vm"])
    c = type(host1.config.network)
    v = type(host2.config.network)
    for vm in host1.vm:
    k = vm.config.template
    for vm in host2.vm:
    p = vm.config.template


    for i in range(10):
    t = Thread(target=myfunc, args=(i,))
    t.start()


    """
    OUTPUT
    =======
    Sucessfully connected to vCenter.

    ----i------ 1
    ----i------ 3
    ----i------ 5
    ----i------ 0
    ----i------ 4
    ----i------ 2----i------ 7
    ----i------
    9
    ----i------ 6
    ----i------ 8
    Exception in thread Thread-4:
    Traceback (most recent call last):
    File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
    File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
    File "vcenter_test1.py", line 19, in myfunc
    k = vm.config.template
    File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79,in __get__
    value = self.fget(inst)
    File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
    return self._get_dataobject("config", False)
    File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
    return self._cache[name][0]
    KeyError: 'config'

    Exception in thread Thread-6:
    Traceback (most recent call last):
    File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
    File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
    File "vcenter_test1.py", line 17, in myfunc
    v = type(host2.config.network)
    AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'



    """
    sajuptpm, Aug 22, 2012
    #1
    1. Advertising

  2. On Wednesday, 22 August 2012 17:33:48 UTC+5:30, sajuptpm wrote:
    > Hi,
    >
    >
    >
    > psphere: Python interface for the VMware vSphere Web Services SDK
    >
    >
    >
    > I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.
    >
    >
    >
    >
    >
    > =======Test Code ========
    >
    >
    >
    > import psphere
    >
    > from psphere.client import Client
    >
    > from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
    >
    > client = Client("192.168.0.114", "root", "vmware1") ##vCenter
    >
    > print "\nSucessfully connected to vCenter.\n"
    >
    >
    >
    > from threading import Thread
    >
    >
    >
    > def myfunc(i):
    >
    > host1 = HostSystem.get(client, name="192.168.0.134")
    >
    > host2 = HostSystem.get(client, name="192.168.0.113")
    >
    > print "----i------",i
    >
    > while True:
    >
    > #host1.update(properties=["config", "vm"])
    >
    > #host2.update(properties=["config", "vm"])
    >
    > c = type(host1.config.network)
    >
    > v = type(host2.config.network)
    >
    > for vm in host1.vm:
    >
    > k = vm.config.template
    >
    > for vm in host2.vm:
    >
    > p = vm.config.template
    >
    >
    >
    >
    >
    > for i in range(10):
    >
    > t = Thread(target=myfunc, args=(i,))
    >
    > t.start()
    >
    >
    >
    >
    >
    > """
    >
    > OUTPUT
    >
    > =======
    >
    > Sucessfully connected to vCenter.
    >
    >
    >
    > ----i------ 1
    >
    > ----i------ 3
    >
    > ----i------ 5
    >
    > ----i------ 0
    >
    > ----i------ 4
    >
    > ----i------ 2----i------ 7
    >
    > ----i------
    >
    > 9
    >
    > ----i------ 6
    >
    > ----i------ 8
    >
    > Exception in thread Thread-4:
    >
    > Traceback (most recent call last):
    >
    > File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    >
    > self.run()
    >
    > File "/usr/lib/python2.7/threading.py", line 504, in run
    >
    > self.__target(*self.__args, **self.__kwargs)
    >
    > File "vcenter_test1.py", line 19, in myfunc
    >
    > k = vm.config.template
    >
    > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__
    >
    > value = self.fget(inst)
    >
    > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
    >
    > return self._get_dataobject("config", False)
    >
    > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
    >
    > return self._cache[name][0]
    >
    > KeyError: 'config'
    >
    >
    >
    > Exception in thread Thread-6:
    >
    > Traceback (most recent call last):
    >
    > File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    >
    > self.run()
    >
    > File "/usr/lib/python2.7/threading.py", line 504, in run
    >
    > self.__target(*self.__args, **self.__kwargs)
    >
    > File "vcenter_test1.py", line 17, in myfunc
    >
    > v = type(host2.config.network)
    >
    > AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'
    >
    >
    >
    >
    >
    >
    >
    > """


    use locks please on the cache
    Ramchandra Apte, Aug 24, 2012
    #2
    1. Advertising

  3. sajuptpm

    Guest

    On Wednesday, 22 August 2012 22:03:48 UTC+10, sajuptpm wrote:
    > Hi,
    >
    >
    >
    > psphere: Python interface for the VMware vSphere Web Services SDK
    >
    >
    >
    > I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.
    >
    >
    >
    >
    >
    > =======Test Code ========
    >
    >
    >
    > import psphere
    >
    > from psphere.client import Client
    >
    > from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
    >
    > client = Client("192.168.0.114", "root", "vmware1") ##vCenter
    >
    > print "\nSucessfully connected to vCenter.\n"
    >
    >
    >
    > from threading import Thread
    >
    >
    >
    > def myfunc(i):
    >
    > host1 = HostSystem.get(client, name="192.168.0.134")
    >
    > host2 = HostSystem.get(client, name="192.168.0.113")
    >
    > print "----i------",i
    >
    > while True:
    >
    > #host1.update(properties=["config", "vm"])
    >
    > #host2.update(properties=["config", "vm"])
    >
    > c = type(host1.config.network)
    >
    > v = type(host2.config.network)
    >
    > for vm in host1.vm:
    >
    > k = vm.config.template
    >
    > for vm in host2.vm:
    >
    > p = vm.config.template
    >
    >
    >
    >
    >
    > for i in range(10):
    >
    > t = Thread(target=myfunc, args=(i,))
    >
    > t.start()
    >
    >
    >
    >
    >
    > """
    >
    > OUTPUT
    >
    > =======
    >
    > Sucessfully connected to vCenter.
    >
    >
    >
    > ----i------ 1
    >
    > ----i------ 3
    >
    > ----i------ 5
    >
    > ----i------ 0
    >
    > ----i------ 4
    >
    > ----i------ 2----i------ 7
    >
    > ----i------
    >
    > 9
    >
    > ----i------ 6
    >
    > ----i------ 8
    >
    > Exception in thread Thread-4:
    >
    > Traceback (most recent call last):
    >
    > File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    >
    > self.run()
    >
    > File "/usr/lib/python2.7/threading.py", line 504, in run
    >
    > self.__target(*self.__args, **self.__kwargs)
    >
    > File "vcenter_test1.py", line 19, in myfunc
    >
    > k = vm.config.template
    >
    > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__
    >
    > value = self.fget(inst)
    >
    > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
    >
    > return self._get_dataobject("config", False)
    >
    > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
    >
    > return self._cache[name][0]
    >
    > KeyError: 'config'
    >
    >
    >
    > Exception in thread Thread-6:
    >
    > Traceback (most recent call last):
    >
    > File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    >
    > self.run()
    >
    > File "/usr/lib/python2.7/threading.py", line 504, in run
    >
    > self.__target(*self.__args, **self.__kwargs)
    >
    > File "vcenter_test1.py", line 17, in myfunc
    >
    > v = type(host2.config.network)
    >
    > AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'
    >
    >
    >
    >
    >
    >
    >
    > """


    I'm the author of psphere. I'd be really interested to get some experiencedinsight into this as I really don't know the first thing about thread safety.

    psphere does use a cache, but I'm confused as to how the cache would be thread unsafe as the cache is just a property of an object.

    What I'm suspecting (and this only came to me while writing this post) is that I have a "client" object which is passed into each object and used to retrieve data from the server. See this gist:
    https://gist.github.com/3478641

    When used in a threaded scenario could the client be used by multiple objects simultaneously and the replies would come in out of order?

    A couple of years ago I started psphere to learn Python and I always hoped that I would have an experienced developer join me. I did learn Python but I never got that experienced developer so when it comes to issues like thisI'm definitely out of my depth! Please if someone could spend the time with me I would really appreciate it!
    , Aug 26, 2012
    #3
  4. sajuptpm

    Guest

    On Sunday, 26 August 2012 22:45:25 UTC+10, wrote:
    > On Wednesday, 22 August 2012 22:03:48 UTC+10, sajuptpm wrote:
    >
    > > Hi,

    >
    > >

    >
    > >

    >
    > >

    >
    > > psphere: Python interface for the VMware vSphere Web Services SDK

    >
    > >

    >
    > >

    >
    > >

    >
    > > I already developed an app using https://bitbucket.org/jkinred/psphere.But getting lot of errors since psphere is not thread safe (I think). So iwrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > > =======Test Code ========

    >
    > >

    >
    > >

    >
    > >

    >
    > > import psphere

    >
    > >

    >
    > > from psphere.client import Client

    >
    > >

    >
    > > from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource

    >
    > >

    >
    > > client = Client("192.168.0.114", "root", "vmware1") ##vCenter

    >
    > >

    >
    > > print "\nSucessfully connected to vCenter.\n"

    >
    > >

    >
    > >

    >
    > >

    >
    > > from threading import Thread

    >
    > >

    >
    > >

    >
    > >

    >
    > > def myfunc(i):

    >
    > >

    >
    > > host1 = HostSystem.get(client, name="192.168.0.134")

    >
    > >

    >
    > > host2 = HostSystem.get(client, name="192.168.0.113")

    >
    > >

    >
    > > print "----i------",i

    >
    > >

    >
    > > while True:

    >
    > >

    >
    > > #host1.update(properties=["config", "vm"])

    >
    > >

    >
    > > #host2.update(properties=["config", "vm"])

    >
    > >

    >
    > > c = type(host1.config.network)

    >
    > >

    >
    > > v = type(host2.config.network)

    >
    > >

    >
    > > for vm in host1.vm:

    >
    > >

    >
    > > k = vm.config.template

    >
    > >

    >
    > > for vm in host2.vm:

    >
    > >

    >
    > > p = vm.config.template

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > > for i in range(10):

    >
    > >

    >
    > > t = Thread(target=myfunc, args=(i,))

    >
    > >

    >
    > > t.start()

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > > """

    >
    > >

    >
    > > OUTPUT

    >
    > >

    >
    > > =======

    >
    > >

    >
    > > Sucessfully connected to vCenter.

    >
    > >

    >
    > >

    >
    > >

    >
    > > ----i------ 1

    >
    > >

    >
    > > ----i------ 3

    >
    > >

    >
    > > ----i------ 5

    >
    > >

    >
    > > ----i------ 0

    >
    > >

    >
    > > ----i------ 4

    >
    > >

    >
    > > ----i------ 2----i------ 7

    >
    > >

    >
    > > ----i------

    >
    > >

    >
    > > 9

    >
    > >

    >
    > > ----i------ 6

    >
    > >

    >
    > > ----i------ 8

    >
    > >

    >
    > > Exception in thread Thread-4:

    >
    > >

    >
    > > Traceback (most recent call last):

    >
    > >

    >
    > > File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

    >
    > >

    >
    > > self.run()

    >
    > >

    >
    > > File "/usr/lib/python2.7/threading.py", line 504, in run

    >
    > >

    >
    > > self.__target(*self.__args, **self.__kwargs)

    >
    > >

    >
    > > File "vcenter_test1.py", line 19, in myfunc

    >
    > >

    >
    > > k = vm.config.template

    >
    > >

    >
    > > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line79, in __get__

    >
    > >

    >
    > > value = self.fget(inst)

    >
    > >

    >
    > > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config

    >
    > >

    >
    > > return self._get_dataobject("config", False)

    >
    > >

    >
    > > File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line116, in _get_dataobject

    >
    > >

    >
    > > return self._cache[name][0]

    >
    > >

    >
    > > KeyError: 'config'

    >
    > >

    >
    > >

    >
    > >

    >
    > > Exception in thread Thread-6:

    >
    > >

    >
    > > Traceback (most recent call last):

    >
    > >

    >
    > > File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

    >
    > >

    >
    > > self.run()

    >
    > >

    >
    > > File "/usr/lib/python2.7/threading.py", line 504, in run

    >
    > >

    >
    > > self.__target(*self.__args, **self.__kwargs)

    >
    > >

    >
    > > File "vcenter_test1.py", line 17, in myfunc

    >
    > >

    >
    > > v = type(host2.config.network)

    >
    > >

    >
    > > AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > >

    >
    > > """

    >
    >
    >
    > I'm the author of psphere. I'd be really interested to get some experienced insight into this as I really don't know the first thing about thread safety.
    >
    >
    >
    > psphere does use a cache, but I'm confused as to how the cache would be thread unsafe as the cache is just a property of an object.
    >
    >
    >
    > What I'm suspecting (and this only came to me while writing this post) isthat I have a "client" object which is passed into each object and used toretrieve data from the server. See this gist:
    >
    > https://gist.github.com/3478641
    >
    >
    >
    > When used in a threaded scenario could the client be used by multiple objects simultaneously and the replies would come in out of order?
    >
    >
    >
    > A couple of years ago I started psphere to learn Python and I always hoped that I would have an experienced developer join me. I did learn Python but I never got that experienced developer so when it comes to issues like this I'm definitely out of my depth! Please if someone could spend the time with me I would really appreciate it!


    Just to clarify, the real Client is here:
    https://github.com/jkinred/psphere/blob/master/psphere/client.py#L45
    , Aug 26, 2012
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Francis Reed
    Replies:
    2
    Views:
    716
    Erik Funkenbusch
    Apr 8, 2006
  2. Irmen de Jong
    Replies:
    5
    Views:
    376
    Irmen de Jong
    Sep 28, 2004
  3. Gabriel Rossetti
    Replies:
    0
    Views:
    1,300
    Gabriel Rossetti
    Aug 29, 2008
  4. Ian Collins

    Re: How to Easily Make a Function Thread-Safe

    Ian Collins, Jan 12, 2009, in forum: C Programming
    Replies:
    4
    Views:
    873
    Phil Carmody
    Jan 13, 2009
  5. John Nagle
    Replies:
    5
    Views:
    454
    John Nagle
    Mar 12, 2012
Loading...

Share This Page