comparing two lists

[Ladislav Andel]

Hi,
what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.

Furthermore, I can have the opposite situation.
This time I need to remove from memlist a host which is not in dblist.
How would you do this?

The way how it works is that DBlist is loaded every 10 minutes and
compares with memlist.
The memlist should be the same as dblist.

Could you help me, please?
I'm working on my version of this but somebody might be quicker than me.
In case I have it done I will post it.

Thanks,
Lada






There is



and wt

 

[kyosohma]

Hi,
what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.

Furthermore, I can have the opposite situation.
This time I need to remove from memlist a host which is not in dblist.
How would you do this?

The way how it works is that DBlist is loaded every 10 minutes and
compares with memlist.
The memlist should be the same as dblist.

Could you help me, please?
I'm working on my version of this but somebody might be quicker than me.
In case I have it done I will post it.

Thanks,
Lada

There is

and wt

On lists that change in memory, I use the following looping structure:

for i in range(len(in_memory_list)-1,-1,-1)

This loops over the list backwards. I found this method here:
http://mail.python.org/pipermail/python-list/1999-September/012451.html

Hopefully this will fulfill your needs. I've used it to great effect!

Mike

 

[Ladislav Andel]

Hi,
what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.

Furthermore, I can have the opposite situation.
This time I need to remove from memlist a host which is not in dblist.
How would you do this?

The way how it works is that DBlist is loaded every 10 minutes and
compares with memlist.
The memlist should be the same as dblist.

Could you help me, please?
I'm working on my version of this but somebody might be quicker than me.
In case I have it done I will post it.

Thanks,
Lada

There is

and wt

On lists that change in memory, I use the following looping structure:

for i in range(len(in_memory_list)-1,-1,-1)

well, actually I will need to iterate over 2 sequences which are not the
same length
and synchronize it against the dblist i have.
But I will look at your post. Thanks a lot.

Lada

 

[Peter Otten]

what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.

You should replace the memlist with a dictionary using (host, id) tuples as
the keys. Here's an example that uses a set but requires you to modify the
<instance N> class:

dblist = [{'id':1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id':3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id':9, 'host':'msn.com','ip_address':'11.3.2.3'}]

class Item(object):
def __init__(self, id, host, **discarded):
self._tuple = (id, host)
def __hash__(self):
return hash(self._tuple)
def __eq__(self, other):
return self._tuple == other._tuple
def __repr__(self):
return "Item(id=%r, host=%r)" % self._tuple

items = set([Item(1, "google.com")])
for d in dblist:
item = Item(**d)
if item not in items:
print "adding", item
items.add(item)
else:
print item, "already there"

Peter

 

[Ladislav Andel]

Ladislav Andel wrote:

what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.

You should replace the memlist with a dictionary using (host, id) tuples as
the keys. Here's an example that uses a set but requires you to modify the
<instance N> class:

dblist = [{'id':1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id':3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id':9, 'host':'msn.com','ip_address':'11.3.2.3'}]

class Item(object):
def __init__(self, id, host, **discarded):
self._tuple = (id, host)
def __hash__(self):
return hash(self._tuple)
def __eq__(self, other):
return self._tuple == other._tuple
def __repr__(self):
return "Item(id=%r, host=%r)" % self._tuple

items = set([Item(1, "google.com")])
for d in dblist:
item = Item(**d)
if item not in items:
print "adding", item
items.add(item)
else:
print item, "already there"

Thank you for this nice solution. I wouldn't be able to write it this
way at all
but what about removing from memlist if there is less items in dblist
than in items (instances)?
I will have to iterate over items(instances) and remove that one which
is not in dblist I guess.

Lada

 

[Ladislav Andel]

Well, I know it's quite ugly what I did to the code, but any
improvements are welcome.
Actually, it doesn't work as it should yet.
The items should stay the same as at the beginning.
I say in one sentence what I want to achieve:
Synchronized items with dblist.
It should remove hosts from items which are not dblist
or it should add a new host from dblist which is not in items.
I'm sorry I'm not very skilled programmer.

What O notations(algorithm complexity) I introduced to it?
I know made it much slower this way.


dblist = [{'id':1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id':3, 'host':'msn.com','ip_address':'11.3.2.3'},
{'id':9, 'host':'yahoo.com','ip_address':'5.6.7.8'}]


class Item(object):
def __init__(self, id, host, **discarded):
self._tuple = (id, host)
def __hash__(self):
return hash(self._tuple)
def __eq__(self, other):
return self._tuple == other._tuple
def __repr__(self):
return "Item(id=%r, host=%r)" % self._tuple

def syncMemAndDBlist_1():
for d in dblist:
item = Item(**d)
if item not in items:
print "adding", item
items.add(item)
else:
print item, "already there"

def syncMemAndDBlist_2(a):
while a > 0:
for i in items:
if i not in dblistitems:
items.remove(i)
break
a -= 1
if len(items) == 0:
syncMemAndDBlist_1()

items = set([Item(1, "google.com"), Item(3, "yahoo.com"), Item(9,
"msn.com")])

dblistitems = set()
for d in dblist:
dblistitems.add(Item(**d))

a = len(items)
b = len(dblistitems)

if a < b:
syncMemAndDBlist_1()
else:
syncMemAndDBlist_2(a)

print 'final items are' ,items

 

[Peter Otten]

Ladislav Andel wrote:

what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.

You should replace the memlist with a dictionary using (host, id) tuples
as the keys. Here's an example that uses a set but requires you to modify
the <instance N> class:

dblist = [{'id':1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id':3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id':9, 'host':'msn.com','ip_address':'11.3.2.3'}]

class Item(object):
def __init__(self, id, host, **discarded):
self._tuple = (id, host)
def __hash__(self):
return hash(self._tuple)
def __eq__(self, other):
return self._tuple == other._tuple
def __repr__(self):
return "Item(id=%r, host=%r)" % self._tuple

items = set([Item(1, "google.com")])
for d in dblist:
item = Item(**d)
if item not in items:
print "adding", item
items.add(item)
else:
print item, "already there"

Thank you for this nice solution. I wouldn't be able to write it this
way at all

Then think twice before you use it. The dictionary approach should be
straightforward.

but what about removing from memlist if there is less items in dblist
than in items (instances)?
I will have to iterate over items(instances) and remove that one which
is not in dblist I guess.

Yes, but again, if you use a dictionary instead of a list the lookup will be
efficient.

To follow up on my previous post: with sets there is a concise spelling:

items &= set(Item(**d) for d in dblist)

or even

items.intersection_update(Item(**d) for d in dblist)

if you don't mind object identity.

Peter

 

[Ladislav Andel]

Ladislav Andel wrote:

Ladislav Andel wrote:



what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.


You should replace the memlist with a dictionary using (host, id) tuples
as the keys. Here's an example that uses a set but requires you to modify
the <instance N> class:

dblist = [{'id':1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id':3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id':9, 'host':'msn.com','ip_address':'11.3.2.3'}]

class Item(object):
def __init__(self, id, host, **discarded):
self._tuple = (id, host)
def __hash__(self):
return hash(self._tuple)
def __eq__(self, other):
return self._tuple == other._tuple
def __repr__(self):
return "Item(id=%r, host=%r)" % self._tuple

items = set([Item(1, "google.com")])
for d in dblist:
item = Item(**d)
if item not in items:
print "adding", item
items.add(item)
else:
print item, "already there"

Thank you for this nice solution. I wouldn't be able to write it this
way at all

Then think twice before you use it. The dictionary approach should be
straightforward.

but what about removing from memlist if there is less items in dblist
than in items (instances)?
I will have to iterate over items(instances) and remove that one which
is not in dblist I guess.

Yes, but again, if you use a dictionary instead of a list the lookup will be
efficient.

To follow up on my previous post: with sets there is a concise spelling:

items &= set(Item(**d) for d in dblist)

or even

items.intersection_update(Item(**d) for d in dblist)

if you don't mind object identity.

Peter

Thank you very much for this. I think everybody should ignore my
previous email.. Shame on me.

 

[Ladislav Andel]

Ladislav Andel wrote:

Ladislav Andel wrote:



what would be the most efficient way to do following?

I have a list of dictionaries taken from DB e.g.
dblist = [{'id:1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id:3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id:9, 'host':'msn.com','ip_address':'11.3.2.3'}]

and list of object instances in memory(it's just for example)
which are looping within itself and testing particular hosts

memlist = [<instance 1>,<instance 2>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 9 and memlist[1].host is msn.com etc.

Now I want to add a new instance to memlist since id=3(in dblist) is not
in memlist.
How would you iterate through it and insert a new instance?

The result should be:
memlist = [<instance 1>,<instance 2>, <instance 3>]
memlist[0].id is 1 and memlist[0].host is google.com etc.
memlist[1].id is 3 and memlist[1].host is yahoo.com etc.
memlist[2].id is 9 and memlist[2].host is msn.com etc.


You should replace the memlist with a dictionary using (host, id) tuples
as the keys. Here's an example that uses a set but requires you to modify
the <instance N> class:

dblist = [{'id':1, 'host':'google.com','ip_address':'1.2.3.4'},
{'id':3, 'host':'yahoo.com','ip_address':'5.6.7.8'},
{'id':9, 'host':'msn.com','ip_address':'11.3.2.3'}]

class Item(object):
def __init__(self, id, host, **discarded):
self._tuple = (id, host)
def __hash__(self):
return hash(self._tuple)
def __eq__(self, other):
return self._tuple == other._tuple
def __repr__(self):
return "Item(id=%r, host=%r)" % self._tuple

items = set([Item(1, "google.com")])
for d in dblist:
item = Item(**d)
if item not in items:
print "adding", item
items.add(item)
else:
print item, "already there"

Thank you for this nice solution. I wouldn't be able to write it this
way at all

Then think twice before you use it. The dictionary approach should be
straightforward.

but what about removing from memlist if there is less items in dblist
than in items (instances)?
I will have to iterate over items(instances) and remove that one which
is not in dblist I guess.

Yes, but again, if you use a dictionary instead of a list the lookup will be
efficient.

To follow up on my previous post: with sets there is a concise spelling:

items &= set(Item(**d) for d in dblist)

or even

items.intersection_update(Item(**d) for d in dblist)

if you don't mind object identity.

Peter

Thanks again for this solution, but I forgot to say that in the instance
is a looping call which
need to be stopped before deleting any instance from items.
So I need to call stopLoop method in the given item in items before it
gets removed.
If there is any addition to items it's quite easy to call
item.startLoop() method.


my class would look like (I use twisted but it should not make any
difference):

class Item(object):
def __init__(self, id, host, interval, **discarded):
self._tuple = (id, host)
self.interval = interval
def __hash__(self):
return hash(self._tuple)
def __eq__(self, other):
return self._tuple == other._tuple
def __repr__(self):
return "Item(id=%r, host=%r)" % self._tuple
def startLoop(self):
self.l = task.LoopingCall(self.serverTest)
self.l.start(self.interval, False)
def serverTest(self):
return getReply()
# this is just for example. It returns something on each start of the loop
# but because it sends packets within network, it takes some time before it gets a reply
# so if I remove this object instance before I get reply it gets broken somewhere else
# and I need to stop the loop before removing. Hopefully, this make sence.

def stopLoop(self):
self.l.stop()

 

[Peter Otten]

need to be stopped before deleting any instance from items.
So I need to call stopLoop method in the given item in items before it
gets removed.
If there is any addition to items it's quite easy to call
item.startLoop() method.

Unless you want to rely on the __del__ method or introduce another
complication (weak references) you have to be explicit:

# untested
# removing items
db_items = set(Item(**d) for d in dblist)
delenda = items - db_items
for item in delenda:
item.stopLoop()
items.remove(item)

(I use twisted but it should not make any difference):

I've not worked with twisted, so I can't confirm that.
I can imagine that they provide their own way to spell a finalizer...

Peter

 

[Ladislav Andel]

I have learnt a lot from your example and used it for my purpose.
Thank you, it very helped me.
Lada