how to find not the next sibling but the 2nd sibling or find sibling "a" OR sinbling "b"

[localpricemaps]

i have some html which looks like this where i want to scrape out the
href stuff (the www.cnn.com part)

<div class="noFood">Cheese</div>
<div class="food">Blue</div>
<a class="btn" href = "http://www.cnn.com">


so i wrote this code which scrapes it perfectly:

for incident in row('div', {'class':'noFood'}):
b = incident.findNextSibling('div', {'class': 'food'})
print b
n = b.findNextSibling('a', {'class': 'btn'})
print n
link = n['href'] + "','"

problem is that sometimes the 2nd tag , the <div class="food"> tag , is
sometimes called food, sometimes called drink. so sometimes it looks
like this:

<div class="noFood">Cheese</div>
<div class="drink">Pepsi</div>
<a class="btn" href = "http://www.cnn.com">

how do i alter my script to take into account the fact that i will
sometimes have food and sometimes have drink as the class name? is
there a way to say "look for food or drink" or a way to say "look for
this incident and then find not the next sibling but the 2nd next
sibling" if that makes any sense?

thanks

 

[Kent Johnson]

i have some html which looks like this where i want to scrape out the
href stuff (the www.cnn.com part)

<div class="noFood">Cheese</div>
<div class="food">Blue</div>
<a class="btn" href = "http://www.cnn.com">


so i wrote this code which scrapes it perfectly:

for incident in row('div', {'class':'noFood'}):
b = incident.findNextSibling('div', {'class': 'food'})
print b
n = b.findNextSibling('a', {'class': 'btn'})
print n
link = n['href'] + "','"

problem is that sometimes the 2nd tag , the <div class="food"> tag , is
sometimes called food, sometimes called drink.

Apparently you are using Beautiful Soup. The value in the attribute
dictionary can be a callable; try this:

def isFoodOrDrink(attr):
return attr in ['food', 'drink']

b = incident.findNextSibling('div', {'class': isFoodOrDrink})

Alternately you could omit the class spec and check for it in code.

Kent

 

[localpricemaps]

i actually realized there are 3 potentials for class names. either
food or drink or dessert. so my question is whether or not i can alter
your function to look like this?

def isFoodOrDrinkOrDesert(attr):
return attr in ['food', 'drink', 'desert']


thanks in advance for the help

i have some html which looks like this where i want to scrape out the
href stuff (the www.cnn.com part)

<div class="noFood">Cheese</div>
<div class="food">Blue</div>
<a class="btn" href = "http://www.cnn.com">


so i wrote this code which scrapes it perfectly:

for incident in row('div', {'class':'noFood'}):
b = incident.findNextSibling('div', {'class': 'food'})
print b
n = b.findNextSibling('a', {'class': 'btn'})
print n
link = n['href'] + "','"

problem is that sometimes the 2nd tag , the <div class="food"> tag , is
sometimes called food, sometimes called drink.

Apparently you are using Beautiful Soup. The value in the attribute
dictionary can be a callable; try this:

def isFoodOrDrink(attr):
return attr in ['food', 'drink']

b = incident.findNextSibling('div', {'class': isFoodOrDrink})

Alternately you could omit the class spec and check for it in code.

Kent

 

[Fredrik Lundh]

i actually realized there are 3 potentials for class names. either
food or drink or dessert. so my question is whether or not i can alter
your function to look like this?

def isFoodOrDrinkOrDesert(attr):
return attr in ['food', 'drink', 'desert']

what happens when you try that ?

</F>

 

[Kent Johnson]

i actually realized there are 3 potentials for class names. either
food or drink or dessert. so my question is whether or not i can alter
your function to look like this?

def isFoodOrDrinkOrDesert(attr):
return attr in ['food', 'drink', 'desert']

Check the spelling of 'dessert' and give it a try.

Kent

 

[homepricemaps]

ok i found something that works. instead of using the def i did this:

for incident in row('div', {'class': 'food' or 'drink' }):

and it worked!

only thing is that i think i am messing up the logic and here is why

So when i run my script i get results, meaning it scrapes some stuff
out,
but then i get errors and where i am told:

TypeError: unsupported operand type(s) for +: 'NullType' and 'str'


Is this because of the logic in my code? i mean what i want the script
to
do is look for the <Tr> tag and then find the first div tag named food
or
drink, find its sibling named food, drink or dessert and then find the
button tag which is the following sibling and THEN scrape out the href.
is
it possible that after it finds that first div and looks for the next
sibling and then the next siblings href that it then tries to run the
same
process all over again starting with the 2nd div tag and being that it
can't
find the another div tag after the 2nd div tag that it trips up?

know what i mean?


here is my code:

for row in bs('tr'):
for incident in row('div', {'class': 'food' or 'drink'}):
b = incident.findNextSibling('div', {'class': 'food' or 'drink'
or
'dessert'})
n = b.findNextSibling('a', {'class': 'btn'})
link= n['href'] + "','"

 

[Fredrik Lundh]

ok i found something that works. instead of using the def i did this:

for incident in row('div', {'class': 'food' or 'drink' }):

and it worked!

'food' or 'drink' doesn't do what you think it does:
'food'
{'class': 'food'}

</F>

 

[localpricemaps]

hey fredrik,

i don't understand what you are saying

 

[Brett Hoerner]

hey fredrik,

i don't understand what you are saying

Do what he showed in the Python interactive shell,

"or" returns the first true element, anything but False or None, I
think... so 'food' (a string) is true, and always will return in that
code.

http://diveintopython.org/power_of_introspection/and_or.html

Brett

 

[Steve Holden]

(e-mail address removed) wrote: [...]
"or" returns the first true element, anything but False or None, I
think... so 'food' (a string) is true, and always will return in that
code.

Just in case newbies are reading: in Python several different values are
considered false in the context of an "if" statement. These include

False # Boolean False
0 # The integer zero
0.0 # Floating-point zero
(0+0j) # Complex zero
None # The None object
[] # The empty list
() # The empty tuple
{} # The empty dictionary

This is mainly to allow the convenience of writing

if thing:
...

However, one has to be careful that code that needs to treat None
differently from [] uses explicit testing such as

if thing is None:
...

You can also construct your own classes so their instances evaluate to
True or False according to your needs.

regards
Steve

 

[localpricemaps]

well actually all i want it to do is find the first thing that shows up
whether its class:food or class: drink so that works for me. only
thing is that after it finds class:food i think it runs through the
html again and finds the following class:drink and being that there is
not class tag after that class: drink tag it fails.

 

[Fredrik Lundh]

well actually all i want it to do is find the first thing that shows up
whether its class:food or class: drink so that works for me.

what makes you think that looking "food" only will find either "food"
or "drink" ?

</F>