IOError and Try Again to loop the loop.

[The Danny Bos]

Heya,

I'm running a py script that simply grabs an image, creates a
thumbnail and uploads it to s3. I'm simply logging into ssh and
running the script through Terminal. It works fine, but gives me an
IOError every now and then.

I was wondering if I can catch this error and just get the script to
start again?
I mean, in Terminal it dies anyway, so I have to start it again by
hand, which is a pain as it dies so sporadically. Can I automate this
error, catch it and just get it to restart the loop?

Thanks for your time and energy,

Danny

 

[Chris Rebert]

Heya,

I'm running a py script that simply grabs an image, creates a
thumbnail and uploads it to s3. I'm simply logging into ssh and
running the script through Terminal. It works fine, but gives me an
IOError every now and then.

I was wondering if I can catch this error and just get the script to
start again?
I mean, in Terminal it dies anyway, so I have to start it again by
hand, which is a pain as it dies so sporadically. Can I automate this
error, catch it and just get it to restart the loop?

Of course. Use try-except;
http://docs.python.org/tutorial/errors.html#handling-exceptions
Here's one way to do it:

for image in images:#or whatever the actual loop is
# whatever
while True:
try:
part_that_may_raise_IOError()
except IOError:
print("IOError; Retrying...")
else:
break
# whatever other stuff

Cheers,
Chris

 

[MRAB]

Heya,

I'm running a py script that simply grabs an image, creates a
thumbnail and uploads it to s3. I'm simply logging into ssh and
running the script through Terminal. It works fine, but gives me an
IOError every now and then.

I was wondering if I can catch this error and just get the script to
start again?
I mean, in Terminal it dies anyway, so I have to start it again by
hand, which is a pain as it dies so sporadically. Can I automate this
error, catch it and just get it to restart the loop?

Thanks for your time and energy,

Exceptions can be caught. You could do something like this:

while True:
try:
do_something()
break
except IOError:
pass

 

[Alex Hall]

It seems like seeing the code, or at least the bit in question, would
be easier, but what about:

for img in range(0, len(imagesToUpload)
try:
#do the thumbnail / upload thing on images
exceptIOError:
i=0

This will duplicate a lot of the images already processed, but you
said you are just restarting the script anyway. If you are in a while
loop, just move the processing to the for loop and use the while to
add all images to be processed to a list.

 

[The Danny Bos]

Thanks gang,
I'm gonna paste what I've put together, doesn't seem right. Am I way
off?

Here's my code.
- It goes through a table Item
- Matches that Item ID to an API call
- Grabs the data, saves it and creates the thumbnail
- It dies due to Timeouts and Other baloney, all silly, nothing code
based.

items = Item.objects.all().filter(cover='').order_by('-reference_id')
for item in items:
url = "http://someaddress.org/books/?issue=%s" % item.reference_id

url_array = []
url_open = urllib.urlopen(url)
url_read = url_open.read().decode('utf-8')

try:
url_data = simplejson.loads(url_read)
url_array.append(url_data)

for detail in url_array:
if detail['artworkUrl']:
cover_url = detail['artworkUrl'].replace(' ','%20')
cover_open = urllib.urlretrieve(cover_url)
cover_name = os.path.split(cover_url)[1]

item.cover.save(cover_name, File(open(cover_open[0])),
save=True) ## Create and save Thumbnail

print "Cover - %s: %s" % (item.number, url)
else:
print "Missing - %s: %s" % (item.number, url)

except ValueError:
print "Error Processing record: %s: %s" % (item.reference_id, url)
pass
except IOError:
print "IOError; Retrying..."
pass

print "Done"

 

[Chris Rebert]

Thanks gang,
I'm gonna paste what I've put together, doesn't seem right. Am I way
off?

Here's my code.
 - It goes through a table Item
 - Matches that Item ID to an API call
 - Grabs the data, saves it and creates the thumbnail
 - It dies due to Timeouts and Other baloney, all silly, nothing code
based.

items = Item.objects.all().filter(cover='').order_by('-reference_id')
for item in items:
       url = "http://someaddress.org/books/?issue=%s" % item.reference_id

       url_array = []
       url_open = urllib.urlopen(url)
       url_read = url_open.read().decode('utf-8')

       try:
               url_data = simplejson.loads(url_read)
               url_array.append(url_data)

               for detail in url_array:

Unless I'm missing something, there's no need for url_array to exist
at all. It starts out empty, you append url_data to it, then you
iterate over it as `detail`; and you don't touch it anywhere else in
the loop. Just s/detail/url_data/ and excise url_array altogether. As
a bonus, there'll be one less level of indentation.

Also, the reason your code doesn't work (currently, it just skips to
the next item upon error) is because you're missing a surrounding
`while True` loop (and associated embedded `break`) to do the retrying
(see my or MRAB's examples).

Additionally, stylistically I'd prefer the try-excepts to cover
smaller and more targeted areas of the code, rather than having one
giant blanket one for the entire loop body; perhaps that's just me
though.

Cheers,
Chris

 

[The Danny Bos]

Thanks Chris,

Agreed some of the code is a lot useless, I need to go through that
stuff.
So something like this (apologies for asking for some details, I'm not
good at catching):

items = Item.objects.all().filter(cover='').order_by('-reference_id')
for item in items:
url = "http://someaddress.org/books/?issue=%s" %
item.reference_id
url_array = []
url_open = urllib.urlopen(url)
url_read = url_open.read().decode('utf-8')

while True:
try:
url_data = simplejson.loads(url_read)
url_array.append(url_data)
for detail in url_array:
if detail['artworkUrl']:
cover_url =
detail['artworkUrl'].replace(' ','%20')
cover_open =
urllib.urlretrieve(cover_url)
cover_name = os.path.split(cover_url)
[1]
item.cover.save(cover_name,
File(open(cover_open[0])), save=True)
print "Cover - %s: %s" % (item.number,
url)
else:
print "Missing - %s: %s" %
(item.number, url)
break
except ValueError:
print "Error Processing record: %s: %s" %
(item.reference_id, url)
pass
except IOError:
print "IOError; Retrying..."
pass

print "Done"

Thanks gang,
I'm gonna paste what I've put together, doesn't seem right. Am I way
off?

Here's my code.
 - It goes through a table Item
 - Matches that Item ID to an API call
 - Grabs the data, saves it and creates the thumbnail
 - It dies due to Timeouts and Other baloney, all silly, nothing code
based.

items = Item.objects.all().filter(cover='').order_by('-reference_id')
for item in items:
       url = "http://someaddress.org/books/?issue=%s" % item.reference_id

       url_array = []
       url_open = urllib.urlopen(url)
       url_read = url_open.read().decode('utf-8')

       try:
               url_data = simplejson.loads(url_read)
               url_array.append(url_data)

               for detail in url_array:

Unless I'm missing something, there's no need for url_array to exist
at all. It starts out empty, you append url_data to it, then you
iterate over it as `detail`; and you don't touch it anywhere else in
the loop. Just s/detail/url_data/ and excise url_array altogether. As
a bonus, there'll be one less level of indentation.

Also, the reason your code doesn't work (currently, it just skips to
the next item upon error) is because you're missing a surrounding
`while True` loop (and associated embedded `break`) to do the retrying
(see my or MRAB's examples).

Additionally, stylistically I'd prefer the try-excepts to cover
smaller and more targeted areas of the code, rather than having one
giant blanket one for the entire loop body; perhaps that's just me
though.

Cheers,
Chris

 

[Chris Rebert]

Thanks gang,
I'm gonna paste what I've put together, doesn't seem right. Am I way
off?

Here's my code.
 - It goes through a table Item
 - Matches that Item ID to an API call
 - Grabs the data, saves it and creates the thumbnail
 - It dies due to Timeouts and Other baloney, all silly, nothing code
based.

items = Item.objects.all().filter(cover='').order_by('-reference_id')
for item in items:
       url = "http://someaddress.org/books/?issue=%s" % item.reference_id

       url_array = []
       url_open = urllib.urlopen(url)
       url_read = url_open.read().decode('utf-8')

       try:
               url_data = simplejson.loads(url_read)
               url_array.append(url_data)

               for detail in url_array:

Unless I'm missing something, there's no need for url_array to exist
at all. It starts out empty, you append url_data to it, then you
iterate over it as `detail`; and you don't touch it anywhere else in
the loop. Just s/detail/url_data/ and excise url_array altogether. As
a bonus, there'll be one less level of indentation.

Also, the reason your code doesn't work (currently, it just skips to
the next item upon error) is because you're missing a surrounding
`while True` loop (and associated embedded `break`) to do the retrying
(see my or MRAB's examples).

Additionally, stylistically I'd prefer the try-excepts to cover
smaller and more targeted areas of the code, rather than having one
giant blanket one for the entire loop body; perhaps that's just me
though.

Thanks Chris,

Agreed some of the code is a lot useless, I need to go through that
stuff.
So something like this (apologies for asking for some details, I'm not
good at catching):

items = Item.objects.all().filter(cover='').order_by('-reference_id')
for item in items:
   url = "http://someaddress.org/books/?issue=%s" %
item.reference_id
   url_array = []
   url_open = urllib.urlopen(url)
   url_read = url_open.read().decode('utf-8')

   while True:
       try:

               break
       except ValueError:
               print "Error Processing record: %s: %s" %
(item.reference_id, url)
               pass
       except IOError:
               print "IOError; Retrying..."
               pass

print "Done"

Yes, correct, that's essentially it, although the `pass` statements
are superfluous, and I would personally put the `break` in a
separate+new else-clause of the try-except for clarity; so the
try-except part of the code would look like:

try:
# lots of code here
except ValueError:
print "Error Processing record: %s: %s" % (item.reference_id, url)
except IOError:
print "IOError; Retrying..."
else:
break

Also, please avoid top-posting in the future;
http://en.wikipedia.org/wiki/Top-posting#Top-posting

Cheers,
Chris