'error reading datastream' -- loading file only when transfer is complete?

[liuliuliu]

hello --

i'm running python/pygame on maemo (nokia 770). my situation is that
i'm continually scouring this one directory for incoming files. if i
see if there's a new file (coming in via wireless scp), i proceed to
load it and process it.

however, i think i am running into the issue that my program starts to
load the file after it recognises there is new data, but before the
file has completely transferred, so at unpredictable times i get a
pygame.error: Error reading from datastream.

what is the easiest way to work out this issue? easy being the key
word. thank you very much!

christine

 

[Andrew Robert]

hello --

i'm running python/pygame on maemo (nokia 770). my situation is that
i'm continually scouring this one directory for incoming files. if i
see if there's a new file (coming in via wireless scp), i proceed to
load it and process it.

however, i think i am running into the issue that my program starts to
load the file after it recognises there is new data, but before the
file has completely transferred, so at unpredictable times i get a
pygame.error: Error reading from datastream.

what is the easiest way to work out this issue? easy being the key
word. thank you very much!

christine

You might want to test for file locking before attempting to use

 

[liuliuliu]

thanks - i'm looking, but i found this as well. actually, does this
work?

import os
os.access(file, os.R_OK) # is it readable?

is this valid:

{ i have my file identified }
isFileAccessible = os.access(file, os.R_OK)
while !isFileAccessible:
isFileAccessible = os.access(file, os.R_OK)

and then whenever it's true it can only then proceed to load the file?

thanks, christine

 

[Jim Segrave]

You might want to test for file locking before attempting to use

Or, really crude - when you see the file, record the time and the file
size, but don't attempt to process it yet. Wait for a short interval,
then check the size again. If it's changed, wait again. When it gives
the same size after a delay period, assume all the data is there.

This isn't a good method, but it's simple to implement and will reduce
the occurrence of attempts to process a file which is still in
transit/

 

[liuliuliu]

thanks all. this seemed to work:

import os
stat = os.stat(str(newFiles[index]))
size1 = stat.st_size
pygame.time.wait(100)
stat = os.stat(str(newFiles[index]))
size2 = stat.st_size
while size2 > size1:
size1 = size2
pygame.time.wait(500)
stat = os.stat(str(newFiles[index]))
size2 = stat.st_size
# procede to load complete file

my delay is so long because somehow pygame (or file-writing) runs
relatively slow on maemo/nokia770. even a delay of 400 milliseconds was
too quick to accommodate the slothly rate of an incoming file.