Progress when parsing a large file with SAX

[marc.omorain]

Hi there,

I have a 28mb XML file which I parse with SAX. I have some processing
to do in the startElement / endElement callbacks, which slows the
parsing down to about 60 seconds on my machine.

My application is unresponsive for this time, so I would like to show
a progress bar. I could show a spinner to show that the application is
responsive, but I would prefer to show a percentage. Is there any way
to query the parser to see how many bytes of the input file have been
processed so far?

Thanks,

Marc

 

[Diez B. Roggisch]

Hi there,

I have a 28mb XML file which I parse with SAX. I have some processing
to do in the startElement / endElement callbacks, which slows the
parsing down to about 60 seconds on my machine.

My application is unresponsive for this time, so I would like to show
a progress bar. I could show a spinner to show that the application is
responsive, but I would prefer to show a percentage. Is there any way
to query the parser to see how many bytes of the input file have been
processed so far?

I'd create a file-like object that does this for you. It should wrap the
original file, and count the number of bytes delivered. Something along
these lines (untested!!!):

class PercentageFile(object):

def __init__(self, filename):
self.size = os.stat(filename)[6]
self.delivered = 0
self.f = file(filename)

def read(self, size=None):
if size is None:
self.delivered = self.size
return self.f.read()
data = self.f.read(size)
self.delivered += len(data)
return data

@property
def percentage(self):
return float(self.delivered) / self.size * 100.0

Diez

 

[Anastasios Hatzis]

Diez B. Roggisch wrote:

....

I got the same problem with large XML as Marc.

So you deserve also my thanks for the example.

class PercentageFile(object):

def __init__(self, filename):
self.size = os.stat(filename)[6]
self.delivered = 0
self.f = file(filename)

def read(self, size=None):
if size is None:
self.delivered = self.size
return self.f.read()
data = self.f.read(size)
self.delivered += len(data)
return data

I guess some client impl need to call read() on a wrapped xml file until
all portions of the file are read.

@property
def percentage(self):
return float(self.delivered) / self.size * 100.0

@property?

What is that supposed to do?

Anastasios

 

[Diez B. Roggisch]

Diez B. Roggisch wrote:

...

I got the same problem with large XML as Marc.

So you deserve also my thanks for the example.

class PercentageFile(object):

def __init__(self, filename):
self.size = os.stat(filename)[6]
self.delivered = 0
self.f = file(filename)

def read(self, size=None):
if size is None:
self.delivered = self.size
return self.f.read()
data = self.f.read(size)
self.delivered += len(data)
return data

I guess some client impl need to call read() on a wrapped xml file until
all portions of the file are read.

You should fed the PercentageFile-object to the xml-parser, like this:

parser = xml.sax.make_parser()
pf = PercentageFile(filename)
parser.parse(pf)

@property?

What is that supposed to do?

It's making percentage a property, so that you can access it like this:

pf.percentage

instead of

pf.percentage()

Google python property for details, or pydoc property.

Diez

 

[marc.omorain]

Thanks guys!

I'll try it out later today and let you know how I get on.