avro slow?

[Miki Tebeka]

Greetings,

I'm reading some data from avro file using the avro library. It takes about a minute to load 33K objects from the file. This seem very slow to me, specially with the Java version reading the same file in about 1sec.

Here is the code, am I doing something wrong?

import avro.datafile
import avro.io
from time import time

def load(filename):
fo = open(filename, "rb")
reader = avro.datafile.DataFileReader(fo, avro.io.DatumReader())
for i, record in enumerate(reader):
pass

return i + 1

def main(argv=None):
import sys
from argparse import ArgumentParser

argv = argv or sys.argv

parser = ArgumentParser(description="Read avro file")


start = time()
num_records = load("events.avro")
end = time()

print("{0} records in {1} seconds".format(num_records, end - start))

if __name__ == "__main__":
main()

 

[Stefan Behnel]

Dan Stromberg, 06.05.2011 00:36:

Python is often more about programmer efficiency than
machine efficiency. With cost per MIPS going down and the price of
programmer time going up, it seems a good idea.

Especially when you also count the MIPS improvement during the time it
takes to write the code.

Stefan