Save Binary data.

[GMane Python]

Hello All.
I have a program that downloads 'gigabytes' of Axis NetCam photos per day.
Right now, I set up the process to put the images into a queue, and every 30
or so seconds, 'pop' them from the queue and save them to disc. I save
them as individual files.

I think that I'd like to modify it to save into one file 100-200 images,
so that I don't have directories with 50,000-90,000 frames before handing
that off to a DivX Encoder.

I don't know if I need to use something like cPickle, or maybe just save
them as a binary data file (which would be a temp file until later in the
day when I open it to begin the encoding process.)

Can someone please help me with some direction?


Thank you!
Dave

 

[Larry Bates]

Images are binary data, don't do anything to them just save them
to files on disk in their binary format. The extra processing of
pickling them isn't going to help. Directories with large
numbers of files was a problem in FAT16 and FAT32 filesystems but
not really a problem in NTFS or Linux (at least that I've found).
Appending 100-200 images together into a single file will surely
cut down on the number of files. A lot depends on what the next
step in the process is expecting (e.g. individual files or a
a stream of data). If it is individual files, you will have to
split them back apart anyway so keeping them as individual files
is a benefit.

Larry Bates

 

[Benjamin Niemann]

Hello All.
I have a program that downloads 'gigabytes' of Axis NetCam photos per
day.
Right now, I set up the process to put the images into a queue, and every
30
or so seconds, 'pop' them from the queue and save them to disc. I save
them as individual files.

I think that I'd like to modify it to save into one file 100-200 images,
so that I don't have directories with 50,000-90,000 frames before handing
that off to a DivX Encoder.

I don't know if I need to use something like cPickle, or maybe just save
them as a binary data file (which would be a temp file until later in the
day when I open it to begin the encoding process.)

Can someone please help me with some direction?

You could use the tarfile module to create a single file holding a arbitrary
number of files - including compression, if you want, but if your images
are JPEGs then further compression is mostly a waste of CPU cycles. You can
extract the images later on with either a python script and tarfile or
using the standard commandline tool 'tar'.

If the images are uncompressed anyway, you could have a look at the netpbm
suite and its fileformat (which is pretty simple, but uncompressed and
would bloat JPEGs to a multiple of the original filesize) which supports
'image sequences'. Perhaps a DivX encoder could even support this
fileformat directly as input.

 

[Mike Meyer]

Directories with large numbers of files was a problem in FAT16 and
FAT32 filesystems but not really a problem in NTFS or Linux (at
least that I've found).

Depends on how you define "large" and what Linux file system you're
using. Of course, if you open the directory in a GUI directory
browser, you're probably going to be unhappy no matter what the
underlying file system.

The standard Unix solution to this is to break the files out into
subdirectories. Create a subdirectory with the name being the first
few letters of the file name, and then store the file in that
subdirectory. Easy to do programmatically, it's still easy to find
files "by hand", and you can make it as fine-grained as you want.

<mike