exclude binary files from os.walk

[rbt]

Is there an easy way to exclude binary files (I'm working on Windows XP)
from the file list returned by os.walk()?

Also, when reading files and you're unsure as to whether or not they are
ascii or binary, I've always thought it safer to 'rb' on the read, is
this correct... and if so, what's the reasoning behind this? Again all
of this pertains to files on Windows XP and Python 2.4

Many thanks!

 

[Grant Edwards]

Is there an easy way to exclude binary files (I'm working on
Windows XP) from the file list returned by os.walk()?

Sure, assuming you can provide a rigorous definition of 'binary
files'.

Also, when reading files and you're unsure as to whether or
not they are ascii or binary, I've always thought it safer to
'rb' on the read, is this correct...

That depends on what you want. Adding a 'b' will disable the
cr/lf handling. Not sure what else it does.

and if so, what's the reasoning behind this?

Behind what?

 

[rbt]

Sure, assuming you can provide a rigorous definition of 'binary
files'.

non-ascii

 

[Larry Bates]

There's no definitive way of telling a file is
"non-ascii". Bytes in a binary file define
perfectly good ascii characters. Windows
depends on file extensions to try to keep track
of the "type" of data in a file, but that isn't
foolproof. I can rename a plain ascii file with
a .EXE extension.

We could be of more help, if you would take the
time to explain a little about what you are trying
to do.

Larry Bates

 

[sp1d3rx]

you might want to look up the 'isascii' function...
i.e. - can be represented using just 7-bits.

 

[Dan Perl]

Is there an easy way to exclude binary files (I'm working on Windows XP)
from the file list returned by os.walk()?

Also, when reading files and you're unsure as to whether or not they are
ascii or binary, I've always thought it safer to 'rb' on the read, is this
correct... and if so, what's the reasoning behind this? Again all of this
pertains to files on Windows XP and Python 2.4

Please clarify: is your question about identifying binary (non-ascii) files
or about using os.walk?

 

[Grant Edwards]

There's no definitive way of telling a file is "non-ascii".
Bytes in a binary file define perfectly good ascii characters.

As long as bit 7 is a 0.

Traditional ASCII only allows/defines the values 0x00 through
0x7f. If that's what is meant by "ASCII", then a file
containting bytes greater than 0x7F is not ASCII.

If all bytes are 0x7F or below, the file _may_ be ASCII, but
there's now way to tell if it _is_ ASCII unless you ask the
creator of the file. It could be Baudot or some other encoding
that doesn't use bit 7. Or, it could just be binary data that
happens to have bit 7 == 0.

We could be of more help, if you would take the time to
explain a little about what you are trying to do.

Yup.

 

[Craig Ringer]

non-ascii

That's not really safe when dealing with utf-8 files though, and IIRC
with UCS2 or UCS4 as well. The Unicode BOM its self might (I'm not sure)
qualify as ASCII.

 

[Mark McEahern]

> Is there an easy way to exclude binary files (I'm working on Windows

XP) from the file list returned by os.walk()?

Sure, piece of cake:

#!/usr/bin/env python

import os

def textfiles(path):
include = ('.txt', '.csv',)
for root, dirs, files in os.walk(path):
for name in files:
prefix, ext = os.path.splitext(name)
if ext.lower() not in include:
continue
filename = os.path.join(root, name)
yield filename

path = os.getcwd()
for name in textfiles(path):
print name

;-)

// m

 

[Alex Martelli]

non-ascii

The only way to tell for sure if a file contains only ASCII characters
is to read the whole file and check. You _are_, however, using a very
strange definition of "binary". A file of text in German, French or
Italian, for example, is likely to be one you'll define as "binary" --
just as soon as it contains a vowel with accent or diaeresis, for
example. On the other hand, you want to consider "non-binary" a file
chock full of hardly-ever-used control characters, just because the
American Standard Code for Information Interchange happened to
standardize them once upon a time? Most people's intuitive sense of
what "binary" means would rebel against both of these choices, I think;
calling a file "binary" because its contents are, say, the string
'El perro de aguas español.\n' (the n-with-tilde in "español"
disqualifies it from being ASCII), while another whose contents are 32
bytes all made up of 8 zero bits each (ASCII 'NUL' characters) is to be
considered "non-binary".

In any case, since you need to open and read all the files to check them
for "being binary", either by your definition or whatever heuristics you
might prefer, you would really not ``excluded them from os.walk'', but
rather filter os.walk's results by these criteria.


Alex

 

[Alex Martelli]

That's not really safe when dealing with utf-8 files though, and IIRC
with UCS2 or UCS4 as well. The Unicode BOM its self might (I'm not sure)
qualify as ASCII.

Nope, both bytes in the BOM have the high-order bit set -- they're 0xFF
and 0xFE -- so they definitely don't qualify as ASCII.


Alex

 

[Bengt Richter]

Please clarify: is your question about identifying binary (non-ascii) files
or about using os.walk?

I have a feeling it's about walking directories and identifying which files
to should be "cooked" (to normalize line endings when opened and read).

Regards,
Bengt Richter