reading multiple files

[Mag Gam]

I have 3 files which are constantly being updated therefore I use tail
-f /var/log/file1, tail -f /var/log/file2, and tail -f /var/log/file3

For 1 file I am able to manage by
tail -f /var/log/file1 | python prog.py

prog.py looks like this:
f=sys.stdin
for line in f:
print line

But how can I get data from /var/log/file2 and /var/log/file3 ? I
prefer a native python way instead of doing tail -f

TIA

 

[Alain Ketterlin]

I have 3 files which are constantly being updated therefore I use tail
-f /var/log/file1, tail -f /var/log/file2, and tail -f /var/log/file3

For 1 file I am able to manage by
tail -f /var/log/file1 | python prog.py

prog.py looks like this:
f=sys.stdin
for line in f:
print line

But how can I get data from /var/log/file2 and /var/log/file3 ?

Use shell tricks, e.g., with bash:

yourpythonprog <(tail -f .../file1) <(tail -f .../file2) <(...)

and let your prog open its three parameters like regular files (they are
fifos actually). If your shell doesn't support <(...), create the fifos
and redirect tail output before launching your prog.

If you want "purer" python, launch the three "tail -f" with subprocess,
and use the select module to get input (you didn't explain the logic you
will follow to track three files---you may not need select if you expect
one line from each file before waiting for the next line of any).

I prefer a native python way instead of doing tail -f

Emulating tail will require a lot of stat/seeks, and finding lines will
require an additional level of complexity.

Also, tail -f has a cost [*]. The only way to avoid it is to use
inotify, which seems to have a python interface, available at
http://pyinotify.sourceforge.net/ (I've never used it). Again, emulating
tail -f with inotify is significant work.

-- Alain.

[*] Paul Rubin is one of the authors, I think he reads this group.