Piping binary data to an external program

Discussion in 'Ruby' started by |MKSM|, Mar 6, 2006.

  1. |MKSM|

    |MKSM| Guest

    Hello,

    I'm working on a logparser and i've run into some issues. It will
    parse OpenBSD PF logs. They are tcpdump format logs and BSD normally
    compress them.

    Here is the usage I have in mind:

    "gzip -cd log.gz | ruby logparser.rb --today"

    I have the following code:

    Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
    =09in_io.write($stdin.read)
    =09in_io.close
    =09$log =3D out_io.read
    }

    The script freezes on the open3 line and doesn't continue. I've tested
    several other methods but it doesn't seem to work.

    Any suggestions on how this can be done?

    Regards,

    Ricardo.
     
    |MKSM|, Mar 6, 2006
    #1
    1. Advertising

  2. |MKSM| wrote:
    > Hello,
    >
    > I'm working on a logparser and i've run into some issues. It will
    > parse OpenBSD PF logs. They are tcpdump format logs and BSD normally
    > compress them.
    >
    > Here is the usage I have in mind:
    >
    > "gzip -cd log.gz | ruby logparser.rb --today"
    >
    > I have the following code:
    >
    > Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
    > in_io.write($stdin.read)
    > in_io.close
    > $log = out_io.read
    > }
    >
    > The script freezes on the open3 line and doesn't continue. I've tested
    > several other methods but it doesn't seem to work.
    >
    > Any suggestions on how this can be done?


    It's likely that you run into a deadlock caused by pipe buffer sizes. I
    suggest to not read and write the whole content but to do it in chunks.
    Also, I'd separate the reading and writing code into two threads.

    Untested:

    Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
    t = Thread.new(in_io) do |out|
    while ( buff = $stdin.read( 1024 ) )
    out.write(buff)
    end
    out.close
    end
    $log = ""
    while ( b = out_io.read(1024))
    $log << b
    end
    }


    HTH

    Kind regards

    robert
     
    Robert Klemme, Mar 6, 2006
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    291
    Simon Forman
    Jul 18, 2006
  2. Replies:
    12
    Views:
    621
    Jeff Higgins
    Jul 4, 2007
  3. Thomas Kellerer
    Replies:
    47
    Views:
    1,550
    Martin Gregorie
    Apr 12, 2009
  4. Tim Arnold

    piping input to an external script

    Tim Arnold, May 11, 2009, in forum: Python
    Replies:
    9
    Views:
    331
    Steve Howell
    May 12, 2009
  5. Bazsl

    Piping binary files

    Bazsl, Oct 15, 2007, in forum: Ruby
    Replies:
    1
    Views:
    129
    Bazsl
    Oct 15, 2007
Loading...

Share This Page