How to parse network traffic from tcpdump in ruby?

M

Martin Kahlert

Hi!

i have a bunch of network traffic to analyse. The traffic has been
captured with something like "tcpdump -w traffic -s 0".

Is there any fast method to parse the traffic file and get the packets'
headers as well as their payload with relative small effort?

Of course i would like to do that in ruby, but if there are only perlish
ways, i will use that, too. The timeframe of the project doesn't allow
me a lot of try and error.

Any hints are very appreciated!

Thanks in advance for any help
Martin.
 
G

Gavin Sinclair

Hi!
i have a bunch of network traffic to analyse. The traffic has been
captured with something like "tcpdump -w traffic -s 0".

Is there any fast method to parse the traffic file and get the packets'
headers as well as their payload with relative small effort?

Of course i would like to do that in ruby, but if there are only perlish
ways, i will use that, too. The timeframe of the project doesn't allow
me a lot of try and error.

Any hints are very appreciated!

If you give a sample of the input and a description of the desired output,
I'm sure someone will have an answer very quickly.

Cheers,
Gavin
 
M

Martin Kahlert

Hi Gavin,

If you give a sample of the input and a description of the desired output,
I'm sure someone will have an answer very quickly.

For the input side:
Run a
tcpdump -i eth0 -w traffic -s 0
on your box while doing some internet surfing.

For the output side i would like to see something like this:
Package sent at [timestamp] from [IP-address:port] to [IP-address:port]:
HTTP/1.1 .....

Hope that helps
Martin.
 
W

William Morgan

Excerpts (reformatted) from Martin Kahlert's mail of 27 Aug 2004 (EDT):
For the input side:
Run a
tcpdump -i eth0 -w traffic -s 0
on your box while doing some internet surfing.

For the output side i would like to see something like this:
Package sent at [timestamp] from [IP-address:port] to [IP-address:port]:
HTTP/1.1 .....

Frankly, I would run the output through tcpdump again, like this:

tcpdump -r traffic -vv -A

and then parse the output with some regexps in Ruby. That's the easiest
thing to do.

Alternatively, there is a libpcap-ruby package somewhere that might work
for you. It hasn't been updated in a while and doesn't seem to work for
wireless packets, but it probably works fine for run-of-the-mill
ethernet packets. (The tcpdump solution works fine for dumping wireless
packets if you use -s 0.)

HTH,
 
W

Will Drewry

Hi!

i have a bunch of network traffic to analyse. The traffic has been
captured with something like "tcpdump -w traffic -s 0".

Is there any fast method to parse the traffic file and get the packets'
headers as well as their payload with relative small effort?

Of course i would like to do that in ruby, but if there are only perlish
ways, i will use that, too. The timeframe of the project doesn't allow
me a lot of try and error.

Any hints are very appreciated!

Thanks in advance for any help
Martin.
here is the pcap package:
http://www.goto.info.waseda.ac.jp/~fukusima/ruby/pcap-e.html

Here's an example that will print the source addr of every packet in the file:

require 'pcap'
cap = Pcap::Capture.open_offline('/tmp/my.dmp')
cap.each { |pkt| p pkt.src}

Here's another that will count the total number of SYNs - great for
looking back at synfloods :)

require 'pcap'
cap = Pcap::Capture.open_offline('/tmp/my.dmp')
syn = 0
cap.each { |pkt| syn += 1 if pkt.tcp? and pkt.tcp_syn? }
puts syn

And finally one that might be more what you are interested in! This
one will print out the tcp_data. This is where you start to see the
super convenience of using ruby over ethereal for automagic parsing of
a lot of dump data :)

require 'pcap'
cap = Pcap::Capture.open_offline('/tmp/my.dmp')
cap.each { |pkt| p pkt.tcp_data if pkt.tcp? }

I hope that Masaki Fukushima keeps the package alive because it's awesome!

Enjoy! I'd love to see any cooler examples you come up with!
wad
 
M

Martin Kahlert

Hi William,

Frankly, I would run the output through tcpdump again, like this:

tcpdump -r traffic -vv -A

and then parse the output with some regexps in Ruby. That's the easiest
thing to do.

Thanks for your response. What is the -A option for?
My tcpdump version (3.7.2) does not know it.

Regards
Martin.
 
M

Martin Kahlert

Hi Will,

thank you for your hint!
I will give pcap a try. Looks very nice to me.

Regards
Martin.
 
D

David Ross

--- Will Drewry said:
here is the pcap package:

http://www.goto.info.waseda.ac.jp/~fukusima/ruby/pcap-e.html

Here's an example that will print the source addr of
every packet in the file:

require 'pcap'
cap = Pcap::Capture.open_offline('/tmp/my.dmp')
cap.each { |pkt| p pkt.src}

Here's another that will count the total number of
SYNs - great for
looking back at synfloods :)

require 'pcap'
cap = Pcap::Capture.open_offline('/tmp/my.dmp')
syn = 0
cap.each { |pkt| syn += 1 if pkt.tcp? and
pkt.tcp_syn? }
puts syn

And finally one that might be more what you are
interested in! This
one will print out the tcp_data. This is where you
start to see the
super convenience of using ruby over ethereal for
automagic parsing of
a lot of dump data :)

require 'pcap'
cap = Pcap::Capture.open_offline('/tmp/my.dmp')
cap.each { |pkt| p pkt.tcp_data if pkt.tcp? }

I hope that Masaki Fukushima keeps the package alive
because it's awesome!

Enjoy! I'd love to see any cooler examples you come
up with!
wad

Thanks for the examples. Well, if it isn't updated and
it drifts away, someone will maintain it. I use Ruby
very much, and it would be nice to have that library
for use with everyday activities. --David Ross



__________________________________
Do you Yahoo!?
Take Yahoo! Mail with you! Get it on your mobile phone.
http://mobile.yahoo.com/maildemo
 
W

William Morgan

Excerpts (reformatted) from Martin Kahlert's mail of 31 Aug 2004 (EDT):
Thanks for your response. What is the -A option for?
My tcpdump version (3.7.2) does not know it.

-A just dumps the packet contents in ASCII form to stdout. This makes it
easy to read things like HTTP requests. You could use -x for hex output
and parse that instead.

I've been playing around with running tcpdump through IO.popen myself
recently and the only problem is that the output is buffered, so I have
to wait for about 20 packets to hit before I get anything from a
readline() call. If that's a problem, use the pcap library instead. (I
just can't use it because it barfs on wireless packets.)
 
M

Martin Kahlert

Hi!

It's me again.
The pcap ruby library is great!
Works like a charm for me.

The only problem is this:
I sniffed the network traffic while surfing the web (only one
computer involved and only one Internet Exlorer open).

How can i find out the client's request for any server response?
I tried using the ack and seq numbers for that, but they seem
to be quite independent.

For example:
i click a link and before the server had time to send it to
me (completely) i click another one.
How do i know, which packats belong to witch request?
If the server sends the second part of the first requested page
it could have received the second request and has to ack that one
in the first request-response.

Is there an easy way to find out the correct package history?

I want to use the requests (especially the filenames/hostnames)
in order to set up a dummy webspace with the corresponding html documents
i can browse through.

Thanks for any help
Martin.
 
W

William Morgan

Excerpts from Martin Kahlert's mail of 7 Sep 2004 (EDT):
How can i find out the client's request for any server response?
I tried using the ack and seq numbers for that, but they seem
to be quite independent.

Back in the olden days you could use the client TCP port to
differentiate between HTTP requests, as one TCP connection would be
opened for each HTTP request+response. With the advent of Keep-Alive
connections it's more difficult, as multiple transactions will be on the
same TCP connection. But that would be the way to do it: look at the
client TCP port for each packet.

But if this is the level of analysis you want, I think tcpdump is the
wrong (i.e. too primitive of a) tool. You want something at the
application layer, not the TCP layer, like sitting an HTTP proxy in the
middle and capturing traffic content. Unless you're snooping a network
for other people's traffic---then I guess this might be your only
option. :)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top