packet sniffing with WinPcap/ SharpPcap: How to download FLV content noninteractively?

[Ken Fine]

I have been investigating programmatically downloading FLV content from
various sites ("video scraping"??)

Many interactive GUI tools do this, such as the Orbit downloader. At the
heart of them are packet sniffers which inspect the ethernet traffic coming
over the wire. I would like to do this noninteractively in code.

There is a .NET specific flavor of WinPCap called SharpPCap:
http://www.tamirgal.com/home/dev.aspx?Item=SharpPcap
Article on the Code Project:
http://www.codeproject.com/KB/IP/sharppcap.aspx

The first question I have is if server-side visits by the WebClient class
will result in "inspectable" results for FLV content. In other words if you
tell WebClient to go visit a site with FLVs on the page, can the server
sniff its own activity using something like SharpPCap?

The second question I have is more general, and would apply to
noninteractive and interactive apps I might build. The question is how
exactly to use something like SharPCap to find a simple list of FLV links in
the network traffic. I have never used a packet sniffer or sniffing
framework and have no idea how you'd (totally guessing here) pull strings of
interest out of the raw traffic.

Thanks much for any help and insight you can offer.

-KF

 

[Steven Cheng [MSFT]]

Hi KF,

For the FLV sniffing and downloading code, do you means you want to
programmticaly crawl some websites and download the FLV links found on
those websites?

For the two questions you mentioned:

1. If the server-side you mentioned here is the website/web application.
Sure, it can get noticed of all those client requesters(both interactive
users or some crawler). Generally the "UserAgent" http header will indicate
this and such information is normally easily to get at application
layer(such as HTTP or ASP.NET webapplicaiton), the server-side even do not
need to hook into the raw network or transport layer to get the messages.

2. If what you want to do is find the FLV links, I think a straightforward
means is to parse the html resposne content of each web page and look for
the FLV links(through url extensions or <object> tag). For general html
parsing, you can have a look at the following web articles:

#Parsing HTML in Microsoft C#
http://www.developer.com/net/csharp/article.php/2230091

#Parsing HTML without Using the Browser Control
http://www.codeguru.com/vb/vb_internet/html/article.php/c4815/

#A Simple Crawler Using C# Sockets
http://www.codeproject.com/KB/IP/Crawler.aspx

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.



--------------------