Forums
New posts
Search forums
Members
Current visitors
Log in
Register
What's new
Search
Search
Search titles only
By:
New posts
Search forums
Menu
Log in
Register
Install the app
Install
Forums
Archive
Archive
Ruby
how to view net http get response for a web spider tutorial
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Message
[QUOTE="anne001, post: 4513102"] I am trying to run the ruby lynux tutorial on web spiders [URL]http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf[/URL] Apparently, the information has moved on the livejournal.com. The tutorial says "you can get a wealth of information from the http response object returned by the net http get function" If I use puts data, I get The document has moved <A HREF="[URL]http://anne.livejournal.com/profile[/URL]">here</A>.<P if I use puts resp, I get `puts': stack level too deep (SystemStackError) Is the tutorial refering to the data part of the response only? Here is a cgi script on livejournal #!/usr/bin/perl use LWP::Simple; print "Content-type: text/html\n\n"; print get('[URL]http://www.livejournal.com/customview.cgi[/URL]' . '?username=username&styleid=101'); [URL]http://www.livejournal.com/developer/embedding.bml?method=cgi[/URL] Here is the program I am working with --------------------------------------------------------> require 'net/http' h=Net::HTTP.new('[URL="http://www.livejournal.com"]www.livejournal.com[/URL]', 80) friend_arr = [] person= ARGV[0] resp, data = h.get("[URL]http://www.livejournal.com/userinfo.bml?user=#[/URL]{person}",nil) print "Friend list for #{person}\n" #puts resp puts data data.split("\n").each do |line| line.split(",").each do |token| if token =~ /userinfo.bml\?user=([^'&]*)\'/ friend_arr.push #1 print #$1\n" end end end print "\n" friend_arr.each do |friend| print "Parsing #{friend}'s. journal for #{person}'s. comments...\n"; f=File.new("#{person}_#{friend}.txt","w") f.puts 'ruby parse_journal.rb #{friend}#{person}' f.close end [/QUOTE]
Verification
Post reply
Forums
Archive
Archive
Ruby
how to view net http get response for a web spider tutorial
Top