Reply to thread

Message: [QUOTE="anne001, post: 4513140"] Thank you, I will use the proc then. I figured the output problem. The puts is redirected to f.puts, so if I want to see the output, I have to go to the file!!! Took me a while! Thank you for your suggestions: save all http gets to corresponding files, and before doing a get, check if the file is not already on the local server. And waiting every 5 gets should be easy enough. Since the livejournal does not provide comment info anymore in their RSS feed, you have to open every blog, it is already so slow as it is I did not put this change in. Here are the two programs in case someone is interested in this linus tutorial. parse_user.rb --------------------> require 'net/http' h=Net::HTTP.new('[URL="http://www.livejournal.com"]www.livejournal.com[/URL]', 80) friend_arr = [] person= ARGV[0] resp, data = h.get("[URL]http://#[/URL]{person}.livejournal.com/profile",nil) print "Friend list for #{person}\n" data.split("\n").each do |line| line.split(",").each do |token| if token =~ /http:\/\/([^'&]*).livejournal.com\/profile/ friend_arr.push $1 print "#$1\n" end end end print "\n" friend_arr.each do |friend| if friend != person print "Parsing #{friend}'s. journal for #{person}'s. comments...\n"; File.open("#{person}_#{friend}.txt","w") { |f| f.puts %x[ruby parse_journal.rb #{friend} #{person}] } end end parse_journal.rb ------------------------> require 'net/http' person= ARGV[0] watch_for= ARGV[1] h=Net::HTTP.new('[URL="http://www.livejournal.com',80"]www.livejournal.com',80[/URL]) ar =[]; resp, data = h.get("[URL]http://#[/URL]{person}.livejournal.com/data/rss", nil) data.split("\n").each do |line| if line =~ /#{person}.livejournal.com\/([0-9]*).html/ ar.push $1.to_i end end # to keep one copy of uniq # url_array = ar.uniq.find_all {|x| ar.find_all {|y| y == x }.size == 1 } # to keep one copy of non uniq url_array = ar.uniq.find_all {|x| ar.find_all {|y| y == x }.size > 1 } #url_array=[ 1328508, 13268887] url_array.each { |urlid| resp, data = h.get("http:\/\/#{person}.livejournal.com\/#{urlid}.html", nil) lkforp=1 data.split("\n").each do |line| line.split(",").each do |token| if (lkforp ==1) if token =~ /http:\/\/([^'&]*).livejournal.com\/profile/ if( $1 == watch_for) lkforp=2 end end else if token =~ /(http:\/\/#{person}.livejournal.com\/#{urlid}.html[^']thread=[^']*)\'(.*)<\/td><\/tr>\<tr><td>(.*)<p style=\'margin/ url = $1 comment = $3 comment.gsub!('<br />',"\n") comment.gsub!(/<\/*[^>]*>/,'') print "#{url}\n#{comment}\n" lkforp=1 end end end end } [/QUOTE]

Verification