looping through & downloading web page - WATIR codes

C

curious

What I am trying to realize through code below is to read a line by
line in items.txt, and download each ensuing pages by having WATIR
click 'Next' link.

Now what I wish this code to do is when WATIR reaches the last page,
and there is no more "Next" link on the page, then go to the second
line in items.txt, searching again, and download ensuing pages again.

But unfortunately, the WATIR code close down when there is no more
"Next" link on the first item search.. and it does not go to searching
with the second line in items.txt.

Could anybody help me how I should adjust the code below so that when
there is no more "Next" link on the page, then the second line in
items.txt goes into searching, download each page by clicking "Next"
link.. on and on??

any suggestion will be deeply appreciated.

Thanks.

=====================================================================
require 'watir'

ie = Watir::IE.start("http://www.test.com")

file = File.open("c:/items.txt", "r")
lines = file.readlines()

PageNo = 1

0.upto(lines.length - 1) {|j|

$item = lines[j]

ie.text_field:)name, "item").value = $item

ie.button:)name, "Search").click

a = ie.contains_text("Next")

while a

File.open("#{PageNo}.html", "wb") { |f|

f << '<html>'
f << ie.html
f << '</html>'

PageNo = PageNo + 1

ie.link:)text, "Next").click

}

end

}

file.close()
lines.clear()
 
C

curious

I found out what the problem is..

it is the last part..

ie.link:)text, "Next").click.. in the last web page, although "Next"
text is contained, link is not present since it is the last page..

Is there anyway, I can control "ie.link:)text, "Next").click" by
checking the presence of link beforehand?? so if link is present, then
proceed, otherwise.. halt.. I think my code will do as I wish once
this gets done..

thanks.

Paul said:
curious said:
What I am trying to realize through code below is to read a line by
line in items.txt, and download each ensuing pages by having WATIR
click 'Next' link.

Now what I wish this code to do is when WATIR reaches the last page,
and there is no more "Next" link on the page, then go to the second
line in items.txt, searching again, and download ensuing pages again.

But unfortunately, the WATIR code close down when there is no more
"Next" link on the first item search.. and it does not go to searching
with the second line in items.txt.

Could anybody help me how I should adjust the code below so that when
there is no more "Next" link on the page, then the second line in
items.txt goes into searching, download each page by clicking "Next"
link.. on and on??

any suggestion will be deeply appreciated.

Rather than simply rewriting your code for you, I offer these suggestions:

1. Always close the opened file handle when the write is completed. You are
not closing your file handles until the last file is read, which in the
worst case leaves many file handles open and abandoned.

2. Instead of doing this:

0.upto(lines.length - 1) {|j|
$item = lines[j]

Do it this way:

lines.each do |item|
# ... your code here
end

3. Avoid use of global variables when possible.

I would offer more suggestions but I don't actually understand what your
program is meant to accomplish.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,581
Members
45,055
Latest member
SlimSparkKetoACVReview

Latest Threads

Top