Server redirected too many times - authentication issue

[konrad-g]

Hello,

here's my problem. I'm writing a crawler which needs to access password
protected ASP pages. I know the password protected URL, but accessing
it without login invokes login screen, which (after logging in)
redirects me back to the page. I know URL of that login page too.

Here's my code:

/************** Code begins *************/
URL url = new URL( urlString );
HttpURLConnection c = (HttpURLConnection)url.openConnection();
c.setRequestMethod("POST");
c.setDoInput(true);
c.setDoOutput(true);
c.setUseCaches(false);
OutputStream urlOut = c.getOutputStream();
urlOut.write(("LoginControl1:txtEmail="+URLEncoder.encode("login")+"&LoginC­ontrol1:txtPassword="+URLEncoder.encode("pass")).getBytes());



InputStream content = (InputStream)c.getInputStream();
BufferedReader in = new BufferedReader (new InputStreamReader
(content));
String line;
while ((line = in.readLine()) != null) System.out.println (line);
/************** Code ends *************/

I tried to set up urlString for two values. First of them is login
page, which (after logging in) should redirect me to desired page.
Using it ends up with "Server redirected too
many times" exception. The second option is using desired page's URL
as urlString - but in this case final "while" loop produces login
screen instead of desired page.

Any clues?

Thanks in advance,
Konrad Garus

 

[tom fredriksen]

I tried to set up urlString for two values. First of them is login
page, which (after logging in) should redirect me to desired page.
Using it ends up with "Server redirected too
many times" exception. The second option is using desired page's URL
as urlString - but in this case final "while" loop produces login
screen instead of desired page.

It seems to me that you have ended up in an infinite loop, which the
server is detecting and in the end warning you about. What is probably
happening is that you are not performing a correct login so you are
routed back to the original page, which are routing you to the login
page, where you fail and so on. Not sure if that is correct but it would
seem like it.

So I suggest two things, confirm that this or something similar is the
case first. You can do this by adding code which performs the login
sequence once. Then you print what you are sending and what you are
receiving to check or study whats going on.
Secondly, you need to find out the correct way to perform login. If it
uses HTTP Authentication, which it seems like since you are describing a
login pop up, then its easy to solve this programatically just read the
rfc. If it is a custom made login page, then you need the details of it
to solve the puzzle.

/tom