More regex help

[Support Desk]

I am working on a python webcrawler, that will extract all links from an
html page, and add them to a queue, The problem I am having is building
absolute links from relative links, as there are so many different types of
relative links. If I just append the relative links to the current url, some
websites will send it into a never-ending loop.

What I am looking for is a regexp that will extract the root url from any
url string I pass to it, such as

'http://example.com/stuff/stuff/morestuff/index.html'

Regexp = http:example.com

'http://anotherexample.com/stuff/index.php

Regexp = 'http://anotherexample.com/

'http://example.com/stuff/stuff/

Regext = 'http://example.com'

 

[Support Desk]

Kirk,

That's exactly what I needed. Thx!


-----Original Message-----
From: Kirk Strauser [mailto:[email protected]]
Sent: Wednesday, September 24, 2008 11:42 AM
To: (e-mail address removed)
Subject: Re: More regex help

I am working on a python webcrawler, that will extract all links from an
html page, and add them to a queue, The problem I am having is building
absolute links from relative links, as there are so many different types of
relative links. If I just append the relative links to the current url, some
websites will send it into a never-ending loop.

'/foo')
'http://www.example.com/foo' 'http://slashdot.org/foo')
'http://slashdot.org/foo'