Relative/absolute url parsing

[Michel ( Dagnan )]

Hi!

I have to collect urls from html files, I to transform relative urls to
absolute, I have to handle for example url beginning with '../' and
'./', which is kind of ennoying (my head hurts because of the
debugging process)

Actually, I test every case using regexp. Maybe you can help me finding
something faster?

Thanks.

 

[Alex Young]

Hi!

I have to collect urls from html files, I to transform relative urls to
absolute, I have to handle for example url beginning with '../' and
'./', which is kind of ennoying (my head hurts because of the
debugging process)

Actually, I test every case using regexp. Maybe you can help me finding
something faster?

If you can shoe-horn your problem into Mechanize, it's got a private
method WWW::Mechanize#to_absolute_uri, which does precisely this. Don't
know if that's of any use, but it might be worthwhile looking at how
it's implemented at least.

 

[Peter Szinek]

Hi,

If you can shoe-horn your problem into Mechanize, it's got a private
method WWW::Mechanize#to_absolute_uri, which does precisely this. Don't
know if that's of any use, but it might be worthwhile looking at how
it's implemented at least.

+1 for Alex's solution.

I have tried to implement this in scRUBYt! (I did not know Mechanize's
to_absolute_uri back then) and, well, failed (fortunately I discovered
it in Mechanize since then). My solution worked for 99% of the cases,
but the rest was totally PITA to hunt down. I believe Aaron and the
mechanize community already did this, so why reinvent the wheel?
Believe me, if you can shoe-horn it into Mechanize as Alex suggested, do
it - it will save you lot of time, nerves, headaches, money, whatnot.

Cheers,
Peter
_
http://www.rubyrailways.com :: Ruby and Web2.0 blog
http://scrubyt.org :: Ruby web scraping framework
http://rubykitchensink.ca/ :: The indexed archive of all things Ruby.

 

[Alex Fenton]

Hi!

I have to collect urls from html files, I to transform relative urls to
absolute, I have to handle for example url beginning with '../' and
'./', which is kind of ennoying (my head hurts because of the
debugging process)

Not sure, but perhaps the standard library will do what you want?

Assuming that this_page is the URL of the page you're scraping links from and '../qux' is the link URL you want to absolutise:

SCIPIUS:~ alex$ irb
irb(main):001:0> require 'uri'
=> true
irb(main):002:0> this_page = URI.parse('http://www.abc.org/foo/bar/baz')
=> #<URI::HTTP:0x18c82e URL:http://www.abc.org/foo/bar/baz>
irb(main):003:0> this_page.merge('../qux')
=> #<URI::HTTP:0x303d42 URL:http://www.abc.org/foo/qux>

alex

 

[Michel ( Dagnan )]

Cant find WWW::Mechanize#to_absolute_uri method (in rdoc), but you're
right, the URI lib does exactly what I want.

Ruby is so wonderful

Thank you for helping.

 

[Alex Young]

Cant find WWW::Mechanize#to_absolute_uri method (in rdoc), but you're
right, the URI lib does exactly what I want.

It's a private method - you'll have to do something like
agent.sendto_absolute_uri, path) to get it to work.

Ruby is so wonderful

I know