only once in storage

[a]

Hi,

I am writing a script to download all the links of the whole site. The link
of the web site is not a simple tree. There may be some replicated links
pointing to the same location.

So, I need to walk through the site to extract and push the URLs of each
page into a data structure.
I dont want the replicated links. Every link should only appear once in my
storage.
So, is there any effective way to achieve this?

Thanks

 

[Gunnar Hjalmarsson]

I am writing a script to download all the links of the whole site. The link
of the web site is not a simple tree. There may be some replicated links
pointing to the same location.

So, I need to walk through the site to extract and push the URLs of each
page into a data structure.
I dont want the replicated links. Every link should only appear once in my
storage.
So, is there any effective way to achieve this?

Use a hash.

 

[Peter Makholm]

So, I need to walk through the site to extract and push the URLs of each
page into a data structure.
I dont want the replicated links. Every link should only appear once in my
storage.

You can use a hash to check if an url has been seen before:

my %seen;
my $url;

while ($url = getnext()) {
process_url($url) unless $seen{$url}++;
}

//Makholm

 

[Josef Moellers]

You can use a hash to check if an url has been seen before:

my %seen;
my $url;

while ($url = getnext()) {
process_url($url) unless $seen{$url}++;
}

I'd split that:

1. collect all urls:
$urls{getnext()} = 1;
2. process all unique urls:
process_url($_) foreach (keys %urls);

 

[Peter Makholm]

I'd split that:

1. collect all urls:
$urls{getnext()} = 1;
2. process all unique urls:
process_url($_) foreach (keys %urls);

That would not work if getnext() return a element from a work queue
and process_url() inserted urls in the work queue based on the content
fetched from the url.

I would probally do the check while inserting into the work queue. So
in the above example getnext() is part of the extract urls from
content and process_url is inserting in work queue.

/Makholm

 

[Josef Moellers]

That would not work if getnext() return a element from a work queue
and process_url() inserted urls in the work queue based on the content
fetched from the url.

I would probally do the check while inserting into the work queue. So
in the above example getnext() is part of the extract urls from
content and process_url is inserting in work queue.

Yes, sorry, I ignored the "the web site is not a simple tree".