Server response validation?

[Tuxedo]

This may be a vague question but which perl network tools or procedures are
good to determine if a hostname or IP exist and that is suitable for a cgi
request that allows user-input of any URL string?

An HTTP status code may return any response, such as 200, 404 or something
else. Whatever the response may be, as long as there is an actual response,
DNS record or server existing, a subroutine should return true. If however
the hostname or IP is non-existing, or if the user input has a wrong
syntax, the perl code should return false.

Presumably countless modules and procedures exist that do this and more.
Any ideas for good solutions would be much appreciated.

Many thanks,
Tuxedo

 

[Tuxedo]

The core gethostbyname function will tell you if a given name has an
address in the DNS. The Net:NS module will let you make more specific
queries if you need to.

Thanks for the advise, most helpful!

I'm not sure what I need yet but before making DNS requests I think would
be a good idea to validate the URL input.

I read that gethostbyname is not IPv6 compatible which may run into trouble:
http://stackoverflow.com/questions/11325457/perl-gethostbyname-when-given-ip

The Net:NS tools look interesting and in combination with Net::LibIDN it
appears to support IDNs.

I'm not quite sure how to validate a URI and I guess there is no simple way
to do so by existing core modules.

To cover a growing number of IDNs, IDNsNet-IDN-Encode-2.003 seem to be a
good tool in converting IDNs to their ASCII representations:
http://search.cpan.org/~cfaerber/Net-IDN-Encode-2.003/lib/Net/IDN/Encode.pm

I will test Data::Validate::URI first, which require various non-core
modules, namely Data::Validate::IP, Net:omain::TLD as well as
Net::Netmask.

Determining if an IP exists is harder, since you can't in general assume
any given IP will respond to any particular protocol. Net:ing would be
a start, but a lot of sites block external pings (even though they
shouldn't).


Oh, you didn't say 'HTTP'. If you want to check an HTTP server exists,
you can either just open a connection to port 80 with IO::Socket::INET
(and close it again without sending a response), or send a HEAD request
with LWP::Simple.

You should be aware that some sites may consider this sort of behaviour
to be port-scanning, and may take action to prevent you from doing it.
In particular, you should be very careful not to probe any particular
server too often, as that would almost certainly be considered abuse.

Thanks for the words of warning.

Tuxedo

 

[Tuxedo]

Ben Morrow wrote:

[...]

Socket6.pm will give you the getaddrinfo function for making v4/v6
hostname lookups.
[...]

Well, URI.pm will do basic syntactic validation, which is a start, or
you could use Regexp::Common, which has patterns for validating URIs.
[...]

Hah! You seem to be solving this on your own... good. I didn't know
about that. Note that it doesn't appear to do any network tests, so a
URI like http://example.com/foo/bar will pass.

Yes, I read that in the docs. My first step will be to validate a URI
format and not that it actually exists through a network request. The URI
may be standard domain, a new style IDN representation or even an IP. The
network request comes after. So thanks for Socket6.pm, URI.pm and
Regexp::Common. I will test all!

Tuxedo

 

[Peter J. Holzer]

Note that it doesn't appear to do any network tests, so a URI like
http://example.com/foo/bar will pass.

% wget http://example.com/foo/bar
--2012-11-11 13:43:26-- http://example.com/foo/bar
Resolving example.com (example.com)... 2001:500:88:200::10, 192.0.43.10
Connecting to example.com (example.com)|2001:500:88:200::10|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.iana.org/domains/example/ [following]
--2012-11-11 13:43:27-- http://www.iana.org/domains/example/
Resolving www.iana.org (www.iana.org)... 2620:0:2d0:200::8, 192.0.32.8
Connecting to www.iana.org (www.iana.org)|2620:0:2d0:200::8|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `bar'

[ <=> ] 2,966 --.-K/s in 0.001s

2012-11-11 13:43:28 (3.39 MB/s) - `bar' saved [2966]

SCNR,
hp

 

[Tuxedo]

What does it mean for an IP to exist? Does a hostname exist if there
is an AAAA but no A? How do you want to treat CNAME?

I'm not quite sure.

You really need to nail down the specifics of the problem before you
look for a solution.

You are right, my questions were posted somewhat prematurely.

Have you looked at Regexp-Common or URI
<http://search.cpan.org/~gaas/URI-1.60/>?

Thanks, I'm looking into these. In the meantime, I found that
Data::Validate::URI appears to be working well in validating http and/or
https URI's in case they are the only things one need to do.

Tuxedo