Domain Names

[Terry Burns]

Hi Guys,

I know this is a bit off topic, but im not having any luck googling it.

I want to know where I can go programatically to get a list of all
registered domains on the internet.


Cheers

 

[Mark Rae]

I want to know where I can go programatically to get a list of all
registered domains on the internet.

All registered domains on the Internet...???

There must literally be millions of them!

 

[Brendan Reynolds]

All registered domains on the Internet...???

There must literally be millions of them!

Tens of millions, in fact ...

http://www.whois.sc/internet-statistics/

 

[Terry Burns]

--
Terry Burns
http://TrainingOn.net

Tens of millions, in fact ...

http://www.whois.sc/internet-statistics/

No, i mean is there a way to retrieve a list of domain names from the DNS
server and display them in a database using vb.net.

i was also wondering if it is possible to find the domain name by resolving
it with the ip Address, not sure if that is possible but

any help would be much appreciate .

Thanks Guys.

 

[Brian]

--
Terry Burns
http://TrainingOn.net


No, i mean is there a way to retrieve a list of domain names from the DNS
server and display them in a database using vb.net.

"A list of domain names from the DNS server" doesn't make any sense to me.

Dig is the most versatile DNS client I know (that is, it's better than
nslookup or library calls). Take a look at the man page. It'll give you an
idea of how DNS is expected to be indexed. As far as I know, domain lists
don't exist in a single source. You could call DNS the first federated
database I suppose. If you want to know all the delegates of a given
authority, well, again, I don't think you can even get that.

You can pull A records to your hearts content tho.

i was also wondering if it is possible to find the domain name by
resolving it with the ip Address, not sure if that is possible but

Besides being a living thing, FQDNs and dotted quads are a many to many
relationship. You can also map dandeliones in your neighborhood. I
question the wisdom of both.

 

[Scott M.]

So, you want to programmatically build what a DNS server already does?

 

[Terry Burns]

yes, i am building a crawler which returns the links on a website and stores
them in a database, i am trying to get domain names by
testing to see if i can resolve the ip address, incrementing them by one
each time until it hits a domain, it then stores in a database
and can be used later.

 

[Erik Funkenbusch]

I want to know where I can go programatically to get a list of all
registered domains on the internet.

The only reliable way to do this is to linearly query each possible domain
name starting with a.com, a.net, a.org.. etc.. and ending with
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.com,
..org, etc... (actual maximimum length is probably much longer, that was
just an example).

Unless you have serious bandwidth, and lots of machines to do simultaneous
work, it's not likely you would complete this task in any decent timeframe,
and likely by the time you finished it would be out of date anyways, as
domains expire or are bought daily.

 

[Terry Burns]

Just a quick note here. The Original Post Was made by my Son on my machine.
Just wanted to clear this up. He is sixteen and trying to find out about the
internet and wants to use .net to do it.

Having said this, his question is an interesting one, im not quite sure how
crawlers get notification of newly registered domain names in order to crawl
them.

Does anyone know ? It would seem to me that the DNS servers have all the
information one needs, its just about how to get it ?

 

[Erik Funkenbusch]

Having said this, his question is an interesting one, im not quite sure how
crawlers get notification of newly registered domain names in order to crawl
them.

They don't get notification. They have to just query dns for all possible
name combinations. Unless of course they have some kind of backdoor deal
with a registrar (or someone that works for one). But even then, they'd
only know about accounts registered through that registrar.

Does anyone know ? It would seem to me that the DNS servers have all the
information one needs, its just about how to get it ?

No. DNS is a distributed architecture. The root DNS servers do have all
possible domain names in them, but they just defer to whatever DNS server
is defined for that name to provide everything else. This is a query-only
service. You have to know what you're looking for to get an answer.

 

[Otis Mukinfus]

Just a quick note here. The Original Post Was made by my Son on my machine.
Just wanted to clear this up. He is sixteen and trying to find out about the
internet and wants to use .net to do it.

Having said this, his question is an interesting one, im not quite sure how
crawlers get notification of newly registered domain names in order to crawl
them.

Does anyone know ? It would seem to me that the DNS servers have all the
information one needs, its just about how to get it ?

Crawlers are EVIL!!!

Otis Mukinfus
http://www.arltex.com
http://www.tomchilders.com

 

[Terry Burns]

That was a perticularly unhelpful response. Crawlers are not evil, google is
a crawler and is very useful.

 

[Ray Booysen]

Hi Guys,

I know this is a bit off topic, but im not having any luck googling it.

I want to know where I can go programatically to get a list of all
registered domains on the internet.


Cheers

Hahah, you're joking right?

If you really wanted to do that do a loop on every domain name.

E.g.

a.com
b.com
c.com
..
..
..
aa.com

Then you can do .co.uk, .org, .info and in all overload your ISP's DNS
server nicely

 

[Guest]

Terry,
I don't believe that crawlers (e.g. "googlebot") are domain-aware. They
simply follow links. If you put up a new domain and web site, google is not
going to find it all by itself. Submit the site, and yes, it will get on the
crawler's "ToDo" list.
Peter

 

[Brendan Reynolds]

I'm afraid I can't answer the specific question about how to get the
information from the DNS servers. But the idea of storing that information
in a database does not seem to me to be a practical one. According to that
link I posted earlier, 1,584,998 new domains were registered, and 530,419
previously existing domains were deleted, in the past 24 hours alone. (Data
as of 27 March 2006 at 10:40 GMT). That's more than 18 new domains
registered, and more than 6 existing domains deleted, every second.

 

[OHM \( One Handed Man \)]

Yes, I understand this, but I would probably only need a small subset of
these, just a few million; the allocation of an IP address or range can be
looked up for the area in question. So I would be searching all domains whos
IP address has a UK Origin.

 

[OHM \( One Handed Man \)]

Ive registered a site about a year and a half ago and it was crawled two
months after registrations without any submission from me, so Im not sure
how that happened.

I've managed to get it working now, but its so slow that it will never get
all the domains quick enough.