gethostbyname_ex(hostname) extremely slow (crossposted from stackoverflow)

[anntzer.lee]

Hi,

At startup, IPython (qtconsole) calls "socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP addresses that point to the machine. On a Linux server that I manage this call is extremely slow (>20s)... which I have trouble understanding as "ip addr show" seems to give the same information nearly instantaneously. Is there anything I can do to make this faster? Can this be a network configuration issue (I am behind a router)?

This issue is independent of IPython:

$ time python -c 'import socket; print(socket.gethostbyname_ex(socket.gethostname())[2])'
['192.168.0.102']
python -c 0.07s user 0.02s system 0% cpu 28.190 total

Thanks.

Antony

 

[Roy Smith]

Hi,

At startup, IPython (qtconsole) calls
"socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP
addresses that point to the machine. On a Linux server that I manage this
call is extremely slow (>20s)... which I have trouble understanding as "ip
addr show" seems to give the same information nearly instantaneously. Is
there anything I can do to make this faster? Can this be a network
configuration issue (I am behind a router)?

It's almost certainly some sort of configuration issue which is causing
some DNS query to timeout. The first step, is to figure out which part
is taking so long. Open up a python shell and run:

see how long that takes and what it returns. Then, assuming it returns
a string containing your hostname (massive handwave about what that
actually means), try

and see how long that takes and what it returns. At least at that point
you'll have cut the problem in half.

If I had to guess, you've got a missing PTR record, because that's what
most people screw up.

 

[anntzer.lee]

It is the call to gethostbyname_ex that is very slow. The call to gethostname is quick (and returns the same string as /usr/bin/hostname).

(e-mail address removed) wrote:

Hi,

At startup, IPython (qtconsole) calls

"socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP

addresses that point to the machine. On a Linux server that I manage this

call is extremely slow (>20s)... which I have trouble understanding as "ip

addr show" seems to give the same information nearly instantaneously. Is

there anything I can do to make this faster? Can this be a network

configuration issue (I am behind a router)?

It's almost certainly some sort of configuration issue which is causing

some DNS query to timeout. The first step, is to figure out which part

is taking so long. Open up a python shell and run:





see how long that takes and what it returns. Then, assuming it returns

a string containing your hostname (massive handwave about what that

actually means), try





and see how long that takes and what it returns. At least at that point

you'll have cut the problem in half.



If I had to guess, you've got a missing PTR record, because that's what

most people screw up.

 

[Michael Torrie]

It is the call to gethostbyname_ex that is very slow. The call to
gethostname is quick (and returns the same string as
/usr/bin/hostname).

What gethostbyname_ex and /usr/bin/hostname do are very different
things. gethostbyname_ex does a DNS lookup against a server.
/usr/bin/hostname just checks a local computer setting. I don't see why
you are comparing the two. /usr/bin/hostname is not going to help you
find a list of IP addresses that point to a machine.

 

[Roy Smith]

It is the call to gethostbyname_ex that is very slow. The call to
gethostname is quick (and returns the same string as /usr/bin/hostname).

First, please stop posting with Google Groups. It makes a total mess of
the quoted text.

Next, what happens when you use the command-line tools (nslookup or dig)
to translate your hostname to an IP address, and what happens if you
then try to reverse translate that IP address back to a name?

This issue is independent of IPython:

$ time python -c 'import socket;
print(socket.gethostbyname_ex(socket.gethostname())[2])'
['192.168.0.102']
python -c 0.07s user 0.02s system 0% cpu 28.190 total

The real point is not that it's independent of IPython, it has nothing
to do with Python at all. What you've got here is a network
configuration issue. You would do better to recreate this problem with
the native OS command line tools and then ask about it on a forum
dedicated to your operating system.

 

[Chris Angelico]

At startup, IPython (qtconsole) calls "socket.gethostbyname_ex(socket.gethostname())[2]" to find a list of IP addresses that point to the machine. On a Linux server that I manage this call is extremely slow (>20s)... which I have trouble understanding as "ip addr show" seems to give the same information nearly instantaneously. Is there anything I can do to make this faster? Can this be a network configuration issue (I am behind a router)?

Yes, it most definitely CAN be a network config issue. The C function
you want to be calling is getifaddrs(), and I don't think there's a
way to call that from core Python. But a Google search for 'python
getifaddrs' shows up a few third-party modules that might be of use to
you; that'd be a lot quicker and more reliable than trying to look up
your own hostname and depending on the results.

ChrisA

 

[anntzer.lee]

What gethostbyname_ex and /usr/bin/hostname do are very different
things. gethostbyname_ex does a DNS lookup against a server.
/usr/bin/hostname just checks a local computer setting. I don't see why
you are comparing the two. /usr/bin/hostname is not going to help you
find a list of IP addresses that point to a machine.

I was just replying to the previous comment "name = socket.gethostname() see how long that takes and what it returns. Then, assuming it returns a string containing your hostname (massive handwave about what that actually means)", saying that gethostname resolves my own hostname instantaneously.

 

[anntzer.lee]

Yes, it most definitely CAN be a network config issue. The C function
you want to be calling is getifaddrs(), and I don't think there's a
way to call that from core Python. But a Google search for 'python
getifaddrs' shows up a few third-party modules that might be of use to
you; that'd be a lot quicker and more reliable than trying to look up
your own hostname and depending on the results.

ChrisA

I tried using netifaces (https://pypi.python.org/pypi/netifaces) which seems to rely on getifaddrs (according to the doc, I didn't check the source). Again, it returns nearly instantaneously the correct IP address.

 

[Chris Angelico]

I tried using netifaces (https://pypi.python.org/pypi/netifaces) which seems to rely on getifaddrs (according to the doc, I didn't check the source). Again, it returns nearly instantaneously the correct IP address.

Perfect!

ChrisA

 

[anntzer.lee]

Perfect!

ChrisA

Not really for my use case -- it isn't that *I* want to know my public IP address, but rather that IPython wants to know it. Of course I could patch IPython's source to use netifaces but that sounds like an overkill.

As it happens I found a better way: just add the proper entry to /etc/hosts.

Still, thanks for the suggestions.

Antony

 

[Chris Angelico]

Not really for my use case -- it isn't that *I* want to know my public IP address, but rather that IPython wants to know it. Of course I could patch IPython's source to use netifaces but that sounds like an overkill.

As it happens I found a better way: just add the proper entry to /etc/hosts.

Ah, I see what you mean.

If it were possible with core Python, I would recommend raising a bug
with IPython, as the current method is susceptible to external issues.
But to just get your problem solved, yes, a host file entry sounds
like it's the best way to go.

ChrisA

 

[Roy Smith]

As it happens I found a better way: just add the proper entry to /etc/hosts.

You have not found a better way. You still have a network (or more
specifically, DNS) configuration that's broken.

What you have found is a pragmatic way to solve your immediate problem
and get some work done. That is certainly useful (and I've done it
plenty of times), but you need to understand that what you've done is
hidden the problem, not solved it.

 

[anntzer.lee]

You have not found a better way. You still have a network (or more
specifically, DNS) configuration that's broken.

What you have found is a pragmatic way to solve your immediate problem
and get some work done. That is certainly useful (and I've done it
plenty of times), but you need to understand that what you've done is
hidden the problem, not solved it.

To be honest, knowing nothing about DNS configuration, I don't even know if adding the entry to /etc/hosts is the "proper" fix or if the issue should be fixed somewhere else (or perhaps "didn't know", as you seem to imply that that is not the correct way).

 

[Chris Angelico]

To be honest, knowing nothing about DNS configuration, I don't even know if adding the entry to /etc/hosts is the "proper" fix or if the issue should be fixed somewhere else (or perhaps "didn't know", as you seem to imply that that is not the correct way).

Since you can't change the code, it's almost certainly the best
solution available to you. It's more-or-less equivalent to speeding up
your DNS lookups massively. The main downside is that if something
changes, you need to change it in DNS and in your hosts file; as Roy
says, it's a pragmatic solution rather than a perfect one.

ChrisA