Re: Subprocess puzzle and two questions

Discussion in 'Python' started by Roy Smith, Nov 14, 2012.

  1. Roy Smith

    Roy Smith Guest

    In article <>,
    wrote:

    > I need to time the operation of a command-line utility (specifically
    > nslookup) from within a python program I'm writing.


    Ugh. Why are you doing this? Shelling out to nslookup is an incredibly
    slow and clumsy way of doing name translation. What you really want to
    be doing is calling getaddrinfo() directly.

    See http://docs.python.org/2/library/socket.html#socket.getaddrinfo for
    details.
    Roy Smith, Nov 14, 2012
    #1
    1. Advertising

  2. Roy Smith

    Roy Smith Guest

    In article <>,
    William Ray Wing <> wrote:

    > On Nov 13, 2012, at 11:41 PM, Roy Smith <> wrote:
    >
    > > In article <>,
    > > wrote:
    > >
    > >> I need to time the operation of a command-line utility (specifically
    > >> nslookup) from within a python program I'm writing.

    > >
    > > Ugh. Why are you doing this? Shelling out to nslookup is an incredibly
    > > slow and clumsy way of doing name translation. What you really want to
    > > be doing is calling getaddrinfo() directly.
    > >
    > > See http://docs.python.org/2/library/socket.html#socket.getaddrinfo for
    > > details.
    > > --

    > Because, unless I'm badly mistaken (very possible), getaddrinfo doesn't let
    > me specify the server from which the name is returned. I'm really not after
    > the name, what I'm REALLY after is the fact that a path exists to the name
    > server I specify (and how long it takes to respond). In the "good old days" I
    > would just have ping'd it, but these days more and more DNS boxes (and
    > servers of all sorts) are shutting off their ping response.
    >
    > Thanks, Bill


    Oh, my. You're using DNS as a replacement for ping? Fair enough. In
    that case, all you really care about is that you can connect to port 53
    on the server...

    import socket
    import time
    s = socket.socket()
    t0 = time.time()
    s.connect(('8.8.8.8', 53))
    t1 = time.time()
    print "it took %f seconds to connect" % (t1 - t0)
    Roy Smith, Nov 14, 2012
    #2
    1. Advertising

  3. On Thu, Nov 15, 2012 at 1:22 AM, Roy Smith <> wrote:
    > Oh, my. You're using DNS as a replacement for ping? Fair enough. In
    > that case, all you really care about is that you can connect to port 53
    > on the server...
    >
    > import socket
    > import time
    > s = socket.socket()
    > t0 = time.time()
    > s.connect(('8.8.8.8', 53))
    > t1 = time.time()
    > print "it took %f seconds to connect" % (t1 - t0)


    That assumes that (a) the remote server supports TCP for DNS (since
    UDP is by far the more often used, some name servers don't bother
    supporting TCP), and (b) that connection time for TCP is comparable to
    ping or an actual DNS lookup. But in terms of approximating your
    connection times, that's gotta be way better than shelling out to
    several other processes.

    ChrisA
    Chris Angelico, Nov 14, 2012
    #3
  4. Roy Smith

    Roy Smith Guest

    I wrote:
    >> Oh, my. You're using DNS as a replacement for ping? Fair enough. In
    >> that case, all you really care about is that you can connect to port 53
    >> on the server...
    >>
    >> s = socket.socket()
    >> s.connect(('8.8.8.8', 53))


    In article <>,
    Chris Angelico <> wrote:
    >That assumes that (a) the remote server supports TCP for DNS


    This is true. I honestly don't know what percentage of DNS servers
    out there only support UDP. The two I tried (Google's 8.8.8.8, and my
    Apple TimeCapsule) both supported TCP, but that's hardly a
    representitive sample.

    > and (b) that connection time for TCP is comparable to
    > ping or an actual DNS lookup.


    My first thought to solve both of these is that it shouldn't be too
    hard to hand-craft a minimal DNS query and send it over UDP. Then, I
    hunted around a bit and found that somebody had already done that, in
    spades. Take a look at http://www.dnspython.org; it might be exactly
    what's needed here.
    Roy Smith, Nov 14, 2012
    #4
  5. On Thu, Nov 15, 2012 at 3:20 AM, Roy Smith <> wrote:
    > I wrote:
    >>> Oh, my. You're using DNS as a replacement for ping? Fair enough. In
    >>> that case, all you really care about is that you can connect to port 53
    >>> on the server...
    >>>
    >>> s = socket.socket()
    >>> s.connect(('8.8.8.8', 53))

    >
    > In article <>,
    > Chris Angelico <> wrote:
    >>That assumes that (a) the remote server supports TCP for DNS

    >
    > This is true. I honestly don't know what percentage of DNS servers
    > out there only support UDP. The two I tried (Google's 8.8.8.8, and my
    > Apple TimeCapsule) both supported TCP, but that's hardly a
    > representitive sample.


    I don't know either, all I know is that DNSReport recommends
    supporting TCP, and none of my DNS servers ever fail that check.

    >> and (b) that connection time for TCP is comparable to
    >> ping or an actual DNS lookup.

    >
    > My first thought to solve both of these is that it shouldn't be too
    > hard to hand-craft a minimal DNS query and send it over UDP. Then, I
    > hunted around a bit and found that somebody had already done that, in
    > spades. Take a look at http://www.dnspython.org; it might be exactly
    > what's needed here.


    Yeah, that sounds like a good option. I'm slightly surprised that
    there's no way with the Python stdlib to point a DNS query at a
    specific server, but dnspython might be the solution. On the flip
    side, dnspython is dauntingly large; it looks like a full
    implementation of DNS, but I don't see a simple entrypoint that wraps
    it all up into a simple function that can be bracketed with
    time.time() calls (granted, I only skimmed the docs VERY quickly). So
    it may be simpler to hand-craft an outgoing UDP packet once, save it
    as a string literal, send that, and just wait for any response. That
    eliminates all DNS protocolling and just times the round trip.

    ChrisA
    Chris Angelico, Nov 14, 2012
    #5
  6. Roy Smith

    Roy Smith Guest

    In article <>,
    Chris Angelico <> wrote:

    > I'm slightly surprised that there's no way with the Python stdlib to
    > point a DNS query at a specific server


    Me too, including the "only slightly" part. The normal high-level C
    resolver routines (getaddrinfo/getnameinfo, or even the old
    gethostbyname series), don't expose any way to do that. You have to dig
    quite far down in the resolver library stack to get to the point where
    you can do that. The concept of not knowing or caring which specific
    server has the data you need is quite deeply baked into the basic DNS
    architecture.
    Roy Smith, Nov 15, 2012
    #6
  7. On Thu, Nov 15, 2012 at 12:49 PM, Roy Smith <> wrote:
    > In article <>,
    > Chris Angelico <> wrote:
    >
    >> I'm slightly surprised that there's no way with the Python stdlib to
    >> point a DNS query at a specific server

    >
    > Me too, including the "only slightly" part. The normal high-level C
    > resolver routines (getaddrinfo/getnameinfo, or even the old
    > gethostbyname series), don't expose any way to do that. You have to dig
    > quite far down in the resolver library stack to get to the point where
    > you can do that. The concept of not knowing or caring which specific
    > server has the data you need is quite deeply baked into the basic DNS
    > architecture.


    Indeed. But Python boasts that the batteries are included, and given
    the wealth of other networking facilities that are available, it is a
    bit of a hole that you can't run DNS queries in this way.

    Mind you, if Python's managed to get this far without it being a major
    stumbling-block, that probably means that it's not a serious lack. And
    I don't think many people write DNS *servers* in Python. (Most people
    don't write DNS servers at all, since BIND exists. But I did exactly
    that this week, since it would be easier than most other options.)

    ChrisA
    Chris Angelico, Nov 15, 2012
    #7
  8. Roy Smith

    Roy Smith Guest

    In article <>,
    Chris Angelico <> wrote:

    > Indeed. But Python boasts that the batteries are included, and given
    > the wealth of other networking facilities that are available, it is a
    > bit of a hole that you can't run DNS queries in this way.


    Think of the socket and struct modules as a pile of carbon rods and gobs
    of zinc paste, from which you can assemble your own batteries, and make
    them in exactly the shape and size you need.
    Roy Smith, Nov 15, 2012
    #8
  9. On Thu, Nov 15, 2012 at 1:10 PM, Roy Smith <> wrote:
    > In article <>,
    > Chris Angelico <> wrote:
    >
    >> Indeed. But Python boasts that the batteries are included, and given
    >> the wealth of other networking facilities that are available, it is a
    >> bit of a hole that you can't run DNS queries in this way.

    >
    > Think of the socket and struct modules as a pile of carbon rods and gobs
    > of zinc paste, from which you can assemble your own batteries, and make
    > them in exactly the shape and size you need.


    Then assembly language is a pile of protons, neutrons, and electrons...

    :)

    ChrisA
    Chris Angelico, Nov 15, 2012
    #9
  10. Roy Smith

    Dave Angel Guest

    On 11/14/2012 09:21 PM, Chris Angelico wrote:
    > On Thu, Nov 15, 2012 at 1:10 PM, Roy Smith <> wrote:
    >> In article <>,
    >> Chris Angelico <> wrote:
    >>
    >>> Indeed. But Python boasts that the batteries are included, and given
    >>> the wealth of other networking facilities that are available, it is a
    >>> bit of a hole that you can't run DNS queries in this way.

    >> Think of the socket and struct modules as a pile of carbon rods and gobs
    >> of zinc paste, from which you can assemble your own batteries, and make
    >> them in exactly the shape and size you need.

    > Then assembly language is a pile of protons, neutrons, and electrons...


    And real machine language (microcode) is a pile of quarks; fermions
    versus bosons. But in recent years, you pretty much have to work at
    Intel to see that part of the processor.



    --

    DaveA
    Dave Angel, Nov 15, 2012
    #10
  11. Chris Angelico <> writes:

    > On Thu, Nov 15, 2012 at 12:49 PM, Roy Smith <> wrote:
    >> In article <>,
    >> Chris Angelico <> wrote:
    >>
    >>> I'm slightly surprised that there's no way with the Python stdlib to
    >>> point a DNS query at a specific server

    >>
    >> Me too, including the "only slightly" part. The normal high-level C
    >> resolver routines (getaddrinfo/getnameinfo, or even the old
    >> gethostbyname series), don't expose any way to do that. You have to dig
    >> quite far down in the resolver library stack to get to the point where
    >> you can do that. The concept of not knowing or caring which specific
    >> server has the data you need is quite deeply baked into the basic DNS
    >> architecture.

    >
    > Indeed. But Python boasts that the batteries are included, and given
    > the wealth of other networking facilities that are available, it is a
    > bit of a hole that you can't run DNS queries in this way.
    >
    > Mind you, if Python's managed to get this far without it being a major
    > stumbling-block, that probably means that it's not a serious lack. And
    > I don't think many people write DNS *servers* in Python. (Most people
    > don't write DNS servers at all, since BIND exists. But I did exactly
    > that this week, since it would be easier than most other options.)
    >


    Indeed. Most people would prefer if random applications didn't make
    their own decisions about using specific DNS servers. That way, the
    users can make their own configuration choices (gai.conf, nsswitch.conf)
    according to their site preferences.

    If your application needs that level of control (if you're writing a
    nslookup replacement for some reason, perhaps), dnspython
    (www.dnspython.org) seems to have it.

    --
    regards,
    kushal
    Kushal Kumaran, Nov 15, 2012
    #11
  12. Roy Smith

    Aahz Guest

    DNS from Python (was Re: Subprocess puzzle and two questions)

    In article <>,
    Chris Angelico <> wrote:
    >On Thu, Nov 15, 2012 at 3:20 AM, Roy Smith <> wrote:
    >>
    >> My first thought to solve both of these is that it shouldn't be too
    >> hard to hand-craft a minimal DNS query and send it over UDP. Then, I
    >> hunted around a bit and found that somebody had already done that, in
    >> spades. Take a look at http://www.dnspython.org; it might be exactly
    >> what's needed here.

    >
    >Yeah, that sounds like a good option. I'm slightly surprised that
    >there's no way with the Python stdlib to point a DNS query at a
    >specific server, but dnspython might be the solution. On the flip
    >side, dnspython is dauntingly large; it looks like a full
    >implementation of DNS, but I don't see a simple entrypoint that wraps
    >it all up into a simple function that can be bracketed with
    >time.time() calls (granted, I only skimmed the docs VERY quickly). So
    >it may be simpler to hand-craft an outgoing UDP packet once, save it
    >as a string literal, send that, and just wait for any response. That
    >eliminates all DNS protocolling and just times the round trip.


    From one of my scripts lying around:

    domain = MAILTO.split('@',1)[1]
    server = str(dns.resolver.query(domain, 'MX')[0].exchange)

    You'll need to play around a bit to find out what that does, but it
    should point you in the right direction.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "LL YR VWL R BLNG T S" -- www.nancybuttons.com
    Aahz, Nov 15, 2012
    #12
  13. Roy Smith

    Nobody Guest

    On Wed, 14 Nov 2012 20:49:19 -0500, Roy Smith wrote:

    >> I'm slightly surprised that there's no way with the Python stdlib to
    >> point a DNS query at a specific server

    >
    > Me too, including the "only slightly" part. The normal high-level C
    > resolver routines (getaddrinfo/getnameinfo, or even the old
    > gethostbyname series), don't expose any way to do that.


    That's because the high-level routines aren't tied to DNS.

    gethostbyname() and getaddrinfo() use the NSS (name-service switch)
    mechanism, which is configured via /etc/nsswitch.conf. Depending upon
    configuration, hostnames can be looked up via a plain text file
    (/etc/hosts), Berkeley DB files, DNS, NIS, NIS+, LDAP, WINS, etc. DNS is
    just one particular back-end, which may or may not be used on any given
    system.

    If you specifically want to perform DNS queries, you have to use a
    DNS-specific interface (e.g. the res_* functions described in the
    resolver(3) manpage), or raw sockets, rather than a high-level interface
    such as gethostbyname() or getaddrinfo().
    Nobody, Nov 15, 2012
    #13
  14. Roy Smith

    Roy Smith Guest

    In article <>,
    Nobody <> wrote:

    > That's because the high-level routines aren't tied to DNS.


    This is true.

    >> gethostbyname() and getaddrinfo() use the NSS (name-service switch)

    > mechanism, which is configured via /etc/nsswitch.conf. Depending upon
    > configuration, hostnames can be looked up via a plain text file
    > (/etc/hosts), Berkeley DB files, DNS, NIS, NIS+, LDAP, WINS, etc.


    Gethostbyname() long predates NSS. For that matter, I think it even
    predates DNS (i.e. back to the days when /etc/hosts was the *only* way
    to look up a hostname).

    But, that's a nit.
    Roy Smith, Nov 16, 2012
    #14
  15. Roy Smith

    Nobody Guest

    On Thu, 15 Nov 2012 20:07:38 -0500, Roy Smith wrote:

    >>> gethostbyname() and getaddrinfo() use the NSS (name-service switch)

    >> mechanism, which is configured via /etc/nsswitch.conf. Depending upon
    >> configuration, hostnames can be looked up via a plain text file
    >> (/etc/hosts), Berkeley DB files, DNS, NIS, NIS+, LDAP, WINS, etc.

    >
    > Gethostbyname() long predates NSS.


    Before NSS there was host.conf, which provided similar functionality
    except that the set of mechanisms was fixed (they were built into libc
    rather than being dynamically-loaded libraries) and it only applied to
    hostnames (NSS is also used for getpwent(), getprotoent(), etc).

    > For that matter, I think it even predates DNS (i.e. back to the days
    > when /etc/hosts was the *only* way to look up a hostname).
    >
    > But, that's a nit.


    Indeed; the main point is that gethostbyname() has never been specific to
    DNS.
    Nobody, Nov 17, 2012
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Oliver Wong
    Replies:
    1
    Views:
    3,431
  2. GenxLogic
    Replies:
    3
    Views:
    1,241
    andrewmcdonagh
    Dec 6, 2006
  3. hiral
    Replies:
    2
    Views:
    583
    Jean-Michel Pichavant
    May 5, 2010
  4. Dennis Lee Bieber

    Re: Subprocess puzzle and two questions

    Dennis Lee Bieber, Nov 14, 2012, in forum: Python
    Replies:
    0
    Views:
    180
    Dennis Lee Bieber
    Nov 14, 2012
  5. Kushal Kumaran

    Re: Subprocess puzzle and two questions

    Kushal Kumaran, Nov 14, 2012, in forum: Python
    Replies:
    0
    Views:
    206
    Kushal Kumaran
    Nov 14, 2012
Loading...

Share This Page