threading a thread

Discussion in 'Python' started by tubby, Feb 27, 2007.

  1. tubby

    tubby Guest

    I have a program written in Python that checks a class B network (65536
    hosts) for web servers. It does a simple TCP socket connect to port 80
    and times out after a certain periods of time. The program is threaded
    and can do all of the hosts in about 15 minutes or so. I'd like to make
    it so that I can check for other open ports (instead of just port 80)
    and I'd like to thread the port checks as well.

    Right now I'm just prototyping and the threaded hosts portion works very
    well for my needs. I'd just like to add a threaded ports check and
    wanted to know if anyone had done something similar in Python.

    Many thanks
     
    tubby, Feb 27, 2007
    #1
    1. Advertising

  2. tubby wrote:
    > Right now I'm just prototyping and the threaded hosts portion
    > works very well for my needs. I'd just like to add a threaded
    > ports check and wanted to know if anyone had done something
    > similar in Python.


    Taken the vast amount of threads you'll need, there will be a big
    overhead. Using a different means of concurrency is advisable.

    BTW, why not use nmap?

    Regards,


    Björn

    --
    BOFH excuse #394:

    Jupiter is aligned with Mars.
     
    Bjoern Schliessmann, Feb 27, 2007
    #2
    1. Advertising

  3. tubby

    tubby Guest

    Bjoern Schliessmann wrote:
    > tubby wrote:
    >> Right now I'm just prototyping and the threaded hosts portion
    >> works very well for my needs. I'd just like to add a threaded
    >> ports check and wanted to know if anyone had done something
    >> similar in Python.

    >
    > Taken the vast amount of threads you'll need, there will be a big
    > overhead. Using a different means of concurrency is advisable.
    >
    > BTW, why not use nmap?


    Have you tried it? Nmap is sequential. It takes about 5 hours to do web
    server checks (port 80 only) on a class B network... I think it does
    ports in parallel, but not hosts. Even when you use the insane time
    setting the wait is untimely.

    I can do the same thing in roughly 15 minutes with Python or Ruby using
    threads. Granted, Nmap is much more configurable and flexible, but it
    simply does not scale to address large networks...nessus is the same way.

    Also remember that we're dealing with IPv4 networks now. How will we
    deal with larger IPv6 address spaces. Besides clustering and distributed
    processing (mapreduce), it seems that threads may help deal with some of
    the scaling issues I face right now.
     
    tubby, Feb 27, 2007
    #3
  4. tubby wrote:

    > Have you tried it? Nmap is sequential.


    RTFM?

    | NMAP(1) Nmap Reference Guide NMAP(1)
    | [...]
    | TIMING AND PERFORMANCE
    | [...] While Nmap utilizes parallelism and many advanced
    | algorithms to accelerate these scans, the user has ultimate
    | control over how Nmap runs.
    |
    | --min-hostgroup <numhosts>; --max-hostgroup <numhosts>
    | (Adjust parallel scan group sizes)
    | [...]
    | --min-parallelism <numprobes>; --max-parallelism <numprobes>
    | (Adjust probe parallelization)
    | [...]

    > I can do the same thing in roughly 15 minutes with Python or Ruby
    > using threads.


    Have fun.

    > Also remember that we're dealing with IPv4 networks now. How will
    > we deal with larger IPv6 address spaces. Besides clustering and
    > distributed processing (mapreduce), it seems that threads may help
    > deal with some of the scaling issues I face right now.


    Please observe that there are simpler and easier (in many cases)
    means of parallelisation. For example Unix' select().

    Regards,


    Björn

    --
    BOFH excuse #368:

    Failure to adjust for daylight savings time.
     
    Bjoern Schliessmann, Feb 28, 2007
    #4
  5. tubby

    tubby Guest

    Bjoern Schliessmann wrote:
    > tubby wrote:
    >
    >> Have you tried it? Nmap is sequential.

    >
    > RTFM?


    I urge you to actually try it and see for yourself. From my experience,
    it sucks... even when only doing 1 port it takes hours regarless of what
    the man page implies.

    I'll figure it out, thanks, Tubby.
     
    tubby, Feb 28, 2007
    #5
  6. tubby

    tubby Guest

    Bjoern Schliessmann wrote:

    > RTFM?


    One last things... here's a *very* small sample netstat output from a
    threaded py script:

    > tcp 0 1 192.168.1.100:41066 192.168.17.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:46412 192.168.5.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:58297 192.168.83.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:44011 192.168.157.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:51936 192.168.243.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:40812 192.168.39.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:41903 192.168.155.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:39110 192.168.35.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:33060 192.168.59.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:33060 192.168.59.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:46544 192.168.15.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:51863 192.168.119.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:51666 192.168.117.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:60085 192.168.85.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:57431 192.168.195.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:48253 192.168.31.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:48253 192.168.31.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:55402 192.168.251.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:48510 192.168.159.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:46516 192.168.23.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:60412 192.168.73.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:56050 192.168.127.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:58080 192.168.199.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:58080 192.168.199.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:55805 192.168.253.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:57871 192.168.69.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:50699 192.168.225.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:50245 192.168.227.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:34634 192.168.161.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:43256 192.168.97.132:www SYN_SENT
    > tcp 0 1 192.168.1.100:58740 192.168.173.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:40337 192.168.211.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:46584 192.168.15.4:www SYN_SENT
    > tcp 0 1 192.168.1.100:37866 192.168.175.132:www SYN_SENT


    Now, try doing a netstat while you're running a 'parallel' nmap report
    and see the difference for yourself. It's huge. Have you ever actually
    tried what you recommended???
     
    tubby, Feb 28, 2007
    #6
  7. tubby wrote:

    > I have a program written in Python that checks a class B network (65536
    > hosts) for web servers. It does a simple TCP socket connect to port 80
    > and times out after a certain periods of time. The program is threaded
    > and can do all of the hosts in about 15 minutes or so. I'd like to make
    > it so that I can check for other open ports (instead of just port 80)
    > and I'd like to thread the port checks as well.


    So far, I understand that you have a program with multithreading, but it
    only threads the host checking (because it actually scans one port
    only).


    > Right now I'm just prototyping and the threaded hosts portion works very
    > well for my needs. I'd just like to add a threaded ports check and
    > wanted to know if anyone had done something similar in Python.


    What I do *not* understand if this is a question about:

    - port checking (asume not, because the program already checks a port,
    so you can actually see how it's done)

    - threading (asume not, because the program already is multithreading,
    so you can actually see how it's done)

    - modifying your program (asume not, you did not copy it here).


    So, for us be able to help you, what can not you do?

    Regards,

    --
    .. Facundo
    ..
    Blog: http://www.taniquetil.com.ar/plog/
    PyAr: http://www.python.org/ar/
     
    Facundo Batista, Mar 5, 2007
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Karl
    Replies:
    0
    Views:
    3,687
  2. Jerry Sievers

    thread, threading; how to kill a thread?

    Jerry Sievers, Nov 17, 2004, in forum: Python
    Replies:
    12
    Views:
    1,158
    Mustafa Demirhan
    Nov 19, 2004
  3. Replies:
    9
    Views:
    1,055
    Mark Space
    Dec 29, 2007
  4. Steven Woody
    Replies:
    0
    Views:
    431
    Steven Woody
    Jan 9, 2009
  5. Steven Woody
    Replies:
    0
    Views:
    461
    Steven Woody
    Jan 9, 2009
Loading...

Share This Page