threading a thread

[tubby]

I have a program written in Python that checks a class B network (65536
hosts) for web servers. It does a simple TCP socket connect to port 80
and times out after a certain periods of time. The program is threaded
and can do all of the hosts in about 15 minutes or so. I'd like to make
it so that I can check for other open ports (instead of just port 80)
and I'd like to thread the port checks as well.

Right now I'm just prototyping and the threaded hosts portion works very
well for my needs. I'd just like to add a threaded ports check and
wanted to know if anyone had done something similar in Python.

Many thanks

 

[Bjoern Schliessmann]

Right now I'm just prototyping and the threaded hosts portion
works very well for my needs. I'd just like to add a threaded
ports check and wanted to know if anyone had done something
similar in Python.

Taken the vast amount of threads you'll need, there will be a big
overhead. Using a different means of concurrency is advisable.

BTW, why not use nmap?

Regards,


Björn

 

[tubby]

Taken the vast amount of threads you'll need, there will be a big
overhead. Using a different means of concurrency is advisable.

BTW, why not use nmap?

Have you tried it? Nmap is sequential. It takes about 5 hours to do web
server checks (port 80 only) on a class B network... I think it does
ports in parallel, but not hosts. Even when you use the insane time
setting the wait is untimely.

I can do the same thing in roughly 15 minutes with Python or Ruby using
threads. Granted, Nmap is much more configurable and flexible, but it
simply does not scale to address large networks...nessus is the same way.

Also remember that we're dealing with IPv4 networks now. How will we
deal with larger IPv6 address spaces. Besides clustering and distributed
processing (mapreduce), it seems that threads may help deal with some of
the scaling issues I face right now.

 

[Bjoern Schliessmann]

Have you tried it? Nmap is sequential.

RTFM?

| NMAP(1) Nmap Reference Guide NMAP(1)
| [...]
| TIMING AND PERFORMANCE
| [...] While Nmap utilizes parallelism and many advanced
| algorithms to accelerate these scans, the user has ultimate
| control over how Nmap runs.
|
| --min-hostgroup <numhosts>; --max-hostgroup <numhosts>
| (Adjust parallel scan group sizes)
| [...]
| --min-parallelism <numprobes>; --max-parallelism <numprobes>
| (Adjust probe parallelization)
| [...]

I can do the same thing in roughly 15 minutes with Python or Ruby
using threads.

Have fun.

Also remember that we're dealing with IPv4 networks now. How will
we deal with larger IPv6 address spaces. Besides clustering and
distributed processing (mapreduce), it seems that threads may help
deal with some of the scaling issues I face right now.

Please observe that there are simpler and easier (in many cases)
means of parallelisation. For example Unix' select().

Regards,


Björn

 

[tubby]

RTFM?

I urge you to actually try it and see for yourself. From my experience,
it sucks... even when only doing 1 port it takes hours regarless of what
the man page implies.

I'll figure it out, thanks, Tubby.

 

[tubby]

RTFM?

One last things... here's a *very* small sample netstat output from a
threaded py script:

tcp 0 1 192.168.1.100:41066 192.168.17.132:www SYN_SENT
tcp 0 1 192.168.1.100:46412 192.168.5.132:www SYN_SENT
tcp 0 1 192.168.1.100:58297 192.168.83.132:www SYN_SENT
tcp 0 1 192.168.1.100:44011 192.168.157.4:www SYN_SENT
tcp 0 1 192.168.1.100:51936 192.168.243.4:www SYN_SENT
tcp 0 1 192.168.1.100:40812 192.168.39.132:www SYN_SENT
tcp 0 1 192.168.1.100:41903 192.168.155.4:www SYN_SENT
tcp 0 1 192.168.1.100:39110 192.168.35.132:www SYN_SENT
tcp 0 1 192.168.1.100:33060 192.168.59.132:www SYN_SENT
tcp 0 1 192.168.1.100:33060 192.168.59.132:www SYN_SENT
tcp 0 1 192.168.1.100:46544 192.168.15.132:www SYN_SENT
tcp 0 1 192.168.1.100:51863 192.168.119.132:www SYN_SENT
tcp 0 1 192.168.1.100:51666 192.168.117.132:www SYN_SENT
tcp 0 1 192.168.1.100:60085 192.168.85.132:www SYN_SENT
tcp 0 1 192.168.1.100:57431 192.168.195.4:www SYN_SENT
tcp 0 1 192.168.1.100:48253 192.168.31.132:www SYN_SENT
tcp 0 1 192.168.1.100:48253 192.168.31.132:www SYN_SENT
tcp 0 1 192.168.1.100:55402 192.168.251.4:www SYN_SENT
tcp 0 1 192.168.1.100:48510 192.168.159.4:www SYN_SENT
tcp 0 1 192.168.1.100:46516 192.168.23.132:www SYN_SENT
tcp 0 1 192.168.1.100:60412 192.168.73.132:www SYN_SENT
tcp 0 1 192.168.1.100:56050 192.168.127.132:www SYN_SENT
tcp 0 1 192.168.1.100:58080 192.168.199.4:www SYN_SENT
tcp 0 1 192.168.1.100:58080 192.168.199.4:www SYN_SENT
tcp 0 1 192.168.1.100:55805 192.168.253.4:www SYN_SENT
tcp 0 1 192.168.1.100:57871 192.168.69.132:www SYN_SENT
tcp 0 1 192.168.1.100:50699 192.168.225.4:www SYN_SENT
tcp 0 1 192.168.1.100:50245 192.168.227.4:www SYN_SENT
tcp 0 1 192.168.1.100:34634 192.168.161.4:www SYN_SENT
tcp 0 1 192.168.1.100:43256 192.168.97.132:www SYN_SENT
tcp 0 1 192.168.1.100:58740 192.168.173.4:www SYN_SENT
tcp 0 1 192.168.1.100:40337 192.168.211.4:www SYN_SENT
tcp 0 1 192.168.1.100:46584 192.168.15.4:www SYN_SENT
tcp 0 1 192.168.1.100:37866 192.168.175.132:www SYN_SENT

Now, try doing a netstat while you're running a 'parallel' nmap report
and see the difference for yourself. It's huge. Have you ever actually
tried what you recommended???

 

[Facundo Batista]

I have a program written in Python that checks a class B network (65536
hosts) for web servers. It does a simple TCP socket connect to port 80
and times out after a certain periods of time. The program is threaded
and can do all of the hosts in about 15 minutes or so. I'd like to make
it so that I can check for other open ports (instead of just port 80)
and I'd like to thread the port checks as well.

So far, I understand that you have a program with multithreading, but it
only threads the host checking (because it actually scans one port
only).

Right now I'm just prototyping and the threaded hosts portion works very
well for my needs. I'd just like to add a threaded ports check and
wanted to know if anyone had done something similar in Python.

What I do *not* understand if this is a question about:

- port checking (asume not, because the program already checks a port,
so you can actually see how it's done)

- threading (asume not, because the program already is multithreading,
so you can actually see how it's done)

- modifying your program (asume not, you did not copy it here).


So, for us be able to help you, what can not you do?

Regards,