download limit

[Mathieu Prevot]

Hi,

I have a multithreaded script that mainly creates several wget
processes to download files. I would like to check/see and eventually
limit the bandwidth of the pool of processes. One way to do this is to
change the number of wget instances, but it's a workaround.

What do you recommend to do the following in python:
1) know the bitrate at the script scale
2) control, and limit or not this bitrate

Thanks,
Mathieu

 

[Martin v. Löwis]

I have a multithreaded script that mainly creates several wget

processes to download files. I would like to check/see and eventually
limit the bandwidth of the pool of processes. One way to do this is to
change the number of wget instances, but it's a workaround.

What do you recommend to do the following in python:
1) know the bitrate at the script scale
2) control, and limit or not this bitrate

I recommend to not use wget, but implement the network access directly
in Python. Then you can easily measure that bitrate, and also limit it.
(by having some threads sleep).

Once you fork out new processes, you lose, unless there is an operating
system bandwidth limitation framework available which works on groups
of processes. Solaris project objects may provide such a thing, but
apart from that, I don't think it's available in any operating system
that you might be using.

Regards,
Martin

 

[Martin v. Löwis]

I have a multithreaded script that mainly creates several wget

processes to download files. I would like to check/see and eventually
limit the bandwidth of the pool of processes. One way to do this is to
change the number of wget instances, but it's a workaround.

What do you recommend to do the following in python:
1) know the bitrate at the script scale
2) control, and limit or not this bitrate

I recommend to not use wget, but implement the network access directly
in Python. Then you can easily measure that bitrate, and also limit it.
(by having some threads sleep).

Once you fork out new processes, you lose, unless there is an operating
system bandwidth limitation framework available which works on groups
of processes. Solaris project objects may provide such a thing, but
apart from that, I don't think it's available in any operating system
that you might be using.

Regards,
Martin

 

[Diez B. Roggisch]

Hi,

I have a multithreaded script that mainly creates several wget
processes to download files. I would like to check/see and eventually
limit the bandwidth of the pool of processes. One way to do this is to
change the number of wget instances, but it's a workaround.

What do you recommend to do the following in python:
1) know the bitrate at the script scale
2) control, and limit or not this bitrate

I'm not aware that python can do that. But under Linux you might
consider using trickled.

Diez