Strategy to Verify Python Program is POST'ing to a web server.

[mzagursk]

Hello Folks,

I am wondering what your strategies are for ensuring that data
transmitted to a website via a python program is indeed from that
program, and not from someone submitting POST data using some other
means. I find it likely that there is no solution, in which case what
is the best solution for sending data to a remote server from a python
program and ensuring that it is from that program?

For example, if I create a website that tracks some sort of
statistical information and don't ensure that my program is the one
that is uploading it, the statistics can be thrown off by people
entering false POST data onto the data upload page. Any remedy?

Thanks

 

[Eden Kirin]

Hello Folks,

I am wondering what your strategies are for ensuring that data
transmitted to a website via a python program is indeed from that
program, and not from someone submitting POST data using some other
means. I find it likely that there is no solution, in which case what
is the best solution for sending data to a remote server from a python
program and ensuring that it is from that program?

For example, if I create a website that tracks some sort of
statistical information and don't ensure that my program is the one
that is uploading it, the statistics can be thrown off by people
entering false POST data onto the data upload page. Any remedy?

Include some hash check in hidden field.

For example, from your python program you will include hidden fields
random_number and hash:

import random, hashlib
my_secret_key = "MySecretKey"
random_number = "%f" % random.random()
hash = hashlib.sha1("%s %s" % (my_secret_key, random_number)).hexdigest()

On the server side check hash with random_number and secret key to
ensure the data is POSTed from your application.

 

[Michael Hrivnak]

Authentication by client SSL certificate is best.

You should also look into restricting access on the server side by IP address.

Michael

 

[Chris Angelico]

I am wondering what your strategies are for ensuring that data
transmitted to a website via a python program is indeed from that
program, and not from someone submitting POST data using some other
means.  I find it likely that there is no solution, in which case what
is the best solution for sending data to a remote server from a python
program and ensuring that it is from that program?

You're correct there: there is no solution. Everything on the other
side of your network cable should be treated as hostile and spoofed.
But the real question is, how much effort are people likely to go to
to avoid using your program?

SSL certificates are good, but they can be stolen (very easily if the
client is open source). Anything algorithmic suffers from the same
issue.

In the example you gave, there's no solution. Someone could easily
spoof it and stuff the ballot. But if you make that more difficult
than the survey is worth, then you can largely trust your data.

The other common reason for wanting to be sure that the far end really
is your script is when you're trusting the client to do data
validation. There's a solution to that one: repeat the validation on
the server, and then it doesn't matter if they use your program or
not. (And before you cry "Isn't that obvious?", a lot of people have
completely missed that point.) In neither case can you prove what
program was on the far end. You're working with network packets, so
anything can be spoofed. You could go a long way toward it, though, by
using something ridiculously complex, such as:

* Client connects via SSL to host, using a known certificate.
* Server verifies certificate, and sends client some Python code to execute..
* Client verifies the server's certificate (vital!).
* Client executes the code it's given, and based on the result, plus
some other data, sends the server a hash value.
* Server executes the same code it gave the client, knows the data it
was working with, and calculates the equivalent hash.
* If the two hashes match, the client is deemed to be valid.

This is a variant of the usual nonce-based hashing systems, where the
nonce in question is actually executable code. By randomizing the
code, you can make it difficult for any non-Python program to
duplicate the hash algorithm. But it still won't provide certainty, by
any means.

I've spent quite a bit of time this past fortnight explaining some of
these concepts to my boss and one of my coworkers; they were building
a rather elaborate system but didn't realise that, apart from
requiring about three times as much data from /dev/random, it wasn't
materially different from a simple SSL cert check...

Chris Angelico

 

[Michael Hrivnak]

SSL certificates are good, but they can be stolen (very easily if the
client is open source). Anything algorithmic suffers from the same
issue.

This is only true if you distribute your app with one built-in
certificate, which does indeed seem like a bad idea. When you know
your user base though, especially if this is a situation with a small
number of deployments, than you can distribute a unique certificate to
each client, signed by your CA. Not knowing what kind of statistics
the OP is trying to collect, we really don't know if this client will
be running in one place or thousands.

Even if there will be thousands of deployments, you could generate an
RSA key-pair on the client similar to how an ssh client does, and use
that to sign the data. Then you can at least track which client each
submission came from (storing the public key and IP address), and then
remove submissions as necessary if you detect abuse.

In the example you gave, there's no solution. Someone could easily
spoof it and stuff the ballot. But if you make that more difficult
than the survey is worth, then you can largely trust your data.

You could go a long way toward it, though, by
using something ridiculously complex, such as:

* Client connects via SSL to host, using a known certificate.
* Server verifies certificate, and sends client some Python code to execute.
* Client verifies the server's certificate (vital!).
* Client executes the code it's given, and based on the result, plus
some other data, sends the server a hash value.
* Server executes the same code it gave the client, knows the data it
was working with, and calculates the equivalent hash.
* If the two hashes match, the client is deemed to be valid.

An authentication process that involves the client executing code
supplied by the server opens up one single point of failure (server is
compromised or man-in-the-middle attack is happening) by which
arbitrary code could get executed on the client. Yikes! It's ok to
execute server-supplied code in a sandbox (i.e. javascript), but I
would never want to use software that sends me code over the network
to be executed directly on my system (unless that's the express
purpose of the software, like celery). Besides, it seems that all
you've accomplished is verifying that the client can execute python
code and you've made it a bit less convenient to attack.

The TLS handshake really does verify that the client has a certificate
which has been previously signed by the CA. If you can get signed
certs to each deployment, that is spectacular security that will serve
you well. The above sounds a bit like you're trying to create a new
cipher based on exchanged code that gets executed. I encourage you to
not reinvent the wheel, and stick with the ciphers that are already
standard in the SSL/TLS handshake.

If you cannot uniquely authenticate each client (either through a
signed cert or by having the user supply credentials interactively),
then you'll have to accept that you cannot trust the submitted data
100%, and just take measures to mitigate abuse.

Michael

 

[Paul Rubin]

For example, if I create a website that tracks some sort of
statistical information and don't ensure that my program is the one
that is uploading it, the statistics can be thrown off by people
entering false POST data onto the data upload page. Any remedy?

If you're concerned about unauthorized users posting random crap, the
obvious solution is configure your web server to put password protection
on the page.

If you're saying AUTHORIZED users (those allowed to use the program to
post stuff) aren't trusted to not bypass the program, you've basically
got a DRM problem, especially if you think the users might
reverse-engineer the program to figure out the protocol. The most
effective approaches generally involve delivering the program in the
form of a hardware product that's difficult to tamper with. That's what
cable TV boxes amount to, for example.

What is the application, if you can say? That might help get better
answers.

 

[Terry Reedy]

Hello Folks,

I am wondering what your strategies are for ensuring that data
transmitted to a website via a python program is indeed from that
program, and not from someone submitting POST data using some other
means. I find it likely that there is no solution, in which case what
is the best solution for sending data to a remote server from a python
program and ensuring that it is from that program?

For example, if I create a website that tracks some sort of
statistical information and don't ensure that my program is the one
that is uploading it, the statistics can be thrown off by people
entering false POST data onto the data upload page. Any remedy?

You have not specified all the parameters of the problem. Are there a
limited number of copies of your program or are they distrubuted freely?
What about multiple votes from one program?

Corporate proxy votes (which are a legally important type of statistical
information) work as follows. Each shareholder is mailed or emailed a
'control number'. Attend stockholder meeting in person, mail proxy vote,
or login with any browser with control number. Repeat votes by the same
control id supercede previous vote. There should be a 'thank you for
voting' response for each vote. I suspect IP addr. is recorded with vote
too. I have not heard of specific problems with electronic proxy voting.

 

[Chris Angelico]

This is only true if you distribute your app with one built-in
certificate, which does indeed seem like a bad idea.  When you know
your user base though, especially if this is a situation with a small
number of deployments, than you can distribute a unique certificate to
each client, signed by your CA.

That changes it from verifying the program to verifying the user. It's
a somewhat different beast, but it still leaves the possibility of
snagging the cert and using it in another program. Same with IP
address checks. You can't prove that the other end is a particular
program.

An authentication process that involves the client executing code
supplied by the server opens up one single point of failure (server is
compromised or man-in-the-middle attack is happening) by which
arbitrary code could get executed on the client.  Yikes!

Yeah, hence the part of verifying the server's cert too. That one is a
bit safer though; nobody but you will have that certificate, so it's
not as easy to take and put into another program. But this whole
scheme was meant from the start to be ridiculous.

If ...
then you'll have to accept that you cannot trust the submitted data
100%, and just take measures to mitigate abuse.

I still stand by my original point, namely that the "if" on here is
superfluous, and the "then" is unconditional. But the measures you
describe _do_ reduce the likelihood significantly.

ChrisA

 

[Gregory Ewing]

Besides, it seems that all
you've accomplished is verifying that the client can execute python
code and you've made it a bit less convenient to attack.

And that only if the attacker isn't a Python programmer.
If he is, he's probably writing his attack program in
Python anyway.

Although if you were devious, and you detected that such
an attack was in progress, you could lull him into a sense
of security and then send him some Python code to pwn his
machine...

 

[Chris Angelico]

And that only if the attacker isn't a Python programmer.
If he is, he's probably writing his attack program in
Python anyway.

I was thinking you'd have it call on various functions defined
elsewhere in the program, forcing him to pretty much have the whole
original code in there.

ChrisA

 

[Nobody]

I am wondering what your strategies are for ensuring that data
transmitted to a website via a python program is indeed from that
program, and not from someone submitting POST data using some other
means.

Any remedy?

Supply the client with tamper-proof hardware containing a private key.

Either that, or just accept that it cannot be done. Compare the amount of
effort game developers put into trying to implement tamper-proofing in
software with how little success they've had.

 

[Steven D'Aprano]

Supply the client with tamper-proof hardware containing a private key.

Is that resistant to man-in-the-middle attacks by somebody with a packet
sniffer watching the traffic between the device and the website?

Either that, or just accept that it cannot be done. Compare the amount
of effort game developers put into trying to implement tamper-proofing
in software with how little success they've had.

Exactly.

 

[Paul Rubin]

Is that resistant to man-in-the-middle attacks by somebody with a packet
sniffer watching the traffic between the device and the website?

Sure, why not? As long as the crypto is done properly, that is.

But, there is also the matter of securing the path from the data to the
hardware. I don't have the impression that the OP has really thought
this through.