optional static typing for Python

R

Russ P.

A while back I came across a tentative proposal from way back in 2000
for optional static typing in Python:

http://www.python.org/~guido/static-typing

Two motivations were given:

-- faster code
-- better compile-time error detection

I'd like to suggest a third, which could help extend Python into the
safety-critical domain:

-- facilitates automated source-code analysis

There has been much heated debate in the past about whether Python is
appropriate for safety-critical software. Some argue that, with
thorough testing, Python code can be as reliable as code in any
language. Well, maybe. But then, a famous computer scientist once
remarked that,

"Program testing can be used to show the presence of bugs, but never
to show their absence!" --Edsger Dijkstra

The next step beyond extensive testing is automated, "static" analysis
of source-code ("static" in the sense of analyzing it without actually
running it). For example, Spark Ada is a subset of Ada with
programming by contract, and in some cases it can formally prove the
correctness of a program by static analysis.

Then there is Java Pathfinder (http://javapathfinder.sourceforge.net),
an "explicit state software model checker." The developers of JPF
wanted
to use it on a prototype safety-critical application that I wrote in
Python, but JPF only works on Java code. We considered somehow using
Jython and Jythonc, but neither did the trick. So they ended up having
someone manually convert my Python code to Java! (The problem is that
my code was still in flux, and the Java and Python versions have now
diverged.)

In any case, optional static typing in Python would help tremendously
here. The hardest part of automated conversion of Python to a
statically typed language is the problem of type inference. If the
types are explicitly declared, that problem obviously goes away.
Explicit typing would also greatly facilitate the development of a
"Python Pathfinder," so the conversion would perhaps not even be
necessary in the first place.

Note also that, while "static" type checking would be ideal,
"explicit" typing would be a major step in the right direction and
would probably be much easier to implement. That is, provide a syntax
to explicitly declare types, then just check them at run time. A
relatively simple pre-processor could be implemented to convert the
explicit type declarations into "isinstance" checks or some such
thing. (A pre-processor command-line argument could be provided to
disable the type checks for more efficient production runs if
desired.)

I noticed that Guido has expressed further interest in static typing
three or four years ago on his blog. Does anyone know the current
status of this project? Thanks.
 
J

Jarek Zgoda

Russ P. pisze:
I noticed that Guido has expressed further interest in static typing
three or four years ago on his blog. Does anyone know the current
status of this project? Thanks.

I thought it was april fools joke?
 
A

André

A while back I came across a tentative proposal from way back in 2000
for optional static typing in Python:

http://www.python.org/~guido/static-typing

Two motivations were given:

-- faster code
-- better compile-time error detection

I'd like to suggest a third, which could help extend Python into the
safety-critical domain:

-- facilitates automated source-code analysis

There has been much heated debate in the past about whether Python is
appropriate for safety-critical software. Some argue that, with
thorough testing, Python code can be as reliable as code in any
language. Well, maybe. But then, a famous computer scientist once
remarked that,

"Program testing can be used to show the presence of bugs, but never
to show their absence!" --Edsger Dijkstra

The next step beyond extensive testing is automated, "static" analysis
of source-code ("static" in the sense of analyzing it without actually
running it). For example, Spark Ada is a subset of Ada with
programming by contract, and in some cases it can formally prove the
correctness of a program by static analysis.

Then there is Java Pathfinder (http://javapathfinder.sourceforge.net),
an "explicit state software model checker." The developers of JPF
wanted
to use it on a prototype safety-critical application that I wrote in
Python, but JPF only works on Java code. We considered somehow using
Jython and Jythonc, but neither did the trick. So they ended up having
someone manually convert my Python code to Java! (The problem is that
my code was still in flux, and the Java and Python versions have now
diverged.)

In any case, optional static typing in Python would help tremendously
here. The hardest part of automated conversion of Python to a
statically typed language is the problem of type inference. If the
types are explicitly declared, that problem obviously goes away.
Explicit typing would also greatly facilitate the development of a
"Python Pathfinder," so the conversion would perhaps not even be
necessary in the first place.

Note also that, while "static" type checking would be ideal,
"explicit" typing would be a major step in the right direction and
would probably be much easier to implement. That is, provide a syntax
to explicitly declare types, then just check them at run time. A
relatively simple pre-processor could be implemented to convert the
explicit type declarations into "isinstance" checks or some such
thing. (A pre-processor command-line argument could be provided to
disable the type checks for more efficient production runs if
desired.)

I noticed that Guido has expressed further interest in static typing
three or four years ago on his blog. Does anyone know the current
status of this project? Thanks.

Perhaps this: http://www.python.org/dev/peps/pep-3107/ might be
relevant?
André
 
A

Arnaud Delobelle

Thanks. If I read this correctly, this PEP is on track for Python 3.0.
Wonderful!

Note that annotations do not provide explicit typing, AFAIK:

def f(x:int) -> int: return x*2

is stricly equivalent to

def f(x): return x*2
f.__annotations__ = {'x':int, 'return':int}

You still need to write a type-checking wrapper. PEP 3107 mentions two
such tools:
* http://oakwinter.com/code/typecheck/
* http://maxrepo.info/taxonomy/term/3,6/all
Neither require annotations.
 
A

ajaksu

I hope Guido doesn't find out about that!

He does know, follow http://mail.python.org/pipermail/python-dev/2007-April/072419.html
and get a laugh too! :)

On a more serious note, I suggest you look up ShedSkin, Pyrex and
Cython, where the static typing is used for better performance. And
with Psyco, you get dynamic typing with JIT specialization. In core
CPython, little is being done AFAIK to support static typing per-se,
but annotations can (and probably will) be used for that end. However,
I don't think it'll interfere in how CPython executes the program.

Daniel
 
C

Christian Heimes

Arnaud said:
Note that annotations do not provide explicit typing, AFAIK:

def f(x:int) -> int: return x*2

is stricly equivalent to

def f(x): return x*2
f.__annotations__ = {'x':int, 'return':int}

Your assumption is correct.

Christian
 
P

Paddy

A while back I came across a tentative proposal from way back in 2000
for optional static typing in Python:

http://www.python.org/~guido/static-typing

Two motivations were given:

-- faster code
-- better compile-time error detection

I'd like to suggest a third, which could help extend Python into the
safety-critical domain:

-- facilitates automated source-code analysis

There has been much heated debate in the past about whether Python is
appropriate for safety-critical software. Some argue that, with
thorough testing, Python code can be as reliable as code in any
language. Well, maybe. But then, a famous computer scientist once
remarked that,

"Program testing can be used to show the presence of bugs, but never
to show their absence!" --Edsger Dijkstra

The next step beyond extensive testing is automated, "static" analysis
of source-code ("static" in the sense of analyzing it without actually
running it). For example, Spark Ada is a subset of Ada with
programming by contract, and in some cases it can formally prove the
correctness of a program by static analysis.

Then there is Java Pathfinder (http://javapathfinder.sourceforge.net),
an "explicit state software model checker." The developers of JPF
wanted
to use it on a prototype safety-critical application that I wrote in
Python, but JPF only works on Java code. We considered somehow using
Jython and Jythonc, but neither did the trick. So they ended up having
someone manually convert my Python code to Java! (The problem is that
my code was still in flux, and the Java and Python versions have now
diverged.)

In any case, optional static typing in Python would help tremendously
here. The hardest part of automated conversion of Python to a
statically typed language is the problem of type inference. If the
types are explicitly declared, that problem obviously goes away.
Explicit typing would also greatly facilitate the development of a
"Python Pathfinder," so the conversion would perhaps not even be
necessary in the first place.

Note also that, while "static" type checking would be ideal,
"explicit" typing would be a major step in the right direction and
would probably be much easier to implement. That is, provide a syntax
to explicitly declare types, then just check them at run time. A
relatively simple pre-processor could be implemented to convert the
explicit type declarations into "isinstance" checks or some such
thing. (A pre-processor command-line argument could be provided to
disable the type checks for more efficient production runs if
desired.)

I noticed that Guido has expressed further interest in static typing
three or four years ago on his blog. Does anyone know the current
status of this project? Thanks.

If static typing is optional then a program written in a dynamic
language that passes such an automated static analysis of source code
would have to be a simple program written in a simplistic way, and
also in a static style.

Having used such formal tools on hardware designs that are expressed
using statically typed languages such as Verilog and VHDL, we don't
have the problem of throwing away run time typing, but we do get other
capacity problems with formal proofs that mean only parts of a design
can be formally prooved, or we can proof that an assertion holds only
as far as the engine has resources to expand the assertion.
We tend to find a lot of bugs in near complete designs by the random
generation of test cases and the automatic checking of results. In
effect, two (or more) programs are created by different people and
usually in different languages. One is written in say Verilog and can
be used to create a chip, another is written by the Verification group
in a 'higher level language' (in terms of chip testing), a tunable
randomized test generator then generates plausible test streams that
the Verification model validates. The test stream is applied to the
Verilog model and the Verilogs responses checked against the
Verification models output and any discrepancy flagged. Coverage
metrics (line coverage , statement coverage, expression coverage ...),
are gathered to indicate how well the design/program is being
exercised.

I would rather advocate such random test generation methods as being
more appropriate for testing software in safety critical systems when
the programming language is dynamic.

- Paddy.
P.S. I think that random generation has also been used to test
compilers by automatically generating correct source for compilation.
 
P

Paul Rubin

Paddy said:
I would rather advocate such random test generation methods as being
more appropriate for testing software in safety critical systems when
the programming language is dynamic.

That method totally failed to find the Pentium FDIV bug, and they use
static proof checkers like ACL2 now.
 
R

Russ P.

On Jan 27, 5:03 pm, Paddy
If static typing is optional then a program written in a dynamic
language that passes such an automated static analysis of source code
would have to be a simple program written in a simplistic way, and
also in a static style.

Yes, but for safety-critical software you usually want the simplest
possible solution. The last think you want is an unnecessarily "fancy"
design. Unless there is a darn good reason to write a "non-static"
program, you just don't do it.

You might want to check into what the FAA allows in "flight-critical"
code, for example. I am certainly not an expert in that area, but I've
had a passing exposure to it. My understanding is that every possible
branch of the code must be fully and meticulously analyzed and
verified. Hence, the dynamic dispatching of ordinary object-oriented
code is either prohibited or severely frowned upon.
Having used such formal tools on hardware designs that are expressed
using statically typed languages such as Verilog and VHDL, we don't
have the problem of throwing away run time typing, but we do get other
capacity problems with formal proofs that mean only parts of a design
can be formally prooved, or we can proof that an assertion holds only
as far as the engine has resources to expand the assertion.
We tend to find a lot of bugs in near complete designs by the random
generation of test cases and the automatic checking of results. In
effect, two (or more) programs are created by different people and
usually in different languages. One is written in say Verilog and can
be used to create a chip, another is written by the Verification group
in a 'higher level language' (in terms of chip testing), a tunable
randomized test generator then generates plausible test streams that
the Verification model validates. The test stream is applied to the
Verilog model and the Verilogs responses checked against the
Verification models output and any discrepancy flagged. Coverage
metrics (line coverage , statement coverage, expression coverage ...),
are gathered to indicate how well the design/program is being
exercised.

I would rather advocate such random test generation methods as being
more appropriate for testing software in safety critical systems when
the programming language is dynamic.

Random test generation methods can go a long way, and they certainly
have their place, but I don't think they are a panacea. Coming up with
a random set of cases that flush out every possible bug is usually
very difficult if not impossible. That was the point of the quote in
my original post.
 
K

Kay Schluehr

Note that annotations do not provide explicit typing, AFAIK:

def f(x:int) -> int: return x*2

is stricly equivalent to

def f(x): return x*2
f.__annotations__ = {'x':int, 'return':int}

You still need to write a type-checking wrapper.

Has anyone figured out how to match arguments passed into a "wrapper"
function defined within a decorator onto the signature of f or onto
f.__annotations__ respectively?
PEP 3107 mentions two
such tools:
*http://oakwinter.com/code/typecheck/
*http://maxrepo.info/taxonomy/term/3,6/all
Neither require annotations.

I like the design of the typecheck package. Everying is implemented in
__init__.py ;)
 
P

Paddy

That method totally failed to find the Pentium FDIV bug, and they use
static proof checkers like ACL2 now.

I would doubt that they would use purely static methods to verify such
large and complex chips. You need a spread of methods, and some
techniques find more bugs at different stages of the design. When
alerted to the existence of a bug then you have to create a test find
it and might well choose to create assertions for formal proof on that
section of the design.

Given the complexity of current microprocessors i'm guessing that
their previous testing methods would be too good to just junk in
totality because the FDIV bug was not found. Similarly if they were
not using formal methods then it makes sense to add it too your
arsenal; and unfortunately it takes a mistake like that to allow
different methods to be explored and incorporated.
It's rarely either/or. More a question of the right, cost-effective,
mix of tools to find bugs, and get the product released on time.

- Paddy.
 
P

Paul Rubin

Paddy said:
Given the complexity of current microprocessors i'm guessing that
their previous testing methods would be too good to just junk in
totality because the FDIV bug was not found. Similarly if they were
not using formal methods then it makes sense to add it too your
arsenal; and unfortunately it takes a mistake like that to allow
different methods to be explored and incorporated.

Fair enough. My main issue was against the notion that random testing
is the only thing necessary.
 
P

Paddy

Fair enough. My main issue was against the notion that random testing
is the only thing necessary.

Sorry Paul if I may have given that impression, its just that when you
bring in random testing to a design that until then had only directed
tests you can see the bug rate jump up! Think of a hysteresis curve
that has gone flat with current testing methods as not many new bugs
are being found; add a new test methodology - random testing and you
get a new hysteresis curve added as bugs found jump up again. We
eventually ship the chip and get awarded by one of our large customers
for quality - which happened - so thats why I put it forward.
- Paddy.
 
P

Paul Rubin

Paddy said:
Sorry Paul if I may have given that impression, its just that when you
bring in random testing to a design that until then had only directed
tests you can see the bug rate jump up!

Sure, I agree with that as well, what I should have said was I have an
issue with the notion that testing (of any sort) is all that is needed
to reach high assurance. Directed and random tests BOTH failed to
catch the FDIV bug. You need methods that demonstrate the absence of
defects, not just fail to demonstrate their presence.
 
B

Bruno Desthuilliers

Russ P. a écrit :
A while back I came across a tentative proposal from way back in 2000
for optional static typing in Python:
(snip)

In any case, optional static typing in Python would help tremendously
here. The hardest part of automated conversion of Python to a
statically typed language is the problem of type inference. If the
types are explicitly declared, that problem obviously goes away.
(snip)

Note also that, while "static" type checking would be ideal,
"explicit" typing would be a major step in the right direction

Lord have mercy(tm).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,754
Messages
2,569,528
Members
45,000
Latest member
MurrayKeync

Latest Threads

Top