Rationale for read-only property of co_code

[João Neves]

Hello all,

I've got this question that has been nagging me for a few days now.
What are the reasons for us to have co_code as read-only? I've been
trying to get some info about it, but I kept hitting the wall.

Correct me if I'm wrong, but as far as I understand, co_code
represents the compiled bytecode that should be run when, for
instance, a function is called. Wouldn't it be beneficial for
programmers to be able to change the bytecode in runtime? I mean, one
can't, as far as I'm aware, change the bytecode by accident, so if the
programmer would wish to change a function at runtime, he could do so
at his own risk.

If there is a higher reason behind the read-only property of co_code,
I definitely fail to see it, and would like to know what it is. If
not, why aren't we allowed to write into it?

Thanks in advance,

João Neves

 

[castironpi]

Hello all,

I've got this question that has been nagging me for a few days now.
What are the reasons for us to have co_code as read-only? I've been
trying to get some info about it, but I kept hitting the wall.

Correct me if I'm wrong, but as far as I understand, co_code
represents the compiled bytecode that should be run when, for
instance, a function is called. Wouldn't it be beneficial for
programmers to be able to change the bytecode in runtime? I mean, one
can't, as far as I'm aware, change the bytecode by accident, so if the
programmer would wish to change a function at runtime, he could do so
at his own risk.

If there is a higher reason behind the read-only property of co_code,
I definitely fail to see it, and would like to know what it is. If
not, why aren't we allowed to write into it?

Thanks in advance,

João Neves

Are Python bytes codes Python byte codes? Do you foresee any machine-
dependent optimizations?

 

[João Neves]

Are Python bytes codes Python byte codes?

I'm not quite sure I understood your question, sorry.

Do you foresee any machine-dependent optimizations?

In my personal case, I am not looking for optimizations in the
generated bytecode.

Let me give a very basic example. Say we have these two functions:

def inc(x):
x = x + 1

def dec(x):
x = x - 1

Examining the compiled bytecodes for these two functions:

'|\x00\x00d\x01\x00\x17}\x00\x00d\x00\x00S'

'|\x00\x00d\x01\x00\x18}\x00\x00d\x00\x00S'

Now suppose that I wanted to mess with inc, and have it behave like
dec.

For that, I would like to do something like this, for instance:

Of course, as of this moment, I get a TypeError exception, because
co_code is read-only.

The thing I've been wondering is why _is_ it read-only? In what
circumstances having write access to co_code would break the language
or do some other nasty stuff?

João Neves

 

[Diez B. Roggisch]

I'm not quite sure I understood your question, sorry.

And you won't. He's one of the resident trolls... Better killfile him
ASAP

Diez

 

[Dan Upton]

The thing I've been wondering is why _is_ it read-only? In what

circumstances having write access to co_code would break the language
or do some other nasty stuff?

João Neves

I can't speak to Python's implementation in particular, but
self-modifying code in general is unpleasant. It certainly is
possible to support it in runtime environments, but it's usually not
easy to do so. That is, of course, somewhat dependent on the
implementation of the runtime environment, and even to some degree the
underlying hardware. (For instance, the compiled code you want to run
could be in the hardware cache; if you then change the instructions at
those addresses in memory, it's not always straightforward to get the
processor to realize it needs to load the new data into the
instruction cache.) Plus, I suppose it might be possible to break
strong (even dynamic) typing if you start changing the code around
(although I can't construct an example off the top of my head).

In short, you need a good reason to support self-modifying code, and
my guess is nobody could come up with one for Python.

-dan

 

[castironpi]

I can't speak to Python's implementation in particular, but
self-modifying code in general is unpleasant.  It certainly is
possible to support it in runtime environments, but it's usually not
easy to do so.  That is, of course, somewhat dependent on the
implementation of the runtime environment, and even to some degree the
underlying hardware.  (For instance, the compiled code you want to run
could be in the hardware cache; if you then change the instructions at
those addresses in memory, it's not always straightforward to get the
processor to realize it needs to load the new data into the
instruction cache.)  Plus, I suppose it might be possible to break
strong (even dynamic) typing if you start changing the code around
(although I can't construct an example off the top of my head).

In short, you need a good reason to support self-modifying code, and
my guess is nobody could come up with one for Python.

-dan

Not in the least. You can avoid a lot of if statements, if you have
so many, by enumerating functions to call. "Then call this", i.e.
turn functions on and off. However, there may be data structures
which can accomplish this in constant time too... any takers?

 

[João Neves]

I can't speak to Python's implementation in particular, but
self-modifying code in general is unpleasant. It certainly is
possible to support it in runtime environments, but it's usually not
easy to do so. That is, of course, somewhat dependent on the
implementation of the runtime environment, and even to some degree the
underlying hardware. (For instance, the compiled code you want to run
could be in the hardware cache; if you then change the instructions at
those addresses in memory, it's not always straightforward to get the
processor to realize it needs to load the new data into the
instruction cache.) Plus, I suppose it might be possible to break
strong (even dynamic) typing if you start changing the code around
(although I can't construct an example off the top of my head).

Indeed, the caching issue is a relevant one I guess, and adding the
mechanism to go around that might have a significant impact on
performance.
I haven't looked in detail into it, but I agree with your opinion
concerning strong and dynamic typing. If type check is done while
compiling to bytecode, there is no guarantee the modified bytecode
will respect the rules, for instance. I don't know though, haven't
really checked how it's done at this point.
I will be fiddling around with the Python VM these days anyway, and
since I'm going to muck around the bytecode, I might just try to see
the effects of removing the read-only restriction from co_code.

In short, you need a good reason to support self-modifying code, and
my guess is nobody could come up with one for Python.

-dan

From what I've seen, Python has some support for lambda expressions,
albeit it may be limited. Self-modifying code would greatly enhance
this feature, I guess. Well, at least from a Lisp newbie
perspective.

 

[Chris Mellon]

Indeed, the caching issue is a relevant one I guess, and adding the
mechanism to go around that might have a significant impact on
performance.
I haven't looked in detail into it, but I agree with your opinion
concerning strong and dynamic typing. If type check is done while
compiling to bytecode, there is no guarantee the modified bytecode
will respect the rules, for instance. I don't know though, haven't
really checked how it's done at this point.
I will be fiddling around with the Python VM these days anyway, and
since I'm going to muck around the bytecode, I might just try to see
the effects of removing the read-only restriction from co_code.

There is no need to overwrite co_code. Create a new code object with
your desired bytecode and use that instead.

 

[João Neves]

There is no need to overwrite co_code. Create a new code object with
your desired bytecode and use that instead.

Yes, it may work (haven't tested - isn't there any problem with stuff
like co_name, for instance?), but for simplicity's sake, wouldn't it
be far more convenient if you could just write over co_code?
In the end, it's all a matter of convenience, I guess.

 

[Gabriel Genellina]

Yes, it may work (haven't tested - isn't there any problem with stuff
like co_name, for instance?), but for simplicity's sake, wouldn't it
be far more convenient if you could just write over co_code?
In the end, it's all a matter of convenience, I guess.

Functions aren't just code - they contain the environment needed to
actually call the code. But they're easy to create given a code object:

py> import new
py> new.function
<type 'function'>
py> help(new.function)
Help on class function in module __builtin__:

class function(object)
| function(code, globals[, name[, argdefs[, closure]]])
|
| Create a function object from a code object and a dictionary.
| The optional name string overrides the name from the code object.
| The optional argdefs tuple specifies the default argument values.
| The optional closure tuple supplies the bindings for free variables.

py> import types
py> types.FunctionType is new.function is type(lambda:0)
True

 

[John Nagle]

I can't speak to Python's implementation in particular, but
self-modifying code in general is unpleasant.

Yes. Self-modifying code has a long and painful history.

It's sometimes useful to generate and execute code
dynamically, but that's different than modifying running code.
Python already has "eval", plus the ability to compile and include
new code at run time, so the useful facilities are already working.

John Nagle

 

[João Neves]

Nope: If you change the code in-place, the whole stack's references
to where they were running would need to get updated to corresponding
locations in the new code. _That_ is a lot of work.

Ah, there it is. Now I get it, it makes perfect sense.
Looks like I'll have to stick to the usual mechanisms!
Thanks everyone!

 

[Peter Otten]

Let me give a very basic example. Say we have these two functions:

I suppose you mean
(2, 0)

Examining the compiled bytecodes for these two functions:

'|\x00\x00d\x01\x00\x18}\x00\x00d\x00\x00S'

Now suppose that I wanted to mess with inc, and have it behave like
dec.

(0, 0)

There you are, and there wasn't even a slight chance that you combined the
byte code with an incompatible function signature

Peter

 

[Arnaud Delobelle]

Ah, there it is. Now I get it, it makes perfect sense.
Looks like I'll have to stick to the usual mechanisms!
Thanks everyone!

FWIW, when I need to 'modify' a code object / function object I use
the following functions:

from new import code, function

code_args = (
'argcount', 'nlocals', 'stacksize', 'flags', 'code',
'consts', 'names', 'varnames', 'filename', 'name',
'firstlineno', 'lnotab', 'freevars', 'cellvars'
)

function_args = ('code', 'globals', 'name', 'defaults', 'closure')

def copy_code(code_obj, **kwargs):
"Return a copy of a code object, maybe changing some attributes"
for arg in code_args:
if not kwargs.has_key(arg):
kwargs[arg] = getattr(code_obj, 'co_%s' % arg)
return code(*map(kwargs.__getitem__, code_args))

def copy_function(func_obj, **kwargs):
"Return a copy of a function object, maybe changing some
attributes)"
for arg in function_args:
if not kwargs.has_key(arg):
kwargs[arg] = getattr(func_obj, 'func_%s' % arg)
return function(*map(kwargs.__getitem__, function_args))

# E.g. to change the code object of a function:

f = copy_function(f, code=new_code_object)