SystemError: com_backpatch: offset too large

[alban.minassian]

(Sorry for my bad english )

I convert a php code to python. i have 10000 lines where i add and add
and add string.... like var += "xxxxxx"

And line 5139 i have SystemError: com_backpatch: offset too large.
All work except if i exceed line 5139 !

I dont find solution. I have only found this post
http://groups.google.com/groups?hl=...kpatch&hl=fr&lr=&client=firefox-a&sa=N&tab=wg

There is a solution?
Do i to give up python and return to php?

Thank for your help
Alban

 

[Steve Holden]

(Sorry for my bad english )

I convert a php code to python. i have 10000 lines where i add and add
and add string.... like var += "xxxxxx"

And line 5139 i have SystemError: com_backpatch: offset too large.
All work except if i exceed line 5139 !

I dont find solution. I have only found this post
http://groups.google.com/groups?hl=...kpatch&hl=fr&lr=&client=firefox-a&sa=N&tab=wg

There is a solution?
Do i to give up python and return to php?

Thank for your help
Alban

It might help if we had some idea what the code looked like around line
5139, and if we knew the size of the data concerned, for example.

regards
Steve

 

[Diez B. Roggisch]

http://groups.google.com/groups?hl=...kpatch&hl=fr&lr=&client=firefox-a&sa=N&tab=wg

There is a solution?
Do i to give up python and return to php?

I can't believe that a program with 10000 lines only adding strings can't be
written in a more concise way, which would most probably solve your
problem. So I suggest you post some parts of your code so that we can have
a look at it and suggest a solution.

Apart from that, it seems that if you insist on doing things the way you do
them ringht now, you'll have to stick with php. Sad to admit that, but it
appears to be that way...

 

[Alex Martelli]

(Sorry for my bad english )

I convert a php code to python. i have 10000 lines where i add and add
and add string.... like var += "xxxxxx"

If 'var' is a string, make it into a list instead, change each
instruction such as the above += into a var.append('xxxxxx'), and when
you're finally done do var=''.join(var) -- by far the best way.


Alex

 

[alban.minassian]

It might help if we had some idea what the code looked like around line

5139, and if we knew the size of the data concerned, for example.

Line 5138 : var += "AAA"+"XXXXXX"+"BBB" # <----- OK
Line 5139 : var += "AAA"+"XXXXXX"+"BBB" # <---- !!! --->
com_backpatch: offset too large

If i replace, line 5139, by 'print 'hello python' : --->
com_backpatch: offset too large

len(var)--> 583


Alban

 

[alban.minassian]

Thank, but same problem (now, i have com_backpatch: offset too large
at line 4567 !)
Alban

 

[alban.minassian]

Python supports a prefix opcode to give 32-bit operand values instead of
16-bit values, but the code generator isn't smart enough to use them.

There is a solution 'force' code generator 32-bit operand values instead of
16-bit values ?

Alban

 

[Jeff Shannon]

Thank, but same problem (now, i have com_backpatch: offset too large
at line 4567 !)

You'll probably get more effective help if you give us more
information. In particular, copy & paste the *entire* traceback that
you're getting. It's probably also a good idea to include at least a
segment of relevant code -- a few lines on either side of the one that's
generating the error, plus any functions that are called there.

Given the message that you posted a link to earlier, it sounds like the
problem *might* simply be that your code branch is too long -- that is,
that you've got too many statements at the same scoping level. To fix
this, find a way to reorganize your code to break it up into smaller
functions.

Where are all of these strings coming from? You can almost certainly
find some way to modify the data source into something that you can
iterate over, in a loop, rather than typing 'var += x + y + z' thousands
of times.

Jeff Shannon
Technician/Programmer
Credit International

 

[Steve Holden]

Line 5138 : var += "AAA"+"XXXXXX"+"BBB" # <----- OK
Line 5139 : var += "AAA"+"XXXXXX"+"BBB" # <---- !!! --->
com_backpatch: offset too large

If i replace, line 5139, by 'print 'hello python' : --->
com_backpatch: offset too large

len(var)--> 583


Alban

Sounds like your module is simply too big. Could you split it up into
several modules instead, calling each in turn?

regards
STeve

 

[Jeff Epler]

I wrote in a long-ago python-list thread:

There is a solution 'force' code generator 32-bit operand values instead of
16-bit values ?

The following patch (against CVS head, but it should apply manually to
any recent version of Python) makes all backwards branches use
EXTENDED_ARG unconditionally. This means that the bytecode will be 3
bytes bigger for each such branch, and the bytecode optimizer will not
run (one of its assumptions is that there is no EXTENDED_ARG. It should
detect this automatically, but I made it exit early unconditionally. It
will now run only for functions without backpatched branches, which I
think excludes any function with 'if', 'for', 'and', 'or', chain
comparison, or anything else that makes for a useful function)

A better approach would probably be to add a variable to 'struct
compiling' to denote whether EXTENDED_ARG branches are being used. It
would be initially set to zero, then if the com_backpatch error code is
hit, the compile is restarted with the flag set and all jumps become
EXTENDED_ARG jumps. Now, the price is paid only by overly complex
code objects, but it's still paid by all branches.

The best approach would be to use either of the strategies above, but
then fix the optimizer to notice when an EXTENDED_ARG is zero and remove
it, shifting all affected branches accordingly, to get the best possible
code.

I did not run the test suite after this change, but it does fail to
error out on the two pieces of code in my older message. (it also
compiles this statement with the value increased from ~16k to ~300k:}

You may use the code below under the terms of the Python license.

Jeff

Index: Python/compile.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Python/compile.c,v
retrieving revision 2.337
diff -u -r2.337 compile.c
--- Python/compile.c 7 Nov 2004 14:04:00 -0000 2.337
+++ Python/compile.c 8 Nov 2004 03:03:11 -0000
@@ -503,6 +503,7 @@
unsigned int *blocks = NULL;
char *name;

+ goto exitUnchanged;
/* Bail out if an exception is set */
if (PyErr_Occurred())
goto exitUnchanged;
@@ -1331,11 +1332,14 @@
/* Compile a forward reference for backpatching */
int here;
int anchor;
- com_addbyte(c, op);
+ com_addbyte(c, EXTENDED_ARG);
here = c->c_nexti;
anchor = *p_anchor;
*p_anchor = here;
- com_addint(c, anchor == 0 ? 0 : here - anchor);
+ anchor = anchor == 0 ? 0 : here - anchor;
+ com_addint(c, anchor >> 16);
+ com_addbyte(c, op);
+ com_addint(c, anchor & 0xffff);
}

static void
@@ -1347,17 +1351,16 @@
int prev;
for (; {
/* Make the JUMP instruction at anchor point to target */
- prev = code[anchor] + (code[anchor+1] << 8);
- dist = target - (anchor+2);
- code[anchor] = dist & 0xff;
+ prev = (code[anchor] << 16)+ (code[anchor+1] << 24)
+ + code[anchor+3] + (code[anchor+4] << 8);
+ dist = target - (anchor+5);
+ code[anchor+3] = dist & 0xff;
dist >>= 8;
- code[anchor+1] = dist;
+ code[anchor+4] = dist & 0xff;
dist >>= 8;
- if (dist) {
- com_error(c, PyExc_SystemError,
- "com_backpatch: offset too large");
- break;
- }
+ code[anchor] = dist & 0xff;
+ dist >>= 8;
+ code[anchor+1] = dist & 0xff;
if (!prev)
break;
anchor -= prev;



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFBjuX8Jd01MZaTXX0RAoRCAJsE0ZJ2tlaQrpQK8RDPh+rtYL0v+ACgjv14
GfSpknOLPBc0oRQ2+83eqpE=
=a7fa
-----END PGP SIGNATURE-----