Too Many if Statements?

[rainbow.cougar]

I tried that code. It runs fine.

However, the following gives a SystemError with only 2500 elif's.

...

I tried this with Python 2.3.3 and 2.3.4 (Linux) and both fail.

Just tried it with 2.4.2 On FreeBSD 6.0 and I get the same result:

Traceback (most recent call last):
File "buggyif.py", line 10, in ?
exec prog
SystemError: com_backpatch: offset too large

Curtis

 

[Paul Rubin]

SystemError: com_backpatch: offset too large

Yeah, that sounds like there's some 16-bit fields in the bytecode format.

 

[nnorwitz]

However, the following gives a SystemError with only 2500 elif's.

SystemErrors should never occur, if you see one it's a bug.

[valid program which demonstrates a python bug]

Traceback (most recent call last):
File "iftest.py", line 10, in ?
exec prog
SystemError: com_backpatch: offset too large

I tried this with Python 2.3.3 and 2.3.4 (Linux) and both fail.

Yup, 2.4 fails too. Unfortunately, this looks like a bugger to fix in
2.4. So I doubt it will be fixed for old versions of python. The good
news is that it's fixed for 2.5.

n

 

[Florian Diesch]

slogging_away wrote:

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.

I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.

Mine was a simple

#!/usr/local/bin/python

zot=24999
if zot == 0:
print "It's 0"

if zot == 1:
print "It's 1"

....

if zot == 24999:
print "It's 24999"

generated (I'm ashamed to admit) by a perl script. Is there any good
reason why it is failing? I'd prefer a "Too many silly walks in your
program. Reduce!" to a crash. I could experiment with putting the
matching 'if' at the beginning rather than at the end, but I'm not
sure what that would tell me.

Here[1] it works with 400000 (with 500000 it starts swapping too much)
"if"-statements generated by

==========================================
#!/usr/bin/env python

print """#!/usr/bin/env python

zot=24999
"""

for i in range(0, 400000):
print """
if zot == %d:
print "It's %d"
"""%(i,i)

============================================


[1] Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
[GCC 4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu8)]



Florian

 

[slogging_away]

The OP did not specify whether all of his if-tests were sequential as in
your test or if some were nested. I vaguely remember there being an indent
limit (40??).

Most of the if statements are nested. Almost all of them fall under a
central 'for xxx in range(x,x,x)', (this is the statement that checks
thorugh each of the saved configuration files). Under that 'for'
statment are the bulk of the 'if' statements - some nested and some not
- some also fall under other 'for' statements. The indent level does
not exceed 10..

I'm pretty sure the OP has hit the python script line limit (32767?).

The script is 4903 lines long.

Slightly off topic; I am just a Network Engineer that can write some
code that accomplishes what I need to get done. I'm learning something
new everyday but I am really blown away by the responses to this
thread. I could not buy support this good. Thanks for your responses.

 

[ajones]

Most of the if statements are nested. Almost all of them fall under a
central 'for xxx in range(x,x,x)', (this is the statement that checks
thorugh each of the saved configuration files). Under that 'for'
statment are the bulk of the 'if' statements - some nested and some not
- some also fall under other 'for' statements. The indent level does
not exceed 10..

Has anyone considered that this may be part of the issue? If he is
stepping through a range this is not just X if statements but n * x
where n is the number of loops. Possibly some variables that are not
getting freed between loops? (my guess would be that it is related to
logging) Anyways, no expert here, just wanted to point that out.

 

[slogging_away]

It appears it may not be a 'if' statment limitation at all. This is
because I added another 800 element array in which to store the various
error messages generated when a configuration file error is deteceted
based on their severity level. The simple addition of the array caused
the same symptom stated in the initial posting. This additional array
was removed and the script performs as expected. Adding it back in
cause it to not run - no error message - just a return to the >>> in
the IDLE console window much as if I had executed the 'Check Module'
command.

At this point I guess I'll find another way work around this issue via
some of the previously suggested methods , etc. It appears to be a bug
as far as I can tell.

 

[Magnus Lycka]

Adding it back in
cause it to not run - no error message - just a return to the >>> in
the IDLE console window much as if I had executed the 'Check Module'
command.

What happens if you run it from the command line instead
of IDLE? After all, it might be some problem in IDLE involved
here. Even if it doesn't work correctly outside IDLE, I was
thinking that IDLE might swallow some kind of error message.

 

[bruno at modulix]

It appears it may not be a 'if' statment limitation at all. This is
because I added another 800 element array

Looks like a memory problem then...

in which to store the various
error messages generated when a configuration file error is deteceted
based on their severity level.

Why storing error messages ? Why don't you just write'em out (be it to
stdout or to a file) ?

(snip)

 

[slogging_away]

Looks like a memory problem then...

The system I am using has 2GB of memory, (unless you are syaing the
memory is faulty).

Why storing error messages ? Why don't you just write'em out (be it to
stdout or to a file) ?

I guess I could do that, (write them to a file as they are discovered).
Right now the error messages are stored in the array and then the the
array is scanned via, a for loop and the error messages are written to
several files in different formats based on the severity of the errors,
(on a per device basis, a per severity basis, etc.). This keeps the
write statements to a minimum and in a central location of the script
instead of having several statements for each individual error message
spread throughout the script, (three write statements per error message
at over 500 error messages would be a significant change).

Not to complain but if I can't use arrays then thats a pretty
significant limitation.

 

[Kent Johnson]

I guess I could do that, (write them to a file as they are discovered).
Right now the error messages are stored in the array and then the the
array is scanned via, a for loop and the error messages are written to
several files in different formats based on the severity of the errors,
(on a per device basis, a per severity basis, etc.). This keeps the
write statements to a minimum and in a central location of the script
instead of having several statements for each individual error message
spread throughout the script, (three write statements per error message
at over 500 error messages would be a significant change).

Sounds like you might like the logging module. A single log entry can be
written to multiple destinations based on level or source of the entry.
The distribution of log entries is controlled by the logging
configuration which can be stored in a text file if you like.

Kent

 

[slogging_away]

What happens if you run it from the command line instead
of IDLE? After all, it might be some problem in IDLE involved
here. Even if it doesn't work correctly outside IDLE, I was
thinking that IDLE might swallow some kind of error message.

Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large

This error is not produced with the IDLE console but is seen only when
executing from the command line. I'll search around and see if I can
determine what this means and a possible fix. Thanks for the
suggestion!

 

[Steven D'Aprano]

Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large

Is this a Python error or a shell error?

If it is a Python error, perhaps you would like to post the entire
traceback?

 

[Steve Holden]

Magnus Lycka wrote:




Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large

This error is not produced with the IDLE console but is seen only when
executing from the command line. I'll search around and see if I can
determine what this means and a possible fix. Thanks for the
suggestion!

Now we have seen the real error message it does appear as though this is
an error in the interpreter triggered by an unusually long source
program. See

http://mail.python.org/pipermail/python-list/2004-November/249270.html

for a previous discussion. In that email Diez Roggisch says:

"""I can't believe that a program with 10000 lines only adding strings
can't be written in a more concise way, which would most probably solve
your problem. So I suggest you post some parts of your code so that we
can have a look at it and suggest a solution."""

The same applies here.

Clearly it would be a good idea to remove whatever problem is causing
the error, but in the meantime is there some way your code could be
written more concisely? Or even split up into multiple modules?

regards
Steve

 

[Juho Schultz]

Is this a Python error or a shell error?

If it is a Python error, perhaps you would like to post the entire
traceback?

I would believe CamelCaseErrorMessages are produced by Python.

The message is exactly the same I reported with the 2500 elifs. I fooled
around a bit with this, and it seems that also for/while blocks
containing more than ~4860 lines give this error.

slogging_away claims his script is about 4900 lines, most of that in a
for loop, so my bet is he has trouble with the same bug.

 

[Terry Reedy]

SystemError: com_backpatch: offset too large

This message is generated in the backpatch function in Python/compile.c in
the source tree. (See below: sorry, tab indents did not survive cut and
paste operation.) As the comment says, and the code shows, it patches in a
2-byte jump offset. The error is raised if the offset is not 0 after
beinging divided by 256 twice (ie, by 65536). This code is called from any
function that compiles a construct that potentially jumps: for, while,
if/elif, and short-circuiting logic expressions.

To me, this message indicates not a bug but a mismatch between code demand
and interpreter capability. The solution is to reduce the demand. In the
present case, changing

for line in file:
<extra-long body>

to

def checkline(line):
<extra-long body>
for line in file:
checkline(line)

might be sufficient. If not, break up the <extra-lone body> into multiple
pieces.

Terry Jan Reedy

static void
com_backpatch(struct compiling *c, int anchor)
{
unsigned char *code = (unsigned char *) PyString_AS_STRING(c->c_code);
int target = c->c_nexti;
int dist;
int prev;
for (; {
/* Make the JUMP instruction at anchor point to target */
prev = code[anchor] + (code[anchor+1] << 8);
dist = target - (anchor+2);
code[anchor] = dist & 0xff;
dist >>= 8;
code[anchor+1] = dist;
dist >>= 8;
if (dist) {
com_error(c, PyExc_SystemError,
"com_backpatch: offset too large");
break;
}
if (!prev)
break;
anchor -= prev;
}
}

 

[Terry Reedy]

Clearly it would be a good idea to remove whatever problem is causing
the error,

The problem (see my post of the com_backpatch code) is writing a compound
statement (here a for loop) with a body so large as to require a jump of
more than 64K bytes in the compiled bytecode (ie, from the test at the top
of the loop to the code that follows after the loop). Until the jump limit
is raised (likely a long wait ;-), the OP must factor some of the code out
of the loop.

Terry Jan Reedy

 

[slogging_away]

Now that I know the root cause of the problem, I can write around it as
suggested by Steve Holden and Terry Reedy, (and others). In fact, it
has helped me in a way as I am thinking not in terms of the easiest
solution, (read; the first one that comes to mind), but more effcient
and cleaner ways to write a section of code to accomplish the same
objective. The key was identifying the root cause which was provided by
the error message seen only at the command line level and by
contibutors to this post.

Thanks again for everyone's suggestions and expertise.

 

[Bengt Richter]

The problem (see my post of the com_backpatch code) is writing a compound
statement (here a for loop) with a body so large as to require a jump of
more than 64K bytes in the compiled bytecode (ie, from the test at the top
of the loop to the code that follows after the loop). Until the jump limit
is raised (likely a long wait ;-), the OP must factor some of the code out
of the loop.

Easy example:
... while True:
... try: co = compile('if x:\n'+ n*' a=1\n','','exec')
... except Exception,e: break
... n += 1
... print 'Stopped at n=%s due to %s: %s'%(n, e.__class__.__name__,e)
...

get an idea of where to start that:
1 0 LOAD_NAME 0 (x)
3 JUMP_IF_FALSE 22 (to 28)
6 POP_TOP

2 7 LOAD_CONST 0 (1)
10 STORE_NAME 1 (a)

3 13 LOAD_CONST 0 (1)
16 STORE_NAME 1 (a)

4 19 LOAD_CONST 0 (1)
22 STORE_NAME 1 (a)
25 JUMP_FORWARD 1 (to 29) 10921

back off 1
Stopped at n=10922 due to SystemError: com_backpatch: offset too large

Decided to test the exact 65536 jump with code chunks of 16 byte-codes and one
chunk at the end to make 16 with the last JUMP_FORWARD.
Traceback (most recent call last):
1 0 LOAD_NAME 0 (x)
3 JUMP_IF_FALSE 65520 (to 65526)
6 POP_TOP

2 7 LOAD_CONST 0 (1)
10 LOAD_CONST 1 (2)
13 BINARY_ADD
14 LOAD_CONST 2 (4)
17 BUILD_TUPLE 2
20 STORE_NAME 1 (a)

3 23 LOAD_CONST 0 (1)

So the corner case of 2**16 is ok. Believe it or not, I once discovered a compiler
error based on optimizing a 2**16-involving loop condition as if it were 0 and the
loop didn't execute! IIRC, I bumped into it processing an array of records with a stride of
exactly 2**n and it thought it could calculate a 16-bit number of strides for end of loop.
No good for arraysize/stridesize==2**16 ;-)

If the OP really HAD to, he could always (untested) break

if cond:
too
large
else:
two
large,also
into
if cond:
too
else:
two
if cond:
large
else:
large,also

but that reads gawdawfully. (So, I imagine, does about any code hitting the offset limit ;-)

If it's a matter of too many elifs, the OP could break that more readably, e.g. (untested)

done=True
if cond:
bla
elif c2:
bla 2
...
elif c456:
bla456
else:
done=False

more, done = not done,True
if more and c457:
bla c457
elif c458:
bla458
...
elif c1012:
bla1012
else:
done = False

more, done = not done,True
... etc

But the OP should find another approach I think,
because this is still disgusting code ;-)

Regards,
Bengt Richter

 

[bruno at modulix]

bruno at modulix wrote:




The system I am using has 2GB of memory, (unless you are syaing the
memory is faulty).

Nope, just that I don't know of any system with unlimited memory. BTW,
having 2GB of ram does not mean there are 2GB available for your program.

(And BTW, this is just one of the *possible* reasons of the problem
you've discovered. May be quite unrelated as well...)

I guess I could do that, (write them to a file as they are discovered).

Well, this is how most programs do.

Right now the error messages are stored in the array and then the the
array is scanned via, a for loop and the error messages are written to
several files in different formats based on the severity of the errors,
(on a per device basis, a per severity basis, etc.). This keeps the
write statements to a minimum and in a central location of the script
instead of having several statements for each individual error message
spread throughout the script, (three write statements per error message
at over 500 error messages would be a significant change).

This is a design problem, not an implementation problem. Just factor out
the part that do the error message dispatch into a function, then call
this function instead of storing the message. Or even better, use an
existing logging library...

Not to complain but if I can't use arrays then thats a pretty
significant limitation.

Please understand that available memory is a limitation of the *system*
- you'd get the same limitation with *every* language ever (well, if you
know of a language that expand free ram ad infinitum, please let us know !-)