Segmentation fault, proc, eval, long string

[Pit Capitain]

What is actually going on anyway? Is Ruby creating a closure or
something for each new local variable introduced?

Bob, the whole proc is a closure, so there's a difference between
executing "a = 1" inside or outside of a proc. But I'm not sure whether
it's necessary to evaluate NODE_DASGN_CURR in the recursive way it is
done now. Unfortunately I've not enough time to check this myself. Maybe
someone else can take a look?

Regards,
Pit

 

[Edwin Fine]

I get a segfault too on Ubuntu Edgy x86_64.

efine@ender:/tmp$ uname -a
Linux ender 2.6.17-10-generic #2 SMP Fri Oct 13 15:34:39 UTC 2006 x86_64
GNU/Linux

efine@ender:/tmp$ ruby segfault.rb
the_string length: 186749
Segmentation fault (core dumped)

I don't want to insult anyone's intelligence by stating what could be to
many the obvious, but I'd like to make an observation.

An iterative version of the algorithm might or might not solve the
problem; it all depends how much data has to be stacked on each
iteration (e.g. closures), and where/how it is stacked. If malloc() is
used to create a stack for use with the iterative code, the system will
practically have to run out of VM before the process dies, but using
malloc instead of hardware stack allocation may create performance
problems unless special care is taken, which could be a lot of work.

Is it worth the effort to cater for this special case, given the ulimit
-s workaround? Incidentally, on my system, the default stack size is
only 8MB. That's not a lot these days (given memory capacities these
days).

Just a thought.

 

[Yukihiro Matsumoto]

Hi,

In message "Re: Segmentation fault, proc, eval, long string [Reproduced]"

|There is nothing that is anywhere that deep in the script that I am
|evaluating. So it looks as though the proc object is corrupt??
|
|So maybe this is reproducible?? Well, so it is. If I run this script:

Thank you for the report. Your script helped. Could you check if the
attached patch work for you?

matz.

Index: parse.y
===================================================================
RCS file: /var/cvs/src/ruby/parse.y,v
retrieving revision 1.307.2.47
diff -p -u -1 -r1.307.2.47 parse.y
--- parse.y 2 Nov 2006 06:45:50 -0000 1.307.2.47
+++ parse.y 2 Dec 2006 15:57:23 -0000
@@ -4863,2 +4863,4 @@ gettable(id)

+static VALUE dyna_var_lookup _((ID id));
+
static NODE*
@@ -4891,3 +4893,3 @@ assignable(id, val)
}
- else if (rb_dvar_defined(id)) {
+ else if (dyna_var_lookup(id)) {
return NEW_DASGN(id, val);
@@ -5733,2 +5735,18 @@ top_local_setup()

+static VALUE
+dyna_var_lookup(id)
+ ID id;
+{
+ struct RVarmap *vars = ruby_dyna_vars;
+
+ while (vars) {
+ if (vars->id == id) {
+ vars->val = Qtrue;
+ return Qtrue;
+ }
+ vars = vars->next;
+ }
+ return Qfalse;
+}
+
static struct RVarmap*
@@ -5767,3 +5786,5 @@ dyna_init(node, pre)
for (var = 0; post != pre && post->id; post = post->next) {
- var = NEW_DASGN_CURR(post->id, var);
+ if (RTEST(post->val)) {
+ var = NEW_DASGN_CURR(post->id, var);
+ }
}

 

[Bob Hutchison]

Hi,

Hi,

In message "Re: Segmentation fault, proc, eval, long string
[Reproduced]"
on Fri, 1 Dec 2006 02:50:30 +0900, Bob Hutchison

|There is nothing that is anywhere that deep in the script that I am
|evaluating. So it looks as though the proc object is corrupt??
|
|So maybe this is reproducible?? Well, so it is. If I run this script:

Thank you for the report. Your script helped. Could you check if the
attached patch work for you?

Thanks for the patch. I applied it and tried a few things with it.
The test program I provided does work now, so I think this is solving
the problem. On the other hand, when I try to run the application
that needs it, debian is killing it and I can't see why. The
installation I'm using, as it happens, doesn't have any VM configured
and this has been causing some difficulty recently. Could this patch
cause a lot of memory to be allocated rapidly? If not then there's
something else going on that I'll have to look into (and it is likely
a completely different problem).

Thanks everyone for your help.

Cheers,
Bob

matz.

Index: parse.y
===================================================================
RCS file: /var/cvs/src/ruby/parse.y,v
retrieving revision 1.307.2.47
diff -p -u -1 -r1.307.2.47 parse.y
--- parse.y 2 Nov 2006 06:45:50 -0000 1.307.2.47
+++ parse.y 2 Dec 2006 15:57:23 -0000
@@ -4863,2 +4863,4 @@ gettable(id)

+static VALUE dyna_var_lookup _((ID id));
+
static NODE*
@@ -4891,3 +4893,3 @@ assignable(id, val)
}
- else if (rb_dvar_defined(id)) {
+ else if (dyna_var_lookup(id)) {
return NEW_DASGN(id, val);
@@ -5733,2 +5735,18 @@ top_local_setup()

+static VALUE
+dyna_var_lookup(id)
+ ID id;
+{
+ struct RVarmap *vars = ruby_dyna_vars;
+
+ while (vars) {
+ if (vars->id == id) {
+ vars->val = Qtrue;
+ return Qtrue;
+ }
+ vars = vars->next;
+ }
+ return Qfalse;
+}
+
static struct RVarmap*
@@ -5767,3 +5786,5 @@ dyna_init(node, pre)
for (var = 0; post != pre && post->id; post = post->next) {
- var = NEW_DASGN_CURR(post->id, var);
+ if (RTEST(post->val)) {
+ var = NEW_DASGN_CURR(post->id, var);
+ }
}

----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>

 

[Yukihiro Matsumoto]

Hi,

In message "Re: Segmentation fault, proc, eval, long string [Reproduced]"

|> Thank you for the report. Your script helped. Could you check if the
|> attached patch work for you?
|
|Thanks for the patch. I applied it and tried a few things with it.
|The test program I provided does work now, so I think this is solving
|the problem. On the other hand, when I try to run the application
|that needs it, debian is killing it and I can't see why. The
|installation I'm using, as it happens, doesn't have any VM configured
|and this has been causing some difficulty recently. Could this patch
|cause a lot of memory to be allocated rapidly? If not then there's
|something else going on that I'll have to look into (and it is likely
|a completely different problem).

The patch does rather decrease the amount of memory that Ruby use.
But the original program seems to use tens of thousands of in-block
variables, which themselves consumes more memory than plain local
variables or arrays. So by avoiding segmentation fault, it turns out
to kick the out-of-memory killer of linux kernel.

If it's possible, I'd recommend you to reduce these local variables.

matz.

 

[Bob Hutchison]

Hi,

In message "Re: Segmentation fault, proc, eval, long string
[Reproduced]"
on Sun, 3 Dec 2006 22:51:38 +0900, Bob Hutchison

|> Thank you for the report. Your script helped. Could you check
if the
|> attached patch work for you?
|
|Thanks for the patch. I applied it and tried a few things with it.
|The test program I provided does work now, so I think this is solving
|the problem. On the other hand, when I try to run the application
|that needs it, debian is killing it and I can't see why. The
|installation I'm using, as it happens, doesn't have any VM configured
|and this has been causing some difficulty recently. Could this patch
|cause a lot of memory to be allocated rapidly? If not then there's
|something else going on that I'll have to look into (and it is likely
|a completely different problem).

The patch does rather decrease the amount of memory that Ruby use.
But the original program seems to use tens of thousands of in-block
variables, which themselves consumes more memory than plain local
variables or arrays. So by avoiding segmentation fault, it turns out
to kick the out-of-memory killer of linux kernel.

If it's possible, I'd recommend you to reduce these local variables.

This is consistent with what I'm seeing. Just before you posted your
patch, I made a couple of changes to the application and also
replaced the Proc with a method. The changes to the application would
reduce the maximum number of local variables by typically one third.
The use of a method rather than a Proc avoided the nested parsing and
subsequent stack overflow (and is many times faster for large cases).
Even with that the application barely fits into the available memory.
So if the Proc's local variables cause even a relatively small
proportional increase in memory the application is going to get
itself killed.

Anyway, with the combination of all of this, and a better linux
configuration I think I'll be past this.

Thanks again to everyone for your help.

Cheers,
Bob

matz.

----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>

 

[Yukihiro Matsumoto]

Hi,

In message "Re: Segmentation fault, proc, eval, long string [Reproduced]"

|This is consistent with what I'm seeing. Just before you posted your
|patch, I made a couple of changes to the application and also
|replaced the Proc with a method. The changes to the application would
|reduce the maximum number of local variables by typically one third.
|The use of a method rather than a Proc avoided the nested parsing and
|subsequent stack overflow (and is many times faster for large cases).
|Even with that the application barely fits into the available memory.
|So if the Proc's local variables cause even a relatively small
|proportional increase in memory the application is going to get
|itself killed.
|
|Anyway, with the combination of all of this, and a better linux
|configuration I think I'll be past this.

Good to know that. For your information, YARV does not have this
problem by better in-block variable handling. Another reason to wait
for YARV.

matz.