If you are unhappy with the direction of Ruby 1.8.7+, respond

[David A. Black]

Hi --

I'm really not up for a slanging match so let me just say that, in code I've
written, Ruby < 1.8.7 has demonstrated the behaviour I described. This may
be a fluke or a bug, in which case I will investigate further and file bugs
if necessary.

It's not a bug; there was never any defined order behavior, so no
behavior is either right or wrong.


David

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2)

http://www.wishsight.com => Independent, social wishlist management!

 

[mdiam]

+1 for the main question,
but...

I still feel like Ruby 1.9 should have been 2.0.
Then 1.8.7 could have been 1.9, 1.8.8 could be 1.10,
and so on, ...

I also agree about this. Perhaps it isn't too late
to rename "1.9.1.0.0.1" to "2.0.0.0.1" ??

Also is it possible to make some "ruby20.rb" package
wich emulate some features of 1.9 present in 1.8.7?

Then in ruby-1.8.8 which would be 1.8.6 like
but would allow us to write:

require 'ruby20'

Then all sugar features that are present in ruby 1.8.7
would be enable!
Perhaps this package could even be required from a 1.8.6
ruby version??

-- Maurice

P.S.
+1 for the original question ;-)

 

[Rick DeNatale]

[Note: parts of this message were removed to make it a legal post.]

On Sat, 14 Feb 2009, James Coglan wrote:
Regardless of whether they're *supposed* to, the Rubies I've used before


I think it probably is machine-dependent, or something like that. I've
never heard of hash order being anything but "not guaranteed" before
1.9 -- except that if you iterate twice, I believe it definitely
happens in the same order both times (though I'm not 100% sure that
that's actually specified or guaranteed either).

No, it's in the absence of key collisions, it's really determined by the
hash values of the keys, The iteration is over the hash buckets and hash
buckets marked as empty are skipped.

Consider your examples which I've excerpted:

% all-ruby -e 'h = {2=>"b", 1=>"a"}; p h'
...
ruby-1.8.6head(2009-02-13p330) {1=>"a", 2=>"b"}
...
ruby-1.8.7head(2009-02-13p116) {1=>"a", 2=>"b"}
...
ruby-1.9.1p0(2009-01-30r21907) {2=>"b", 1=>"a"}

Note that for Ruby 1.8.6 and 1.8.7, the iteration order is by the sort order
of the keys. But that's just a side-effect of the hash of a fixnum being
just the internal immediate representation which happens to be n*2 + 1 for a
Fixnum n:

irb(main):001:0> 1.hash
=> 3
irb(main):002:0> 2.hash
=> 5

Had you used what is arguably the more natural order in the literal, i.e. {1
=> "a", 2 => "b"} it would appear as if iteration was by insertion order,
but that's just an accidental artifact.

I suspect that those who observed that Ruby 1.8.7 'preserves' insertion
order are looking at specific artificial cases like this.

 

[James Gray]

No, it's not a gray area. Anyone who has a basic understanding of
hash-tables doesn't even think about the iteration order.

I guess you haven't used Ruby 1.9.

James Edward Gray II

 

[James Gray]

BTW, Perl had no trouble going from 5.8.x to 5.10.x, and I understand
there is a 5.12.x coming. So I don't see a problem with Ruby doing
similar things in the third digit. 1.8.10 works for me, as does
1.9.10.

=> false

James Edward Gray II

 

[David A. Black]

Hi --

No, it's in the absence of key collisions, it's really determined by the
hash values of the keys, The iteration is over the hash buckets and hash
buckets marked as empty are skipped.

Consider your examples which I've excerpted:

% all-ruby -e 'h = {2=>"b", 1=>"a"}; p h'
...
ruby-1.8.6head(2009-02-13p330) {1=>"a", 2=>"b"}
...
ruby-1.8.7head(2009-02-13p116) {1=>"a", 2=>"b"}
...
ruby-1.9.1p0(2009-01-30r21907) {2=>"b", 1=>"a"}

Note that for Ruby 1.8.6 and 1.8.7, the iteration order is by the sort order
of the keys. But that's just a side-effect of the hash of a fixnum being
just the internal immediate representation which happens to be n*2 + 1 for a
Fixnum n:

irb(main):001:0> 1.hash
=> 3
irb(main):002:0> 2.hash
=> 5

Had you used what is arguably the more natural order in the literal, i.e. {1
=> "a", 2 => "b"} it would appear as if iteration was by insertion order,
but that's just an accidental artifact.

That's why I didn't


David

--
David A. Black / Ruby Power and Light, LLC
Ruby/Rails consulting & training: http://www.rubypal.com
Coming in 2009: The Well-Grounded Rubyist (http://manning.com/black2)

http://www.wishsight.com => Independent, social wishlist management!

 

[Stefan Lang]

I'm really not up for a slanging match so let me just say that, in code I've
written, Ruby < 1.8.7 has demonstrated the behaviour I described. This may
be a fluke or a bug, in which case I will investigate further and file bugs
if necessary.

I can tell you how such a bug report would look like:

Bug: In Ruby 1.8.x Hash iteration order .....

Rejected: Hash iteration order in 1.8 is undefined.

What I do know is that I've had code in a library released
over 2 years ago and used by a fair number of people relying on this
behaviour and heard no complaints until 1.8.7 arrived.

A hidden bug in your code because of an incorrect assumption.

I realise that
hashtables are not ordered in 1.8 and have since made my code more robust.
I'm not questioning your examples -- ...

You seem to be ignoring them.

... you can but experiment to see what
works -- and I will try to extract the code from my project that
demonstrates my point when I have some time.

I'm not the least bit interested in who's right or wrong or in berating the
Ruby code devs or indeed anyone else, and was hoping for a productive
discussion, not people repeatedly asserting that I don't know how a hash
table works and trying to shut me up. When I've time I'll see if what I have
is a Ruby bug or something else. Until then I'm canning this conversation.

I and others here took the trouble finding out and explaining
why your code failed. That's a productive discussion. You'd just
have to accept that it's your code that was buggy. There's
nothing bad about that. Everybody writes buggy code sometimes.

Instead you keep looking for a bug in Ruby that doesn't exist,
as has been clearly proven.

Stefan

 

[M. Edward (Ed) Borasky]

I'm not sure who you are calling enterprisey people here. I know that you
have no intent to disparage, but that term does carry a certain meaning of
"the other guys" with some of the Rubyists, (and others) I run into.

To me "enterprisey" means someone working in a large corporate enterprise,
who is following strategies, architectures and standards, promulgated by
enterprise vendors, like Microsoft, and IBM (to the extent IBM still has the
influence to do this), probably because of edicts handed down from on high
from the executives who play golf with sales reps from those vendors.

I'm pretty sure that such "enterprisey" types make up a very small subset of
the Ruby/Rails community. We don't have many sales reps who play golf with
corporate executives.

I saw a (fairly boring) panel discussion on "The Future of Enterprise
Software" by some of these "enterprisey" folks last night, including
the President of Oracle, who gave the keynote address. Yes, there is a
*big* difference between the 5-million ton gorilla Oracle (did you
know there are five million Oracle developers, all connected by Oracle
software?) and a typical Rails development team.

Very true, I've got no argument with that, but it's important to understand
that those who are using packaged distributions of Ruby from debian
(including Ubuntu), RedHat, macports etc. have more trouble keeping 1.8.6
because normal system administration has a nasty tendency to replace 1.8.6
with 1.8.x where x > 6 because they don't differentiate versions with
differences in teeny version numbers as backwards incompatible. While 1.8.7
is backwards compatible with 1.8.6 in theory. Whether or not it really is,
depends on whether or not all of the ancillary ruby code on the particular
system, like Rails, Gems, and local applications have been kept, or brought
up to date.

It's not just the users ... the *packagers* need to be in the loop
too. There are two main package formats -- Debian/Ubuntu and RPM (Red
Hat / SuSE / Mandriva). And there are three major "community" Linux
distros as well as a few "Corporate" Linux entities. Sure, there are
differences in the details of how a Linux distro is packaged and
administered, but the tasks, issues and use cases are the same: first
of all it has to *work*, security holes need to be fixed ASAP,
preferably *before* exploits appear in the wild, and capacity planning
tools need to exist so that a supplier can be sure his service level
agreements are being met. The distinction here is not one of scale --
Oracle vs. a "typical Rails app". The distinction is between a
production, deployed IT environment and a chaotic, risky experimental
venture.

I think we have to start thinking in terms of weeding out some of the
chaos. For example, I would expect openSUSE, Fedora and Ubuntu to be
all shipping a Ruby with *identical* syntax and semantics. I would
expect them all to be shipping a Rails with *identical* behavior. I
would expect to be able to walk up to a Ubuntu, openSUSE or Fedora
server console and type a *distro* install command to bring in any
Gem, *not* "sudo gem install <gem-name>" because *some* gems are in
the package repository but others aren't. In short, I would expect
that there would be *one* Ruby and *one* Rails, differing only in
whether I said "sudo apt-get install rails", "sudo zypper install
rails", or "sudo yum install rails". It should *just work*.

This requires communication. This requires "marketing". This requires
the kind of discussion we are having in this thread within the Ruby
community, but it also requires communication between the Ruby / Rails
communities and the distros. I haven't done much in terms of syncing
up with the openSUSE community on Ruby, but I did get a sense of the
frustration with Ruby in the Gentoo community when I was a Gentoo
user.

 

[Phlip]

No, it's not a gray area. Anyone who has a basic understanding of

hash-tables doesn't even think about the iteration order.

There's a big difference between abstract computer science theory and pragmatic
technology.

Ruby uses option-hashes as an emergent "named parameters" system. We have
hundreds of DSLs written with them.

After these DSLs can see the order that the user-programmer wrote the
parameters, they are free to use this order to grow more powerful.

 

[William James]

[Note: parts of this message were removed to make it a legal
post.]

NO THEY DO NOT. It is the nature of a regular hashtable that
the order of entries in its internal data structures depends
only on the hash function and its inputs. It doesn't know
anything about the order of entries in your hash literal.

I've shown you examples using Ruby *1.8.6* that clearly
demonstrate that it gives a **** about order of appearance
in a literal, or insertion order.

I'm really not up for a slanging match so let me just say
that, in code I've written, Ruby < 1.8.7 has demonstrated the
behaviour I described. This may be a fluke or a bug, in which
case I will investigate further and file bugs if necessary.
What I do know is that I've had code in a library released
over 2 years ago and used by a fair number of people relying
on this behaviour and heard no complaints until 1.8.7
arrived.

It's bad to write code that works by accident.
It's good to write code that works by design.

I realise that hashtables are not ordered in 1.8

Normal hash-tables are ordered nowhere, no-how, and no-way.

and
have since made my code more robust. I'm not questioning
your examples -- you can but experiment to see what works --
and I will try to extract the code from my project that
demonstrates my point when I have some time.

I'm not the least bit interested in who's right or wrong or
in berating the Ruby code devs or indeed anyone else, and was
hoping for a productive discussion, not people repeatedly
asserting that I don't know how a hash table works

Do you understand how a hash-table works? Have you ever
implemented one? Have you studied hash-table code written
by someone else?

A 9-year-old boy wanted to keep track of the color of his
toys. He wrote "car: red" on a piece of paper which he then
put in a bag. Then he did the same with "airplane: silver"
and "ball: blue". Days later, he decided to review the data.
He reached into the bag and pulled out a slip. It didn't
say "car"! It was the slip with the color of the ball!
The slips were coming out in the wrong order!
Devestated, he began crying and ran to his mommy.

 

[Phlip]

Normal hash-tables are ordered nowhere, no-how, and no-way.

A 9-year-old boy wanted to keep track of the color of his
toys. He wrote "car: red" on a piece of paper which he then
put in a bag. Then he did the same with "airplane: silver"
and "ball: blue". Days later, he decided to review the data.
He reached into the bag and pulled out a slip. It didn't
say "car"! It was the slip with the color of the ball!
The slips were coming out in the wrong order!

There's a big difference between abstract computer science theory and pragmatic
technology.

Ruby uses option-hashes as an emergent "named parameters" system. We have
hundreds of DSLs written with them.

After these DSLs can see the order that the user-programmer wrote the
parameters, they are free to use this order to grow more powerful.

Just because one (technological) implementation of hash tables throws
information away, that does not mean it was _required_ to throw that info away.

How about we give you your own special version, HashWithIndifferentOrder, that
guarantees it randomizes key order?

And along the way, we can give you := for assignment, and = to test equality,
because - you know - the = operator has always meant equality, not assignment.

 

[Trans]

Hi Matz, I'm glad to see that you've entered this fray.

I trust that you know me well enough to know, that while I have strong
opinions, I have no ill-will, particularly to you or your language.

 

[Joel VanderWerf]

[Note: parts of this message were removed to make it a legal
post.]

Regardless of whether they're supposed to, the Rubies
I've used before 1.8.7 actually do iterate in insertion
order.
NO THEY DO NOT. It is the nature of a regular hashtable that
the order of entries in its internal data structures depends
only on the hash function and its inputs. It doesn't know
anything about the order of entries in your hash literal.

I've shown you examples using Ruby *1.8.6* that clearly
demonstrate that it gives a **** about order of appearance
in a literal, or insertion order.

I'm really not up for a slanging match so let me just say
that, in code I've written, Ruby < 1.8.7 has demonstrated the
behaviour I described. This may be a fluke or a bug, in which
case I will investigate further and file bugs if necessary.
What I do know is that I've had code in a library released
over 2 years ago and used by a fair number of people relying
on this behaviour and heard no complaints until 1.8.7
arrived.

It's bad to write code that works by accident.
It's good to write code that works by design.

I realise that hashtables are not ordered in 1.8

Normal hash-tables are ordered nowhere, no-how, and no-way.

and
have since made my code more robust. I'm not questioning
your examples -- you can but experiment to see what works --
and I will try to extract the code from my project that
demonstrates my point when I have some time.

I'm not the least bit interested in who's right or wrong or
in berating the Ruby code devs or indeed anyone else, and was
hoping for a productive discussion, not people repeatedly
asserting that I don't know how a hash table works

Do you understand how a hash-table works? Have you ever
implemented one? Have you studied hash-table code written
by someone else?

A 9-year-old boy wanted to keep track of the color of his
toys. He wrote "car: red" on a piece of paper which he then
put in a bag. Then he did the same with "airplane: silver"
and "ball: blue". Days later, he decided to review the data.
He reached into the bag and pulled out a slip. It didn't
say "car"! It was the slip with the color of the ball!
The slips were coming out in the wrong order!
Devestated, he began crying and ran to his mommy.

Um, how is bag["car"] implemented? How is this analogous with a hash? Do
*you* understand how a hash-table works? (Yes, I know you do.)

 

[Brian Candler]

So we agree to disagree. Though I'd be interested in what you'd define
as "Ruby 1.8". Is it 1.8.6? Why not 1.8.2? Does 1.8.5 qualify as being
a "Ruby 1.8", despite the differences to 1.8.6?

Until now, it didn't matter. Most code written under 1.8.6 ran quite
happily under 1.8.5 or 1.8.2, unless it exercised a few specific bugs or
a few edge cases of feature enhancements.

Example: TCPSocket.new now accepts up to 4 arguments instead of just 2.
However few people use the functionality (for binding to an explicit
local IP and/or port). Those who do will need to run the later version
of Ruby.

IMO, the biggest problem with 1.8.7 is that people are getting it
installed by default (e.g. in newer Debian/Ubuntu systems). They are
starting to write code using a whole heap of new features in 1.8.6,
sometimes by accident. This code will not run on production 1.8.6
platforms. Once library writers start doing this, it messes up the whole
library ecosystem.

 

[David Masover]

Ruby uses option-hashes as an emergent "named parameters" system. We
have hundreds of DSLs written with them.
Granted.

After these DSLs can see the order that the user-programmer wrote the
parameters, they are free to use this order to grow more powerful.

How so?

I honestly can't think of a case where I'd want to do that. If I wanted
positional arguments, I'd use them. If I want named arguments, why do I
care about their position?

Just because one (technological) implementation of hash tables throws
information away, that does not mean it was _required_ to throw that
info away.

No, but it's very likely more efficient to do so, and the vast majority
of hashes simply do not need that information.

How about we give you your own special version,
HashWithIndifferentOrder, that guarantees it randomizes key order?

Because most hashes don't need them?

There are several OrderedHash implementations, some of them quite old,
if you really want to depend on hash ordering.

And along the way, we can give you := for assignment, and = to test
equality, because - you know - the = operator has always meant
equality, not assignment.

Ew. I see your point, but I find I do a lot more assignments than tests
for equality, so I prefer less typing for that.

Also not sure what this has to do with the other discussion...

 

[Robert Dober]

I think 1.8.7 was a mistake. It has taken us too long to get to 1.9
and now we are being forced to deal with a unwanted stop-gap measure,
holding it all up yet again.

It indeed was, though technically fine! It seems that it killed its sibling.
R.

 

[Charles Oliver Nutter]

What about 1.8.6.1?

As much as I dislike the patchlevel concept, it would be very cumbersome
to change it now and have a fourth digit. Rubyspec would have to change,
other implementations would have to change...consider the patchlevel
number as the fourth digit and just continue to increment it.

- Charlie

 

[Charles Oliver Nutter]

Hi,

At Fri, 13 Feb 2009 02:36:06 +0900,
Rick DeNatale wrote in [ruby-talk:327919]:

Although, I'm at a loss as to what you'd use for the version number on the
first maintenance release.

1.8.6-p###.

Agreed, just to make more noise about this. The p-numbers aren't great
(they could have just been a fourth digit) but moving to a fourth digit
now would cause mass confusion.

Which do I install, 1.8.6-p287 or 1.8.6.1?

- Charlie

 

[Charles Oliver Nutter]

Someone suggested having 1.8.8 being 1.8.6 syntax and semantics with
only bug fixes, security fixes and efficiency enhancements. That appeals
to me ... that way, "1.8.7" is a development release and "1.8.8" is
stable. But I would think EngineYard would want to "brand" their ruby
1.8.6 implementation in a manner similar to Ruby Enterprise Edition.

IMHO, Engine Yard should not take over Ruby 1.8.6 to take it in new
directions, they are just becoming 1.8.6 *maintainers*. I agree with
Matz that any large-scale changes like REE or other stuff should still
filter down the 1.8 mainline chain.

And look at it this way...the non-Japanese community has perhaps not had
enough say in the direction of 1.8...we don't want to turn around and do
the same thing. If EY becomes 1.8.6 maintainers, they should not repeat
mistakes and make sure the rest of ruby-core and the Japanese Ruby
community are on board with *all* 1.8.6 revisions.

- Charlie

 

[Pit Capitain]

I'm really not up for a slanging match so let me just say that, in code I've
written, Ruby < 1.8.7 has demonstrated the behaviour I described.

This is just not true. Period. And I can prove it.

This is an excerpt from a previous version of your code:

DICTIONARY = RegexpGroup.new({
:OPERATOR => /return|typeof|[\[(\^=,{}:;&|!*?]/.source,
:CONDITIONAL => /\/\*@\w*|\w*@\*\/|\/\/@\w*|@\w+/.source,
:COMMENT1 => /\/\/[^\n]*/.source,
:COMMENT2 => /\/\*[^*]*\*+([^\/][^*]*\*+)*\//.source,
:REGEXP => /\/(\\[\/\\]|[^*\/])(\\.|[^\/\n\\])*\/[gim]*/.source,
:STRING1 => /\'(\\.|[^\'\\])*\'/.source,
:STRING2 => /"(\\.|[^"\\])*"/.source
})

You say that in 1.8.6 the iteration order of the resulting hash has
been the same as the order in the hash literal, but look at the
following output:

puts DICTIONARY
=> (COMMENT1)|(COMMENT2)|(STRING1)|(STRING2)|(CONDITIONAL)|(OPERATOR)|(REGEXP)

As you can see, the iteration order is completely different from the
order in the hash literal. I really don't know what made you think
that 1.8.6 was using insertion order.

What I do know is that I've had code in a library released
over 2 years ago and used by a fair number of people relying on this
behaviour and heard no complaints until 1.8.7 arrived.

Your code was working by accident. Read along:

I realise that
hashtables are not ordered in 1.8 and have since made my code more robust.

You noticed that the iteration order had changed in 1.8.7, so you
changed your code to:

DICTIONARY = RegexpGroup.new.
putOPERATOR, /return|typeof|[\[(\^=,{}:;&|!*?]/.source).
putCONDITIONAL, /\/\*@\w*|\w*@\*\/|\/\/@\w*|@\w+/.source).
putCOMMENT1, /\/\/[^\n]*/.source).
putCOMMENT2, /\/\*[^*]*\*+([^\/][^*]*\*+)*\//.source).
putREGEXP, /\/(\\[\/\\]|[^*\/])(\\.|[^\/\n\\])*\/[gim]*/.source).
putSTRING1, /\'(\\.|[^\'\\])*\'/.source).
putSTRING2, /"(\\.|[^"\\])*"/.source)

I don't know whether this change made a difference on 1.8.7. It
definitely had no effect on the iteration order, neither in 1.8.7 nor
in 1.8.6:

puts DICTIONARY
=> (COMMENT1)|(COMMENT2)|(STRING1)|(STRING2)|(CONDITIONAL)|(OPERATOR)|(REGEXP)

Now you might say that on your machine and with your version of Ruby
the iteration order had been different and that in your environment
the iteration order still had been the same as the literal order. But
that is not true. Read on:

I will try to extract the code from my project that
demonstrates my point when I have some time.

No need to do that. I fixed some bugs in your code so that the
iteration order is really the same as the literal order:

puts DICTIONARY
=> (OPERATOR)|(CONDITIONAL)|(COMMENT1)|(COMMENT2)|(REGEXP)|(STRING1)|(STRING2)

And guess what? Your code stopped working! Your code only works with
the iteration order of 1.8.6. The iteration order must be different
from insertion order, or else your code doesn't work.

Now please stop talking about 1.8.6 having some well defined property
that has been changed in 1.8.7. This simply is not true.

I'm not the least bit interested in who's right or wrong or in berating the
Ruby code devs or indeed anyone else, and was hoping for a productive
discussion (...)

OK. You said in a previous post that you still had one problem with
regular expressions in 1.8.7. In reality, the bug had nothing to do
with regular expressions, but with ... guess what? ... the iteration
order of hashes. If you carefully build all the hashes in the same
order as those you got with Ruby 1.8.6, then all your tests are
working on both 1.8.6 and 1.8.7.

I don't know git, but if you tell me how, I can send you the two bug
fixes and the working hash sequences. I'd suggest that you take a
careful look at those sequences, whether they are as you want them to
be, because they look really weird.

Regards,
Pit