sorting Array of accentuated Strings

[Une Bévue]

I've done a "self <=> anotherString" comparaison which works by itself :

class String
def <=>( aString )
[blahblahblah]
end
end

however if i want to sort an Array of Strings, the Array is sorted as
usual within Ruby (put accentuated characters at the end) :

a = [ "Être", "Fenêtre", "Etre" ]

b = a.sort { | i, j | i <=> j }
puts "b = [ " + b.join(", ") + " ]\n"
# => b = [ Etre, Être, Fenêtre ]

c = a.sort #<=>( aString ) NOT CALLED...
puts "c = [ " + c.join(", ") + " ]\n"
# => c = [ Etre, Fenêtre, Être ]

Why, in the case of "c = a.sort", #<=>( aString ) isn't called ?
the comparaison within an Array could compare different kind of objects
like :

a = [ 0, "a", 9, Time.new ] ???

 

[cruiserdan]

I've done a "self <=> anotherString" comparaison which works by itself :

class String
def <=>( aString )
[blahblahblah]
end
end

however if i want to sort an Array of Strings, the Array is sorted as
usual within Ruby (put accentuated characters at the end) :

a = [ "Être", "Fenêtre", "Etre" ]

b = a.sort { | i, j | i <=> j }
puts "b = [ " + b.join(", ") + " ]\n"
# => b = [ Etre, Être, Fenêtre ]

c = a.sort #<=>( aString ) NOT CALLED...
puts "c = [ " + c.join(", ") + " ]\n"
# => c = [ Etre, Fenêtre, Être ]

Why, in the case of "c = a.sort", #<=>( aString ) isn't called ?
the comparaison within an Array could compare different kind of objects
like :

a = [ 0, "a", 9, Time.new ] ???

I believe that the C implementation Array#sort checks for a String
argument and calls the C string comparison operator directly in that
case. See:

http://svn.ruby-lang.org/cgi-bin/viewvc.cgi/trunk/array.c?view=markup

under the code for sort_2.

You might be interested in using some of the emerging Unicode support
in Ruby. Ruby 2 will have it built-in and there are several libraries
out there, although I don't have any experience using them.

Regards,
Dan Yoder
http://dev.zeraweb.com/

 

[Une Bévue]

You might be interested in using some of the emerging Unicode support
in Ruby. Ruby 2 will have it built-in and there are several libraries
out there, although I don't have any experience using them.

right, thanks, i've only wrote a workaround before getting Ruby 2...

 

[Rishabh Shrivastava]

Une Bév

right, thanks, i've only wrote a workaround before getting Ruby 2...

YEAH, aCTUALLY I WANT TO CREATE A FUNCTION FOR vpn CONNECTION FOR RUBY
IN WATIR,
pLEASE PROVIDE IF IT IS POSSIBLE.

 

[Une Bévue]

You might be interested in using some of the emerging Unicode support
in Ruby.

Unfortunately i can't get :
<ftp://ftp.mars.org/pub/ruby/Unicode.tar.bz2>
may be the server is down ???

 

[MonkeeSage]

right, thanks, i've only wrote a workaround before getting Ruby 2...

Hmmm. Maybe I'm mistaken, but this seems to have nothing to do with
unicode. An ascii char is always going to be less than a utf-8 char,
since utf-8 is a superset of ascii.

Fenêtre <=> Être ->

F (\x46) <=> Ê (\xc3\x8a) ->

-1

To get the right behavior I think you have to translate the utf-8
characters to ascii. You can try something like:

require 'iconv'
class String
def translit
Iconv.iconv('ascii//translit', 'utf-8', self)[0]
end
end
a.sort { | i, j | i.translit <=> j.translit }

But some people have had strange effects from #iconv (e.g., a recent
thread [1]).

Regards,
Jordan

[1] http://groups.google.com/group/comp...5fa49dd700f/26311c1a3844267d#26311c1a3844267d

 

[Axel Etzold]

-------- Original-Nachricht --------

Datum: Sat, 8 Dec 2007 22:15:00 +0900
Von: MonkeeSage <[email protected]>
An: (e-mail address removed)
Betreff: Re: sorting Array of accentuated Strings

right, thanks, i've only wrote a workaround before getting Ruby 2...

Hmmm. Maybe I'm mistaken, but this seems to have nothing to do with
unicode. An ascii char is always going to be less than a utf-8 char,
since utf-8 is a superset of ascii.

Fenêtre <=> Être ->

F (\x46) <=> Ê (\xc3\x8a) ->

-1

To get the right behavior I think you have to translate the utf-8
characters to ascii. You can try something like:

require 'iconv'
class String
def translit
Iconv.iconv('ascii//translit', 'utf-8', self)[0]
end
end
a.sort { | i, j | i.translit <=> j.translit }

But some people have had strange effects from #iconv (e.g., a recent
thread [1]).

Regards,
Jordan

Besides that, the problem of sorting accented strings seems to be
somewhat unsolvable, as different natural languages using the
same accents have different conventions.
I'd claim the highest degree of inconsistency in this issue
for the German language (other proposals invited):

- German phone books sort words containing <A-DIAERESIS>,<O-DIAERESIS>,
<U-DIAERESIS>, as if they were spelled with "AE","OE","UE" instead of <A-DIAERESIS> etc.,
- otherwise, the diacritics are quite often just ignored,
- in Austria, including in phone books, diacritics come behind "z" .... (just like in Swedish, where <A-DIAERESIS>,<O-DIAERESIS> are also used (but consistently),
- French and Spanish use diaeresis on some letters to mark that
they have to be pronounced separately (Citro{"e}n,Camag{"u}ey).

(see: http://en.wikipedia.org/wiki/Collation)

How can one establish a single standard, for all (natural) languages
with such a confusion ?

I'd recommend to use a couple of gsub calls, much like Xavier Noria
proposed in his post

http://groups.google.de/group/comp.lang.ruby/browse_thread/thread/9fbb85fa49dd700f/eed0350375a53abe

and to adapt them to the situation at hand to pre-process the strings
to sort.

Best regards,

Axel

 

[MonkeeSage]

-------- Original-Nachricht --------

Datum: Sat, 8 Dec 2007 22:15:00 +0900
Von: MonkeeSage <[email protected]>
An: (e-mail address removed)
Betreff: Re: sorting Array of accentuated Strings

Hmmm. Maybe I'm mistaken, but this seems to have nothing to do with
unicode. An ascii char is always going to be less than a utf-8 char,
since utf-8 is a superset of ascii.

Fenêtre <=> Être ->

F (\x46) <=> Ê (\xc3\x8a) ->

To get the right behavior I think you have to translate the utf-8
characters to ascii. You can try something like:

require 'iconv'
class String
def translit
Iconv.iconv('ascii//translit', 'utf-8', self)[0]
end
end
a.sort { | i, j | i.translit <=> j.translit }

But some people have had strange effects from #iconv (e.g., a recent
thread [1]).

Regards,
Jordan

Besides that, the problem of sorting accented strings seems to be
somewhat unsolvable, as different natural languages using the
same accents have different conventions.
I'd claim the highest degree of inconsistency in this issue
for the German language (other proposals invited):

- German phone books sort words containing <A-DIAERESIS>,<O-DIAERESIS>,
<U-DIAERESIS>, as if they were spelled with "AE","OE","UE" instead of <A-DIAERESIS> etc.,
- otherwise, the diacritics are quite often just ignored,
- in Austria, including in phone books, diacritics come behind "z" .... (just like in Swedish, where <A-DIAERESIS>,<O-DIAERESIS> are also used (but consistently),
- French and Spanish use diaeresis on some letters to mark that
they have to be pronounced separately (Citro{"e}n,Camag{"u}ey).

(see:http://en.wikipedia.org/wiki/Collation)

How can one establish a single standard, for all (natural) languages
with such a confusion ?

Just to emphasize the point...Greek η (eta) can be transliterated as
e, Ä“ (yet another level of indirection!), h or i.

I'd recommend to use a couple of gsub calls, much like Xavier Noria
proposed in his post

http://groups.google.de/group/comp.lang.ruby/browse_thread/thread/9fb...

and to adapt them to the situation at hand to pre-process the strings
to sort.

Best regards,

Axel

Regards,
Jordan

 

[Une Bévue]

An ascii char is always going to be less than a utf-8 char,
since utf-8 is a superset of ascii.

Fenêtre <=> Être ->

F (\x46) <=> Ê (\xc3\x8a) ->

-1

yes, for sure, but, in order to compare F to Ê i've to know F uses only
one byte and Ê two butes, in other words : decompose a string into an
arrau of characters and compare afterwards...

obviously for the question given by Axel, that's to say ordering between
:

èéêë

various diaresis, i leave the ordering as it is in the unicode number
(UTF-8 in my case)

quiet frankly i don't know what is the french policy for that, i just
want having all e + diaresis between e and f...