Ruby 1.9.1 - invalid multibyte escape: /[\xC0-\xDF]/ (RegexpError)

[Iñaki Baz Castillo]

Hi, using TreeTop parser I had a grammar defined working in Ruby1.8 but it=
=20
fails in 1.9.1:

~# ruby1.8 -e "Regexp.new('[\xC0-\xDF]')"
OK

~# ruby1.9 -e "Regexp.new('[\xC0-\xDF]')"
-e:1:in `initialize': invalid multibyte escape: /[\xC0-\xDF]/ (RegexpErro=
r)

I've found the following text about differences between 1.8 and 1.9:

"It is more rigorous that 1.8 when it comes to detecting invalid code. For=
=20
example, 1.8 accepts /[^\x00-\xa0]/u, while 1.9 complains of invalid multib=
yte=20
escape"

Ok, so how should I write the above Regexp to work on 1.9.1?

Thanks a lot.


=2D-=20


I=C3=B1aki Baz Castillo <[email protected]>

 

[Eric Hodel]

Hi, using TreeTop parser I had a grammar defined working in Ruby1.8 =20=

but it
fails in 1.9.1:

~# ruby1.8 -e "Regexp.new('[\xC0-\xDF]')"
OK

~# ruby1.9 -e "Regexp.new('[\xC0-\xDF]')"
-e:1:in `initialize': invalid multibyte escape: /[\xC0-\xDF]/ =20
(RegexpError)

I've found the following text about differences between 1.8 and 1.9:

"It is more rigorous that 1.8 when it comes to detecting invalid =20
code. For
example, 1.8 accepts /[^\x00-\xa0]/u, while 1.9 complains of invalid =20=

multibyte
escape"

Ok, so how should I write the above Regexp to work on 1.9.1?

Regexp.new '[\xC0-\xDF]', nil, 'n'

 

[Iñaki Baz Castillo]

Ok, so how should I write the above Regexp to work on 1.9.1?

Regexp.new '[\xC0-\xDF]', nil, 'n'

Great! Thanks a lot.


--=20
I=C3=B1aki Baz Castillo
<[email protected]>

 

[Iñaki Baz Castillo]

Ok, so how should I write the above Regexp to work on 1.9.1?

Regexp.new '[\xC0-\xDF]', nil, 'n'

Great! Thanks a lot.

However I don't understant these parameters for Regexp.new.
I read:
http://www.ruby-doc.org/core-1.9/classes/Regexp.html
About the third parameter you use ('n') it doens't appear on the doc =C2=BF=
?

Thanks a lot.


--=20
I=C3=B1aki Baz Castillo
<[email protected]>

 

[Brian Candler]

However I don't understant these parameters for Regexp.new.
I read:
http://www.ruby-doc.org/core-1.9/classes/Regexp.html
About the third parameter you use ('n') it doens't appear on the doc ¿?

All the new stuff to do with String and encodings in ruby 1.9 is
undocumented.

(At least, it's not documented within Ruby itself. You may be able to
purchase a book which has some reverse-engineered documentation)

If you care about stability or documentation, my own advice is to stick
with 1.8 - preferably 1.8.6.

 

[James Gray]

All the new stuff to do with String and encodings in ruby 1.9 is
undocumented.

I've got the majority of the new functionality covered in my m17n =20
series now:

http://blog.grayproductions.net/articles/understanding_m17n

I expect to have the minor side topics I'm still missing covered in =20
the next few weeks.

James Edward Gray II

 

[Jeremy McAnally]

I'm working on documenting some of this stuff when I have time (always
the magic words, eh? :-/). I ran dcov on the whole of Ruby core last
week (results: http://jeremymcanally.com/coverage.html ; it's a little
deceiving since methods like to_yaml I think are actually included
from elsewhere. I'll have to look...), and I'm currently setting up
some tasks for myself to knock things out.

I might setup a Lighthouse for it or something if other people want to
get involved.

--Jeremy

?

All the new stuff to do with String and encodings in ruby 1.9 is
undocumented.

(At least, it's not documented within Ruby itself. You may be able to
purchase a book which has some reverse-engineered documentation)

If you care about stability or documentation, my own advice is to stick
with 1.8 - preferably 1.8.6.

--=20
http://jeremymcanally.com/
http://entp.com/
http://omgbloglol.com

My books:
http://manning.com/mcanally/
http://humblelittlerubybook.com/ (FREE!)

 

[Iñaki Baz Castillo]

=BF?

All the new stuff to do with String and encodings in ruby 1.9 is
undocumented.

(At least, it's not documented within Ruby itself. You may be able to
purchase a book which has some reverse-engineered documentation)

Regexp.new of Ruby 1.9 is obviously documented in:
http://www.ruby-doc.org/core-1.9/classes/Regexp.html
but the number of parameters doesn't match with the reality =C2=BF?

Does it make sense? Isn't that documentation been created with Rdoc?


--=20
I=C3=B1aki Baz Castillo
<[email protected]>

 

[Brian Candler]

Regexp.new of Ruby 1.9 is obviously documented in:
http://www.ruby-doc.org/core-1.9/classes/Regexp.html
but the number of parameters doesn't match with the reality ¿?

Does it make sense? Isn't that documentation been created with Rdoc?

The rdoc is only as good as the comments in the source code.

 

[Brian Candler]

I've got the majority of the new functionality covered in my m17n
series now:

http://blog.grayproductions.net/articles/understanding_m17n

I expect to have the minor side topics I'm still missing covered in
the next few weeks.

This is a good start, but I think it just scratches the surface.

Questions which immediately spring to mind:

* What is the nature of the "compatible" relationship? Does A compatible
with B imply B compatible with A? It's not commutative:

irb(main):002:0> a = "abc".force_encoding("UTF-8")
=> "abc"
irb(main):003:0> b = "def".force_encoding("ISO-8859-1")
=> "def"
irb(main):004:0> Encoding.compatible?(a,b)
=> #<Encoding:UTF-8>
irb(main):005:0> Encoding.compatible?(b,a)
=> #<Encoding:ISO-8859-1>

Also, it's not encodings which are compatible, but actual strings. Two
strings may or may not be compatible, dependent not just on their
encoding, but on their actual content at that instant.

irb(main):006:0> a = "abc\xff".force_encoding("UTF-8")
=> "abc\xFF"
irb(main):007:0> b = "def\xff".force_encoding("ISO-8859-1")
=> "def�"
irb(main):008:0> Encoding.compatible?(a,b)
=> nil

* What about string literals which include escape sequences like \u?
This seems to override the source encoding rule.

$ ruby19
# encoding: ISO-8859-1
puts "abc".encoding
puts "abc\u1234".encoding
^D
ISO-8859-1
UTF-8

* What encoding is chosen for regexp literals? (Seems to be different
rules to string literals). What about string literals which include
#{interpolation}? What about regexp literals which include
#{interpolation}?

* What source encoding and external encoding is used in irb?

* I think it will be worth explaining what you need to do to handle
binary data (using "rb" and "wb", the ASCII-8BIT encoding, how to set
external encoding for STDIN, the fact that read() and gets() return
different encodings for the same data...)

* What actually happens if you use string operations on two strings with
different encodings? e.g. str1 == str2, str1 + str2, str1 << str2? What
about indexing a hash with two strings which are identical byte
sequences but different encodings?

* What do C extension writers need to know about strings? It seems at
the moment there is some magic hidden state (ENC_CODERANGE_7BIT) which
you must remember to update whenever you create or modify a string, and
if you don't, things break badly.
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/329267
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/23155

Regards,

Brian.

 

[James Gray]

* What about string literals which include escape sequences like \u?
This seems to override the source encoding rule.

I plan to cover this in the next article.

* What encoding is chosen for regexp literals? (Seems to be different
rules to string literals). What about string literals which include
#{interpolation}? What about regexp literals which include
#{interpolation}?

I'm going to cover this too.

* I think it will be worth explaining what you need to do to handle
binary data (using "rb" and "wb", the ASCII-8BIT encoding, how to set
external encoding for STDIN, the fact that read() and gets() return
different encodings for the same data...)

Planned for the next article.

* What actually happens if you use string operations on two strings
with
different encodings? e.g. str1 == str2, str1 + str2, str1 << str2?
What
about indexing a hash with two strings which are identical byte
sequences but different encodings?

I feel a gave a much better strategy that prevents you from worrying
about such things. However, that article did link to a detailed
explanation.

James Edward Gray II

 

[James Gray]

I plan to cover this in the next article.


I'm going to cover this too.


Planned for the next article.

I've added a new post to my m17n series covering all of the above and
more:

http://blog.grayproductions.net/articles/miscellaneous_m17n_details

James Edward Gray II