Ruby 1.9.2 UTF-8 Encoding issues whiles reading/writing files

Discussion in 'Ruby' started by Atoli Atoli, Nov 18, 2010.

  1. Atoli Atoli

    Atoli Atoli Guest

    Hello

    My ruby 1.9.2 does some strange things when manipulating encodings
    especially when reading and writing text file.
    I have a file (attached) with broken UTF-8 characters: E5 AD 97 E6 99 2E
    The offending sequence is E6 99

    And here's the example I'm using to try to figure out how the heck does
    this Encoding stuff work:
    #--------------------------------------
    data = File.open("broken.txt", "r:UTF-8") { |f| f.read }
    puts data.valid_encoding?

    utf_a = data.encode("UTF-8", invalid: :replace, undef: :replace,
    replace: "_")
    puts utf_a.valid_encoding?

    utf_b = data.encode("UTF-8", "UTF-8", invalid: :replace, undef:
    :replace, replace: "_")
    puts utf_b.valid_encoding?
    puts (utf_a == utf_b) && (data == utf_b)

    File.open("valid.txt", "w:UTF-8") { |f| f.write(utf_a) }
    #--------------------------------------

    The output is:
    false
    false
    true
    true

    Basically I'm trying to replace the broken sequences with "_", but the
    encode method doesn't seem to do any replacements, maybe because the
    forced encoding is already set to UTF-8?

    I've read James Edward post concerning strings encoding in ruby 1.9 and
    also candlerb's doc, but didn't find anything.

    This can't be so complicated, right? Sure I'm missing something.

    Thank you.

    Atoli.

    Attachments:
    http://www.ruby-forum.com/attachment/5415/broken.txt
     
    Atoli Atoli, Nov 18, 2010
    #1
    1. Advertisements

  2. Atoli Atoli

    brabuhr Guest

    For the case of fixing broken files, I would probably use iconv from the sh=
    ell:

    $ cat broken.txt
    =E5=AD=97?.
    $ iconv -f UTF8 -t UTF8 --byte-subst=3D_ broken.txt
    =E5=AD=97__.

    (I don't know if Ruby's Iconv module supports the subst options.)

    For short strings, this seems to work:

    irb(main):001:0> s =3D "\xE5\xAD\x97\xE6\x99\x2E"
    =3D> "=E5=AD=97\xE6\x99."
    irb(main):002:0> s.encoding
    =3D> #<Encoding:UTF-8>
    irb(main):003:0> s.valid_encoding?
    =3D> false
    irb(main):004:0> t =3D s.chars.map{|c| c.valid_encoding? ? c : '_'}.join
    =3D> "=E5=AD=97__."
    irb(main):005:0> t.valid_encoding?
    =3D> true
    irb(main):006:0> t.encoding
    =3D> #<Encoding:UTF-8>
     
    brabuhr, Nov 18, 2010
    #2
    1. Advertisements

  3. Atoli Atoli

    Atoli Atoli Guest

    Thanks for the tip.

    For now, making ruby "think" the encoding is valid seems to work (it
    doesn't break regular expressions at least).
    So I just encode("UTF-8", "UTF-8", invalid: :replace, undef:
    :replace, replace: "_") each time I read my files.
     
    Atoli Atoli, Nov 18, 2010
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.