ruby unicode/string explosion (0xFF in utf-8)

S

Sylvester T Cat

Hi, I'm using ruby 1.9.2

I'm reading a CSV file that has some non US-ASCII characters. I want
to parse each value in each row and strip out any leading/lagging
potential whitespace.
However, when I come across some unusual characters, I get invalid
byte sequence in UTF-8

here is an example:

irb(main):041:0* a = "\xFF"
=> "\xFF"
irb(main):042:0> a.encoding
=> #<Encoding:UTF-8>
irb(main):043:0> a.strip
ArgumentError: invalid byte sequence in UTF-8
from (irb):43:in `strip'
from (irb):43
from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:44:in `start'
from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:8:in `start'
from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands.rb:23:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'

# so now I'm going to try and change encoding, but this doesn't work
either

irb(main):044:0> a.encode!("ASCII-8BIT", undef: :replace)
Encoding::InvalidByteSequenceError: "\xFF" on UTF-8
from (irb):44:in `encode!'
from (irb):44
from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:44:in `start'
from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:8:in `start'
from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands.rb:23:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'

Is there any way to strip out these characters while staying with
utf-8 encoding?
 
B

Brian Candler

Sylvester T Cat wrote in post #967790:
I'm reading a CSV file that has some non US-ASCII characters. I want
to parse each value in each row and strip out any leading/lagging
potential whitespace.
However, when I come across some unusual characters, I get invalid
byte sequence in UTF-8

I guess it's not genuinely UTF-8.

If you think it *is* sort of broken UTF-8 which includes FF characters
for some reason, then you could force encoding to binary, remove the FF
characters, then force back to UTF-8.

More likely I'd have thought it was a single-byte encoding (like
ISO-8859-1 perhaps). But in any case, if you're just doing CSV parsing,
you can quite legitimately treat UTF-8 as binary - since all you need to
do is recognise commas and double quotes, and the rest just gets passed
through.

More info at
https://github.com/candlerb/string19/blob/master/string19.rb

Or just use ruby 1.8.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,755
Messages
2,569,537
Members
45,023
Latest member
websitedesig25

Latest Threads

Top