Setting encoding of pages in Capybara

[James Coglan]

[Note: parts of this message were removed to make it a legal post.]

Hi all,

Quick encoding question: say I'm trying to grab data from a Japanese page
using Capybara and Rack::Test, and I get badly encoded text in the response.
e.g. running this script:

require 'rubygems'
require 'capybara'
require 'rack/test'
require 'rack/proxy'

Capybara.default_selector = :css

class Japan < Rack:roxy
def rewrite_env(env)
env['HTTP_HOST'] = 'l-tike.com'
env
end
end

session = Capybara::Session.newrack_test, Japan.new)
session.visit '/pickup/concert_more.html'
puts session.body

You'll see weird characters in the output, and I can't find nodes that
should be there with css/xpath. How do I set the encoding so that Nokogiri
parses the page properly?

 

[Mike Dalessio]

[Note: parts of this message were removed to make it a legal post.]

Hi,

Hi all,

Quick encoding question: say I'm trying to grab data from a Japanese page
using Capybara and Rack::Test, and I get badly encoded text in the
response.
e.g. running this script:

First, a quick note, that this question is probably more appropriate for the
capybara or nokogiri mailing lists. You're likely to get a quicker response
from those groups.

require 'rubygems'
require 'capybara'
require 'rack/test'
require 'rack/proxy'

Capybara.default_selector = :css

class Japan < Rack:roxy
def rewrite_env(env)
env['HTTP_HOST'] = 'l-tike.com'
env
end
end

session = Capybara::Session.newrack_test, Japan.new)
session.visit '/pickup/concert_more.html'
puts session.body

It looks like this page claims (in its header) to be encoding in SHIFT_JIS,
but the page is encoded in UTF-8. LibXML's guesses at encoding are not
perfect, and in this case the misleading information causes it to trust the
header and use the wrong encoding.

If this page is edited to contain

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

instead of

<meta http-equiv="Content-Type" content="text/html; charset=Shift_JIS">

then all is well.


Perhaps someone with more experience than me using non-western character
sets will have a deeper insight into libxml's behavior here?