Ways to filter bad(unicode) characters

[Ramza Brown]

I am parsing a collection of URLS; URLS that seem to have Chinese/Indian
and other unicode characters. My question, how can I filter those out
while still leaving room for alpha-numeric and characters that are
typical of a URL or Title

For example I might get a URL with:
http://????????????-????????
title = ??????

where the ? represents some unicode character

I want to filter these out, but leave room for non-alphanumeric characters:

http://www.yahoo.com

--
Berlin Brown
(ramaza3 on freenode)
http://www.newspiritcompany.com
http://www.newspiritcompany.com/newforums
also checkout alpha version of botverse:
http://www.newspiritcompany.com:8086/universe_home

 

[baumanj]

How about:

new_url = ''
url.each_byte {|b| new_url << b if b < 128 }

That should keep all the ASCII bytes and drop all the non-ASCII ones.

 

[Ramza Brown]

How about:

new_url = ''
url.each_byte {|b| new_url << b if b < 128 }

That should keep all the ASCII bytes and drop all the non-ASCII ones.

I didnt want to modify the URL so much as check the how valid (US valid
that is).



--
Berlin Brown
(ramaza3 on freenode)
http://www.newspiritcompany.com
http://www.newspiritcompany.com/newforums
also checkout alpha version of botverse:
http://www.newspiritcompany.com:8086/universe_home

 

[baumanj]

In that case:

invalid = false
url.each_byte {|b| invalid = true if b > 127 }

 

[Dave Burt]

In that case:

invalid = false
url.each_byte {|b| invalid = true if b > 127 }

Or:

require 'enumerator'
class String
def seven_bit_clean?
self.each_byte.all? {|c| c <= 127 }
end
end

Cheers,
Dave