[bug] irb or xterm crash with UTF-8

[Simon Strandgaard]

server> irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*')
=> "pÂÂ

Then it just hangs forever..


server> xterm -version
XFree86 4.2.99.903(174)
server> ruby -v
ruby 1.9.0 (2004-05-17) [i386-freebsd5.1]
server>

 

[Carlos]

server> irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*')
=> "pÂÂ

Then it just hangs forever..

$ irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*').unpack("C*")
=> [112, 194, 128, 194, 144]

I guess the last character is interpreted as a control character by your
xterm. Reading the xterm changelog, version 175 (one more than to have) has
an option to treat characters between 128 and 159 as printable and not
control.

--

 

[Simon Strandgaard]

server> irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*')
=> "pÂÂ

Then it just hangs forever..

$ irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*').unpack("C*")
=> [112, 194, 128, 194, 144]

I guess the last character is interpreted as a control character by your
xterm. Reading the xterm changelog, version 175 (one more than to have) has
an option to treat characters between 128 and 159 as printable and not
control.

That makes sense. Thanks.

BTW: any suggestion for a better alternative than xterm?
hopefully with antialiased font rendering.. (but not qt)

 

[Dave Baldwin]

dave 31% irb
irb(main):001:0> require 'matrix.rb'
=> true
irb(main):002:0> a =Vector[1, 2, 3, 4]
=> Vector[1, 2, 3, 4]
irb(main):003:0> a + a
=> Vector[2, 4, 6, 8]
irb(main):004:0> a * 2
=> Vector[2, 4, 6, 8]
irb(main):005:0> a + 2
TypeError: cannot convert Vector into Float
from /usr/local/lib/ruby/1.8/matrix.rb:1190:in `coerce'
from /usr/local/lib/ruby/1.8/matrix.rb:1190:in `+'
from (irb):5
irb(main):006:0>

dave 32% ruby -v
ruby 1.8.1 (2003-10-31) [powerpc-darwin]

Can anyone point to what is wrong or is this a bug?

Dave.

 

[Yukihiro Matsumoto]

Hi,

In message "Bug in Vector class?"

|Can anyone point to what is wrong or is this a bug?

We defined Vector#+ as addition between vectors. Tell me if it is a
wrong decision.

matz.

 

[Dave Baldwin]

Hi,

In message "Bug in Vector class?"

|Can anyone point to what is wrong or is this a bug?

We defined Vector#+ as addition between vectors. Tell me if it is a
wrong decision.

matz.

I was hoping to see

vector op vector op = +, -, *, / component wise operations
vector op scalar op = +, -, *, /
scalar op vector op = +, -, *

Dave.

 

[Thomas Dickey]

server> irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*')
=> "pÂÂ

Then it just hangs forever..

$ irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*').unpack("C*")
=> [112, 194, 128, 194, 144]

I guess the last character is interpreted as a control character by your
xterm. Reading the xterm changelog, version 175 (one more than to have) has
an option to treat characters between 128 and 159 as printable and not
control.

but the thread seems to indicate it wasn't setup properly for UTF-8 anyway.

That makes sense. Thanks.

BTW: any suggestion for a better alternative than xterm?
hopefully with antialiased font rendering.. (but not qt)

xterm does that...

xterm supports ANSI color, VT220 emulation and UTF-8
There's an faq at
http://invisible-island.net/xterm/xterm.faq.html
ftp://invisible-island.net/xterm/

 

[Simon Strandgaard]

[Simon Strandgaard <[email protected]>, 2004-05-19 13.45 CEST]
server> irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*')
=> "pÂÂ

Then it just hangs forever..

$ irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*').unpack("C*")
=> [112, 194, 128, 194, 144]

I guess the last character is interpreted as a control character by your
xterm. Reading the xterm changelog, version 175 (one more than to have) has
an option to treat characters between 128 and 159 as printable and not
control.

but the thread seems to indicate it wasn't setup properly for UTF-8 anyway.

yes.. I only wanted Ruby to use UTF-8.. but not output UTF-8.
I have one time long time ago made an attempt to switch xterm + the applications I
use to UTF-8.. but that didn't went out successfully. Im using iso8859-1 now,
but I am now considering making an attempt again.

xterm does that...

yes, Carlos mailed me in private about this.
xterm -fa 'Luxi Mono'.

xterm supports ANSI color, VT220 emulation and UTF-8
There's an faq at
http://invisible-island.net/xterm/xterm.faq.html
ftp://invisible-island.net/xterm/

Thanks, lots of useful info. I wasn't aware that so many
other terminals are derived from xterm.


BTW: Thomas, its nice to see you hanging out in comp.lang.ruby

 

[Thomas Dickey]

yes.. I only wanted Ruby to use UTF-8.. but not output UTF-8.
I have one time long time ago made an attempt to switch xterm + the applications I
use to UTF-8.. but that didn't went out successfully. Im using iso8859-1 now,
but I am now considering making an attempt again.

I took a quick look at your example, and (won't have time til the weekend)
did see a problem in xterm that I'll have to investigate.

The -k8 option shouldn't be needed, but does seem to "fix" the problem.

yes, Carlos mailed me in private about this.
xterm -fa 'Luxi Mono'.

Thanks, lots of useful info. I wasn't aware that so many
other terminals are derived from xterm.

not all are derived (in the literal sense), but many borrow ideas.
On occasion I do see literal borrowing (I have a few examples in mind).

BTW: Thomas, its nice to see you hanging out in comp.lang.ruby

google is very helpful (I keep an eye out for problem reports).

 

[Simon Strandgaard]

I took a quick look at your example, and (won't have time til the weekend)
did see a problem in xterm that I'll have to investigate.

The -k8 option shouldn't be needed, but does seem to "fix" the problem.

server> xterm -k8 &
-----
server> xterm -version
XTerm(189)
server> irb
irb(main):001:0> $KCODE='U'
=> "U"
irb(main):002:0> str = [0x70, 0x80, 0x90].pack('U*')
=> "pÂÂ"
irb(main):003:0>

Yes, the -k8 fixes it.

" -/+k8 turn on/off C1-printable classification "

Without the -k8 option it still hangs.


Does any applications output bytes in the C1 area?
Can I safely assume that none is doing so?


I wasn't able to compile xterm with freetype support, but
I have freetype installed.

server> freetype-config --version
9.5.3
server> freetype-config --libs
-L/home/neoneye/stow/freetype/lib -lfreetype -lz
server> freetype-config --cflags
-I/home/neoneye/stow/freetype/include/freetype2 -I/home/neoneye/stow/freetype/include
server>

I tried configuring without the '--disable-freetype' option.

server> ./configure --prefix=/home/neoneye/stow/xterm
loading cache ./config.cache
checking host system type... i386-unknown-freebsd5.1
(cached) Configuring for freebsd5.1
checking for gcc... (cached) gcc
checking whether the C compiler (gcc ) works... yes
checking whether the C compiler (gcc ) is a cross-compiler... no
checking whether we are using GNU C... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking how to run the C preprocessor... (cached) gcc -E
checking whether gcc needs -traditional... (cached) no
checking for mawk... (cached) nawk
checking for a BSD compatible install... (cached) /usr/bin/install -c
checking for AIX... no
checking for POSIXized ISC... no
checking for ncurses/term.h... (cached) no
checking for stdlib.h... (cached) yes
checking for termios.h... (cached) yes
checking for unistd.h... (cached) yes
checking whether time.h and sys/time.h may both be included... (cached) yes
checking for size_t in <sys/types.h> or <stdio.h>... (cached) yes
checking for ANSI C header files... (cached) yes
checking for time_t... (cached) yes
checking for bcopy... (cached) yes
checking for gethostname... (cached) yes
checking for getlogin... (cached) yes
checking for memmove... (cached) yes
checking for strerror... (cached) yes
checking for strftime... (cached) yes
checking for tcgetattr... (cached) yes
checking for waitpid... (cached) yes
checking for memmove... (cached) yes
checking for full tgetent function... (cached) -ltermcap
checking for lastlog.h... (cached) no
checking for paths.h... (cached) yes
checking for lastlog path... (cached) no
checking for utmp implementation... (cached) utmp
checking if utmp.ut_host is declared... (cached) yes
checking if utmp.ut_name is declared... (cached) ut_name
checking for exit-status in utmp... (cached) no
checking if utmp.ut_xtime is declared... (cached) no
checking if utmp.ut_session is declared... (cached) no
checking if utmp is SYSV flavor... (cached) no
checking if you want to link with utempter... no
checking if external errno is declared... (cached) yes
checking if external errno exists... (cached) yes
checking for tty group name... (cached) tty
checking if we may use tty group... (cached) yes
checking for gcc option to accept ANSI C... (cached) -DCC_HAS_PROTOS
checking for working const... (cached) yes
checking for Cygwin environment... (cached) no
checking for mingw32 environment... (cached) no
checking for executable suffix... (cached) no
checking for object suffix... (cached) o
checking for sys/wait.h that is POSIX.1 compatible... (cached) yes
checking for POSIX wait functions... (cached) yes
checking if external sys_nerr is declared... (cached) yes
checking if external sys_nerr exists... (cached) yes
checking if external sys_errlist is declared... (cached) yes
checking if external sys_errlist exists... (cached) yes
checking if we should define SYSV... (cached) no
checking for elf_begin in -lelf... (cached) no
checking if we must define _GNU_SOURCE... (cached) no
checking for X... (cached) libraries /usr/X11R6/lib, headers /usr/X11R6/include
checking for dnet_ntoa in -ldnet... (cached) no
checking for dnet_ntoa in -ldnet_stub... (cached) no
checking for gethostbyname... (cached) yes
checking for connect... (cached) yes
checking for remove... (cached) yes
checking for shmat... (cached) yes
checking for IceConnectionNumber in -lICE... (cached) yes
checking for XOpenDisplay... (cached) no
checking for XOpenDisplay in -lX11... (cached) yes
checking for XtAppInitialize... (cached) no
checking for XtAppInitialize in -lXt... (cached) yes
checking for X11/DECkeysym.h... (cached) yes
checking for X11/Sunkeysym.h... (cached) yes
checking for X11/Xpoll.h... (cached) yes
checking for XextCreateExtension in -lXext... (cached) yes
checking for X11/Xaw/SimpleMenu.h... yes
checking for XawSimpleMenuAddGlobalActions in -lXaw -lXmu... yes
checking for declaration of fd_set... (cached) sys/types.h
checking for IRIX 6.5 baud-rate redefinitions... (cached) no
checking for grantpt... (cached) yes
checking for xterm... (cached) /usr/X11R6/bin/xterm
checking for XKB Bell extension... (cached) yes
checking for Xutf8LookupString... (cached) yes
checking if we should use imake to help... yes
checking for xmkmf... (cached) /usr/X11R6/bin/xmkmf
checking for default terminal-id... vt100
checking for default terminal-type... xterm
checking for private terminfo-directory... none
checking if you want active-icons... yes
checking if you want ANSI color... yes
checking if you want 16 colors like aixterm... yes
checking if you want 256 colors... no
checking if you want 88 colors... no
checking if you want blinking cursor... yes
checking if you want to ignore Linux's broken palette-strings... no
checking if you want to allow broken string-terminators... no
checking if you want printable 128-159... yes
checking if you want bold colors mapped like IBM PC... yes
checking if you want separate color-classes... yes
checking if you want color-mode enabled by default... yes
checking if you want support for color highlighting... yes
checking if you want support for doublesize characters... yes
checking if you want fallback-support for box characters... yes
checking if you want to use FreeType library... yes
checking for xft-config... (cached) /usr/X11R6/bin/xft-config
checking for X FreeType headers... (cached) yes
checking for X FreeType libraries... (cached) yes
checking if you want support for HP-style function keys... no
checking if you want support for SCO-style function keys... no
checking if you want support for internationalization... yes
checking if you want support for initial-erase setup... yes
checking if you want support for input-method... yes
checking if X libraries support input-method... (cached) no
checking if you want support for load-vt-fonts... no
checking if you want support for logging... no
checking if you want support for iconify/maximize translations... yes
checking if you want NumLock to override keyboard tables... yes
checking if you want support for pty-handshaking... yes
checking if you want support for right-scrollbar... yes
checking if you want check for redundant name-change... yes
checking if you want support for session management... yes
checking if you want to use termcap-query/report... no
checking if you want support for tek4014... yes
checking if you want pulldown menus with a toolbar... no
checking if you want VT52 emulation... yes
checking if you want to use luit... no
checking if you want wide-character support... no
checking if you want dynamic-abbreviation support... no
checking if you want DECterm Locator support... no
checking if you want -ziconbeep option... yes
checking if you want debugging traces... no
checking if you want to see long compiling messages... yes
checking if you want magic cookie emulation... no
checking if you want to turn on gcc warnings... no
updating cache ./config.cache
creating ./config.status
creating Makefile
creating xtermcfg.h
server> gmake
gcc -I. -I. -DHAVE_CONFIG_H -I. -I/usr/X11R6/include -DCSRG_BASED -DFUNCPROTO=15 -DNARROWPROTO -DUTMP -DOSMAJORVERSION=5 -DOSMINORVERSION=1 -I/usr/X11R6/include -DPROJECTROOT='"/usr/X11R6"' -g -O2 yes -c ./button.c
gcc: yes: No such file or directory
In file included from ptyx.h:79,
from xterm.h:255,
from button.c:62:
/usr/X11R6/include/X11/Xft/Xft.h:41:31: freetype/freetype.h: No such file or directory
In file included from ptyx.h:79,
from xterm.h:255,
from button.c:62:
/usr/X11R6/include/X11/Xft/Xft.h:58: syntax error before "_XftFTlibrary"
/usr/X11R6/include/X11/Xft/Xft.h:58: warning: data definition has no type or storage class
/usr/X11R6/include/X11/Xft/Xft.h:92: syntax error before "FT_UInt"
/usr/X11R6/include/X11/Xft/Xft.h:99: syntax error before "FT_UInt"
/usr/X11R6/include/X11/Xft/Xft.h:196: syntax error before '*' token
/usr/X11R6/include/X11/Xft/Xft.h:301: syntax error before '*' token
/usr/X11R6/include/X11/Xft/Xft.h:360: syntax error before "XftLockFace"
/usr/X11R6/include/X11/Xft/Xft.h:360: warning: data definition has no type or storage class
/usr/X11R6/include/X11/Xft/Xft.h:399: syntax error before '*' token
/usr/X11R6/include/X11/Xft/Xft.h:405: syntax error before '*' token
/usr/X11R6/include/X11/Xft/Xft.h:414: syntax error before "FT_UInt"
/usr/X11R6/include/X11/Xft/Xft.h:424: syntax error before "XftCharIndex"
/usr/X11R6/include/X11/Xft/Xft.h:426: warning: data definition has no type or storage class
/usr/X11R6/include/X11/Xft/Xft.h:467: syntax error before '*' token
gmake: *** [button.o] Error 1
server>

It seems as a problem with freetype..

google is very helpful (I keep an eye out for problem reports).

kudos.

 

[Thomas Dickey]

Does any applications output bytes in the C1 area?

most do not (for Linux).

Can I safely assume that none is doing so?

reasonably safe. A real vt220 would respond to control characters there,
but the feature should be inactive in UTF-8 mode.

I wasn't able to compile xterm with freetype support, but
I have freetype installed.

It seems as a problem with freetype..

It seems that freetype has many slightly-incompatible versions. (I'll
check over this to see if I can see where I might fix my test-script).

 

[Simon Strandgaard]

most do not (for Linux).


reasonably safe. A real vt220 would respond to control characters there,
but the feature should be inactive in UTF-8 mode.

<lame idea>
IIUC xterm has C1 enabled per default. Wouldn't it be safer to
let it be disabled per default? If people wants a fully vt220 compatible
terminal, then they can enable it themselves.

It seems that freetype has many slightly-incompatible versions. (I'll
check over this to see if I can see where I might fix my test-script).

I looked a little at your aclocal.m4, (I am no autoconf expert).
It seems as CF_X_FREETYPE primary uses 'xft-config' and then as a
fall-through 'freetype-config'.

I have 2 versions of freetype installed.. a very old version (xft-config)
and the latest version (freetype-config).

Wouldn't it make sense to probe the 'freetype-config' first ?

 

[Thomas Dickey]

<lame idea>
IIUC xterm has C1 enabled per default. Wouldn't it be safer to
let it be disabled per default? If people wants a fully vt220 compatible
terminal, then they can enable it themselves.
</lame idea>

I'd rather just make it work properly (in this case, -k8 is fixing
a coincidental bug).

I looked a little at your aclocal.m4, (I am no autoconf expert).
It seems as CF_X_FREETYPE primary uses 'xft-config' and then as a
fall-through 'freetype-config'.

I have 2 versions of freetype installed.. a very old version (xft-config)
and the latest version (freetype-config).

Wouldn't it make sense to probe the 'freetype-config' first ?

Not exactly - for the configurations I was familiar with, xft-config
provided the information needed, while freetype-config gave only part
of it. (But this is all guesswork, since neither is documented ;-)

 

[Simon Strandgaard]

I'd rather just make it work properly (in this case, -k8 is fixing
a coincidental bug).
Ok.





Not exactly - for the configurations I was familiar with, xft-config
provided the information needed, while freetype-config gave only part
of it. (But this is all guesswork, since neither is documented ;-)

Agree, its unfortunate that there are no standards for app-config files.
I have also had tons of problems integrating them with autoconf.
Furtunatly I don't work on any C/C++ projects at the moment (only
enjoying my addiction to precious Ruby).

Good luck with xterm/ncurses.

 

[Thomas Dickey]

Agree, its unfortunate that there are no standards for app-config files.
I have also had tons of problems integrating them with autoconf.
Furtunatly I don't work on any C/C++ projects at the moment (only
enjoying my addiction to precious Ruby).

perhaps sometime I'll work with Ruby, but for now have more than enough work.

Good luck with xterm/ncurses.

thanks