Automated testing of visual library (ncurses)

[(rkumar) Sentinel]

As i work more on a ncurses widget library, I am wondering how does one
do automated testing for a software that is primarily visual. Whenever i
make changes, I need to ensure that things still show correctly - the
cursor position on screen, widget placement, text placement, text
wrapping, highlighting and attributes of widget, scrolling, cropping,
navigation, key handling, etc.

Currently, i have no way of knowing what's broken.

1. One thought that comes is to take some kind of screen dump and
compare screen dumps when testing. (I am on a unix terminal, of course,
OS X)


2. I don't think logging to a file and comparing logs would be of much
benefit.


Thoughts and pointers welcome.

 

[Phillip Gawlowski]

As i work more on a ncurses widget library, I am wondering how does one
do automated testing for a software that is primarily visual. Whenever i
make changes, I need to ensure that things still show correctly - the
cursor position on screen, widget placement, text placement, text
wrapping, highlighting and attributes of widget, scrolling, cropping,
navigation, key handling, etc.

Thoughts and pointers welcome.

You could take a look at web app testing frameworks (like WATIR, or
Selenium), and see how they do their tests.

That assumes, though, that you have similar access to ncurses widgets as
you'd have for HTML tags.

Another option are tools emulating keyboard / mouse input, and let those
loose on the application with the use of macros / testing frameworks.

Not sure if you can test for "pixel perfectness", but since ncurses'
layout depends on the size of the TTY used, I don't know if that'd be
useful, anyway.

 

[Seebs]

1. One thought that comes is to take some kind of screen dump and
compare screen dumps when testing. (I am on a unix terminal, of course,
OS X)

2. I don't think logging to a file and comparing logs would be of much
benefit.

Hmm.

Suggestion:

See if you can figure out a way to externally command screen(1) to do a
hardcopy dump, and if so, run your stuff in a screen session, then dump
the current state of the display to a file -- I think that should get
you the same characters for the same output regardless of how you got
there.

Basically, you need a terminal emulator which is reliable at obeying
curses commands. Luckily, one exists.

-s

 

[(rkumar) Sentinel]

Basically, you need a terminal emulator which is reliable at obeying
curses commands. Luckily, one exists.

-s

I develop using screen.

Screen's "C-a h" creates a hardcopy *sans* attributes.
But i guess that's a start - I guess one can't have everything.
If i can find a way to get part coverage, that would help.


Unfortunately there's so much that's interactive. Clicking, typing text,
scrolling and then typing text.

Perhaps with a screen hardcopy, I can at least see that all test screens
load and display the basic widgets correctly positioned with starting
data.

btw, is there something i am missing in your statement "which is
reliable at ***obeying curses commands***".

thanks, rk.

 

[Marnen Laibow-Koser]

I develop using screen.

Screen's "C-a h" creates a hardcopy *sans* attributes.
But i guess that's a start - I guess one can't have everything.
If i can find a way to get part coverage, that would help.


Unfortunately there's so much that's interactive. Clicking, typing text,
scrolling and then typing text.

Perhaps with a screen hardcopy, I can at least see that all test screens
load and display the basic widgets correctly positioned with starting
data.

Is there something like Webrat that will work for console apps?

btw, is there something i am missing in your statement "which is
reliable at ***obeying curses commands***".

thanks, rk.

Best,

 

[Seebs]

Screen's "C-a h" creates a hardcopy *sans* attributes.
But i guess that's a start - I guess one can't have everything.
If i can find a way to get part coverage, that would help.

It's at least a start.

Unfortunately there's so much that's interactive. Clicking, typing text,
scrolling and then typing text.
Hmm.

Perhaps with a screen hardcopy, I can at least see that all test screens
load and display the basic widgets correctly positioned with starting
data.

Possibly. You could do hardcopies basically every "frame", too. e.g.,
send the first letter of a password, verify that the hardcopy now shows
the *, or whatever.

You could probably also hack screen to write a second hardcopy file which
is a grid of attributes in some notation, so you could compare the pair
of files.

btw, is there something i am missing in your statement "which is
reliable at ***obeying curses commands***".

Essential issues:

1. There is no guarantee that two correct sets of curses commands which
produce the same exact final screen output do so by sending the same
sequence of characters. There may be many possible ways to, say,
clear all the characters on a given line. Depending on non-obvious
externalities, or current cursor location, you might see anything from
a sequence of spaces to a special line-clear command... Curses is
specifically designed to "optimize" these things, meaning that the
sequence of characters produced is not necessarily consistent for a
given output.
2. Related to the above, curses can and will overwrite things. So
imagine that curses were to send:
foo^H^H^Hbar
You probably want to treat this precisely the same as just sending "bar".

So basically, you want a thing which interprets the terminal escape
sequences curses generates and tells you what the final output looks
like. Happily, screen does that (at least for vt100).

-s

 

[Al]

Check out Expect (TCL extension; http://expect.nist.gov/) if you are
only charactor based.

 

[Michael Fellinger]

As i work more on a ncurses widget library, I am wondering how does one
do automated testing for a software that is primarily visual. Whenever i
make changes, I need to ensure that things still show correctly - the
cursor position on screen, widget placement, text placement, text
wrapping, highlighting and attributes of widget, scrolling, cropping,
navigation, key handling, etc.

Currently, i have no way of knowing what's broken.

1. One thought that comes is to take some kind of screen dump and
compare screen dumps when testing. (I am on a unix terminal, of course,
OS X)


2. I don't think logging to a file and comparing logs would be of much
benefit.


Thoughts and pointers welcome.

I've been working on a pure-ruby terminal emulator the past few
days... unfortunately it doesn't handle many things yet, in particular
scroll regions are problematic.
If you want to try it, or contribute, grab ruby 1.9, ffi-tk, and
http://github.com/manveru/yonde
I'd be happy about anyone who has some experience with terminals, my
best reference right now is the manpage for terminfo...

Anyway, using yonde, you could test any aspect of your application
using ruby, and you can also check things like use of color or other
attributes, which a hardcopy can't give you.

 

[(rkumar) Sentinel]

I've been working on a pure-ruby terminal emulator the past few
days... unfortunately it doesn't handle many things yet, in particular
scroll regions are problematic.
If you want to try it, or contribute, grab ruby 1.9, ffi-tk, and
http://github.com/manveru/yonde

1. Does it work on 1.9
2. Does it require Ruby Tk ?

 

[Michael Fellinger]

1. Does it work on 1.9
2. Does it require Ruby Tk ?

It works _only_ on Ruby 1.9, and it doesn't require Ruby Tk.