VHDL Automated Testing

J

Jonathan Ross

I'm trying to implement an automated testing hardness for a VHDL
project. We compile and link with vhpcomp and fuse respectively.

1. The program generated will only run under xtclsh - is there any way
to get it to run in bash?
2. The following code generates the following output:

PROCESS
BEGIN
REPORT "Hello World!" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
WAIT;
END PROCESS;

ISim> run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2

** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10

INFO: Simulator is stopped.
ISim>

The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?

3. GHDL seems better designed for this use, but we've seen
discrepancies between iSim and XST so we're worried they'd be even
worse with a tool not made by Xilinx. To be fair though, most of the
issues have been iSim issues more than they've been XST issues, so if
GHDL works well it might work better than XST.

Thanks.
 
N

Nicolas Matringe

Le 05/12/2010 01:09, Jonathan Ross a écrit :
[...]

2. The following code generates the following output:

PROCESS
BEGIN
REPORT "Hello World!" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
WAIT;
END PROCESS;

ISim> run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2

** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10

INFO: Simulator is stopped.
ISim>

The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Click to expand...

Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.

Nicolas
 
J

Jonathan Ross

Le 05/12/2010 01:09, Jonathan Ross a écrit :

[...]








2. The following code generates the following output:
Click to expand...
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
Click to expand...
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
Click to expand...
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
Click to expand...
INFO: Simulator is stopped.
ISim>
Click to expand...
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Click to expand...

Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction..

Nicolas
Click to expand...

Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.
 
P

Pontus

Le 05/12/2010 01:09, Jonathan Ross a écrit :
[...]
Click to expand...
2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Click to expand...
Click to expand...
Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.
Click to expand...
Nicolas
Click to expand...

Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.
Click to expand...

Instead of just doing "run" I do "run; exit" which does what you want,
at least for modelsim/riviera.

The simulation will stop at "failure" (configurable, you can stop at
warning if you wish).

Simulation will also stop when there are no more scheduled events, so
by stopping your clock
you will stop the sim.

HTH -- Pontus
 
R

rickman

Le 05/12/2010 01:09, Jonathan Ross a écrit :
[...]
Click to expand...
2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Click to expand...
Click to expand...
Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.
Click to expand...
Nicolas
Click to expand...

Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.
Click to expand...

As Pontus says, you can use a command in ISim to exit the program
after it ends the simulation run. But that doesn't tell your program
why it exited. Is there a simulation result available at the ISim
command line that could be used as an exit condition for the
program?

Rick
 
J

Jonathan Ross

Le 05/12/2010 01:09, Jonathan Ross a écrit :
[...]
2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.
Nicolas
Click to expand...
Click to expand...
Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.
Click to expand...

As Pontus says, you can use a command in ISim to exit the program
after it ends the simulation run.  But that doesn't tell your program
why it exited.  Is there a simulation result available at the ISim
command line that could be used as an exit condition for the
program?

Rick
Click to expand...

I'm not quite following - perhaps because I'm not sure what's meant by
"simulation result." Is there some object that iSim is aware of or can
be made aware of, or is this terminology colloquial? I'm open to
making any changes needed - in fact I won't be using ASSERT for this
(I'm going to follow the xUnit pattern and make my results openly
available at vhdlunit.org).
 
R

rickman

On Dec 5, 1:07 pm, Jonathan Ross <[email protected]>
wrote:
Le 05/12/2010 01:09, Jonathan Ross a écrit :
[...]
2. The following code generates the following output:
     PROCESS
     BEGIN
         REPORT "Hello World!" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
         ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
         ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
         WAIT;
     END PROCESS;
ISim>  run
Simulator is doing circuit initialization process.
at 0 ps: Note: Hello World! (/vhdlunit/).
at 0 ps: Note: 1 = 2 (/vhdlunit/).
at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
at 0 ps: Error: 1 = 2
** Failure:1 = 2
User(VHDL) Code Called Simulation Stop
In process VHDLUNIT.vhd:10
INFO: Simulator is stopped.
ISim>
The program doesn't exit even on failure, so there's no error code for
my harness to use. Also, besides reporting or asserting with severity
failure, I can't figure out how to get it to stop. How to I control my
return code?
Hi
It looks like the simulator DID exit after the failure.
Try to put something that the simulator would execute if it didn't exit
and see what happens, instead of exiting (or not) on the last instruction.
Nicolas
Actually, the simulator is still on. The iSim> prompt means it didn't
exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
it does STOP at the failure. I need it to exit so I can get an exit
code to tell me whether it failed or not by the launching application/
script.
Click to expand...
Click to expand...
As Pontus says, you can use a command in ISim to exit the program
after it ends the simulation run.  But that doesn't tell your program
why it exited.  Is there a simulation result available at the ISim
command line that could be used as an exit condition for the
program?
Click to expand...
Rick
Click to expand...

I'm not quite following - perhaps because I'm not sure what's meant by
"simulation result." Is there some object that iSim is aware of or can
be made aware of, or is this terminology colloquial? I'm open to
making any changes needed - in fact I won't be using ASSERT for this
(I'm going to follow the xUnit pattern and make my results openly
available at vhdlunit.org).
Click to expand...

That's why I'm asking the question, I don't know either. If there is
a way to use command line functions to determine the result of the
simulation... and there is a way to make ISim return an exit code on
exiting, then you can make this work the way I think you want to do
it. But I don't know if ISim will do either of those things.

Would it be easier to have ISim save the log to a file and then read
the log to get a result?

Rick
 
D

d_s_klein

Would it be easier to have ISim save the log to a file and then read
the log to get a result?

Rick
Click to expand...

Do what Rick says.

Create a log file. At the very least, have the word PASS or FAIL
appear in the log file.

Eventually you will want to have something in your log file to tell
you why and when the failure was detected.

Hint: Assertions are your friend.

RK
 
A

Alessandro Basili

Do what Rick says.

Create a log file. At the very least, have the word PASS or FAIL
appear in the log file.

Eventually you will want to have something in your log file to tell
you why and when the failure was detected.

Hint: Assertions are your friend.

RK
Click to expand...

Maybe I haven't quite well understood the topic, but you maybe
interested in this thread:

http://www.velocityreviews.com/forums/t57165-how-to-stop-simulation-in-vhdl.html

One of the first replies from Mike Treseler point to a broken link which
you may check here: http://mysite.ncnetwork.net/reszotzl/
It gives a very good approach to testbench and once there are no
scheduled transitions any longer (through done_s signal) the simulation
should stop automatically.

Hope that helps,

Al
 
J

JimLewis

turn on VHDL-2008 and use:
std.env.stop(2) ;

The intent is that the integer parameter is
returned to the calling program. Not sure
how the simulators are implementing it, but
if it does not work like you expect, submit a
bug report against it.

Best,
Jim
 
P

Paul Uiterlinden

JimLewis said:
turn on VHDL-2008 and use:
std.env.stop(2) ;

The intent is that the integer parameter is
returned to the calling program. Not sure
how the simulators are implementing it, but
if it does not work like you expect, submit a
bug report against it.
Click to expand...

In ModelSim it has been implemented as:

For both STOP and FINISH the STATUS values are those used
in the Verilog $finish task
0 prints nothing
1 prints simulation time and location
2 prints simulation time, location, and statistics about
the memory and CPU times used in simulation

Other STATUS values are interpreted as 0.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Does the last elseif run or not? 6
Issue with simulation time - Modelsim PE Student 10.4 1
concurrent assertion statement and delta races 16
Vhdl Simili, how to change a signal's value? 0
Problem with codewars. 5
Mixing different VHDL revisions between package and entity 4
assert question 4
utf8 to utf16 0

Members online

Total: 20 (members: 2, guests: 18)
Robots: 162

Forum statistics

Threads
473,777
Messages
2,569,604
Members
45,234
Latest member
SkyeWeems

Latest Threads

Top