VHDL Automated Testing

Discussion in 'VHDL' started by Jonathan Ross, Dec 5, 2010.

  1. I'm trying to implement an automated testing hardness for a VHDL
    project. We compile and link with vhpcomp and fuse respectively.

    1. The program generated will only run under xtclsh - is there any way
    to get it to run in bash?
    2. The following code generates the following output:

    PROCESS
    BEGIN
    REPORT "Hello World!" SEVERITY NOTE;
    ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
    ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
    ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
    ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
    ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
    ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
    ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
    ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
    WAIT;
    END PROCESS;

    ISim> run
    Simulator is doing circuit initialization process.
    at 0 ps: Note: Hello World! (/vhdlunit/).
    at 0 ps: Note: 1 = 2 (/vhdlunit/).
    at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
    at 0 ps: Error: 1 = 2

    ** Failure:1 = 2
    User(VHDL) Code Called Simulation Stop
    In process VHDLUNIT.vhd:10

    INFO: Simulator is stopped.
    ISim>

    The program doesn't exit even on failure, so there's no error code for
    my harness to use. Also, besides reporting or asserting with severity
    failure, I can't figure out how to get it to stop. How to I control my
    return code?

    3. GHDL seems better designed for this use, but we've seen
    discrepancies between iSim and XST so we're worried they'd be even
    worse with a tool not made by Xilinx. To be fair though, most of the
    issues have been iSim issues more than they've been XST issues, so if
    GHDL works well it might work better than XST.

    Thanks.
     
    Jonathan Ross, Dec 5, 2010
    #1
    1. Advertising

  2. Le 05/12/2010 01:09, Jonathan Ross a écrit :
    >

    [...]

    > 2. The following code generates the following output:
    >
    > PROCESS
    > BEGIN
    > REPORT "Hello World!" SEVERITY NOTE;
    > ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
    > ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
    > ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
    > ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
    > ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
    > ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
    > ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
    > ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
    > WAIT;
    > END PROCESS;
    >
    > ISim> run
    > Simulator is doing circuit initialization process.
    > at 0 ps: Note: Hello World! (/vhdlunit/).
    > at 0 ps: Note: 1 = 2 (/vhdlunit/).
    > at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
    > at 0 ps: Error: 1 = 2
    >
    > ** Failure:1 = 2
    > User(VHDL) Code Called Simulation Stop
    > In process VHDLUNIT.vhd:10
    >
    > INFO: Simulator is stopped.
    > ISim>
    >
    > The program doesn't exit even on failure, so there's no error code for
    > my harness to use. Also, besides reporting or asserting with severity
    > failure, I can't figure out how to get it to stop. How to I control my
    > return code?


    Hi
    It looks like the simulator DID exit after the failure.
    Try to put something that the simulator would execute if it didn't exit
    and see what happens, instead of exiting (or not) on the last instruction.

    Nicolas
     
    Nicolas Matringe, Dec 5, 2010
    #2
    1. Advertising

  3. On Dec 5, 8:55 am, Nicolas Matringe <> wrote:
    > Le 05/12/2010 01:09, Jonathan Ross a écrit :
    >
    > [...]
    >
    >
    >
    >
    >
    >
    >
    >
    >
    > > 2. The following code generates the following output:

    >
    > >      PROCESS
    > >      BEGIN
    > >          REPORT "Hello World!" SEVERITY NOTE;
    > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
    > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
    > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
    > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
    > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
    > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
    > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
    > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
    > >          WAIT;
    > >      END PROCESS;

    >
    > > ISim>  run
    > > Simulator is doing circuit initialization process.
    > > at 0 ps: Note: Hello World! (/vhdlunit/).
    > > at 0 ps: Note: 1 = 2 (/vhdlunit/).
    > > at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
    > > at 0 ps: Error: 1 = 2

    >
    > > ** Failure:1 = 2
    > > User(VHDL) Code Called Simulation Stop
    > > In process VHDLUNIT.vhd:10

    >
    > > INFO: Simulator is stopped.
    > > ISim>

    >
    > > The program doesn't exit even on failure, so there's no error code for
    > > my harness to use. Also, besides reporting or asserting with severity
    > > failure, I can't figure out how to get it to stop. How to I control my
    > > return code?

    >
    > Hi
    > It looks like the simulator DID exit after the failure.
    > Try to put something that the simulator would execute if it didn't exit
    > and see what happens, instead of exiting (or not) on the last instruction..
    >
    > Nicolas


    Actually, the simulator is still on. The iSim> prompt means it didn't
    exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
    it does STOP at the failure. I need it to exit so I can get an exit
    code to tell me whether it failed or not by the launching application/
    script.
     
    Jonathan Ross, Dec 5, 2010
    #3
  4. Jonathan Ross

    Pontus Guest

    On Dec 5, 7:07 pm, Jonathan Ross <>
    wrote:
    > On Dec 5, 8:55 am, Nicolas Matringe <> wrote:
    >
    >
    >
    > > Le 05/12/2010 01:09, Jonathan Ross a écrit :

    >
    > > [...]

    >
    > > > 2. The following code generates the following output:

    >
    > > >      PROCESS
    > > >      BEGIN
    > > >          REPORT "Hello World!" SEVERITY NOTE;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
    > > >          WAIT;
    > > >      END PROCESS;

    >
    > > > ISim>  run
    > > > Simulator is doing circuit initialization process.
    > > > at 0 ps: Note: Hello World! (/vhdlunit/).
    > > > at 0 ps: Note: 1 = 2 (/vhdlunit/).
    > > > at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
    > > > at 0 ps: Error: 1 = 2

    >
    > > > ** Failure:1 = 2
    > > > User(VHDL) Code Called Simulation Stop
    > > > In process VHDLUNIT.vhd:10

    >
    > > > INFO: Simulator is stopped.
    > > > ISim>

    >
    > > > The program doesn't exit even on failure, so there's no error code for
    > > > my harness to use. Also, besides reporting or asserting with severity
    > > > failure, I can't figure out how to get it to stop. How to I control my
    > > > return code?

    >
    > > Hi
    > > It looks like the simulator DID exit after the failure.
    > > Try to put something that the simulator would execute if it didn't exit
    > > and see what happens, instead of exiting (or not) on the last instruction.

    >
    > > Nicolas

    >
    > Actually, the simulator is still on. The iSim> prompt means it didn't
    > exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
    > it does STOP at the failure. I need it to exit so I can get an exit
    > code to tell me whether it failed or not by the launching application/
    > script.


    Instead of just doing "run" I do "run; exit" which does what you want,
    at least for modelsim/riviera.

    The simulation will stop at "failure" (configurable, you can stop at
    warning if you wish).

    Simulation will also stop when there are no more scheduled events, so
    by stopping your clock
    you will stop the sim.

    HTH -- Pontus
     
    Pontus, Dec 5, 2010
    #4
  5. Jonathan Ross

    rickman Guest

    On Dec 5, 1:07 pm, Jonathan Ross <>
    wrote:
    > On Dec 5, 8:55 am, Nicolas Matringe <> wrote:
    >
    >
    >
    > > Le 05/12/2010 01:09, Jonathan Ross a écrit :

    >
    > > [...]

    >
    > > > 2. The following code generates the following output:

    >
    > > >      PROCESS
    > > >      BEGIN
    > > >          REPORT "Hello World!" SEVERITY NOTE;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
    > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
    > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
    > > >          WAIT;
    > > >      END PROCESS;

    >
    > > > ISim>  run
    > > > Simulator is doing circuit initialization process.
    > > > at 0 ps: Note: Hello World! (/vhdlunit/).
    > > > at 0 ps: Note: 1 = 2 (/vhdlunit/).
    > > > at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
    > > > at 0 ps: Error: 1 = 2

    >
    > > > ** Failure:1 = 2
    > > > User(VHDL) Code Called Simulation Stop
    > > > In process VHDLUNIT.vhd:10

    >
    > > > INFO: Simulator is stopped.
    > > > ISim>

    >
    > > > The program doesn't exit even on failure, so there's no error code for
    > > > my harness to use. Also, besides reporting or asserting with severity
    > > > failure, I can't figure out how to get it to stop. How to I control my
    > > > return code?

    >
    > > Hi
    > > It looks like the simulator DID exit after the failure.
    > > Try to put something that the simulator would execute if it didn't exit
    > > and see what happens, instead of exiting (or not) on the last instruction.

    >
    > > Nicolas

    >
    > Actually, the simulator is still on. The iSim> prompt means it didn't
    > exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
    > it does STOP at the failure. I need it to exit so I can get an exit
    > code to tell me whether it failed or not by the launching application/
    > script.


    As Pontus says, you can use a command in ISim to exit the program
    after it ends the simulation run. But that doesn't tell your program
    why it exited. Is there a simulation result available at the ISim
    command line that could be used as an exit condition for the
    program?

    Rick
     
    rickman, Dec 5, 2010
    #5
  6. On Dec 5, 2:01 pm, rickman <> wrote:
    > On Dec 5, 1:07 pm, Jonathan Ross <>
    > wrote:
    >
    >
    >
    >
    >
    >
    >
    >
    >
    > > On Dec 5, 8:55 am, Nicolas Matringe <> wrote:

    >
    > > > Le 05/12/2010 01:09, Jonathan Ross a écrit :

    >
    > > > [...]

    >
    > > > > 2. The following code generates the following output:

    >
    > > > >      PROCESS
    > > > >      BEGIN
    > > > >          REPORT "Hello World!" SEVERITY NOTE;
    > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
    > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
    > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
    > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
    > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
    > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
    > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
    > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
    > > > >          WAIT;
    > > > >      END PROCESS;

    >
    > > > > ISim>  run
    > > > > Simulator is doing circuit initialization process.
    > > > > at 0 ps: Note: Hello World! (/vhdlunit/).
    > > > > at 0 ps: Note: 1 = 2 (/vhdlunit/).
    > > > > at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
    > > > > at 0 ps: Error: 1 = 2

    >
    > > > > ** Failure:1 = 2
    > > > > User(VHDL) Code Called Simulation Stop
    > > > > In process VHDLUNIT.vhd:10

    >
    > > > > INFO: Simulator is stopped.
    > > > > ISim>

    >
    > > > > The program doesn't exit even on failure, so there's no error code for
    > > > > my harness to use. Also, besides reporting or asserting with severity
    > > > > failure, I can't figure out how to get it to stop. How to I control my
    > > > > return code?

    >
    > > > Hi
    > > > It looks like the simulator DID exit after the failure.
    > > > Try to put something that the simulator would execute if it didn't exit
    > > > and see what happens, instead of exiting (or not) on the last instruction.

    >
    > > > Nicolas

    >
    > > Actually, the simulator is still on. The iSim> prompt means it didn't
    > > exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
    > > it does STOP at the failure. I need it to exit so I can get an exit
    > > code to tell me whether it failed or not by the launching application/
    > > script.

    >
    > As Pontus says, you can use a command in ISim to exit the program
    > after it ends the simulation run.  But that doesn't tell your program
    > why it exited.  Is there a simulation result available at the ISim
    > command line that could be used as an exit condition for the
    > program?
    >
    > Rick


    I'm not quite following - perhaps because I'm not sure what's meant by
    "simulation result." Is there some object that iSim is aware of or can
    be made aware of, or is this terminology colloquial? I'm open to
    making any changes needed - in fact I won't be using ASSERT for this
    (I'm going to follow the xUnit pattern and make my results openly
    available at vhdlunit.org).
     
    Jonathan Ross, Dec 5, 2010
    #6
  7. Jonathan Ross

    rickman Guest

    On Dec 5, 2:57 pm, Jonathan Ross <>
    wrote:
    > On Dec 5, 2:01 pm, rickman <> wrote:
    >
    >
    >
    > > On Dec 5, 1:07 pm, Jonathan Ross <>
    > > wrote:

    >
    > > > On Dec 5, 8:55 am, Nicolas Matringe <> wrote:

    >
    > > > > Le 05/12/2010 01:09, Jonathan Ross a écrit :

    >
    > > > > [...]

    >
    > > > > > 2. The following code generates the following output:

    >
    > > > > >      PROCESS
    > > > > >      BEGIN
    > > > > >          REPORT "Hello World!" SEVERITY NOTE;
    > > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY NOTE;
    > > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY WARNING;
    > > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY ERROR;
    > > > > >          ASSERT 1 = 1 REPORT "1 = 1" SEVERITY FAILURE;
    > > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY NOTE;
    > > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY WARNING;
    > > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY ERROR;
    > > > > >          ASSERT 1 = 2 REPORT "1 = 2" SEVERITY FAILURE;
    > > > > >          WAIT;
    > > > > >      END PROCESS;

    >
    > > > > > ISim>  run
    > > > > > Simulator is doing circuit initialization process.
    > > > > > at 0 ps: Note: Hello World! (/vhdlunit/).
    > > > > > at 0 ps: Note: 1 = 2 (/vhdlunit/).
    > > > > > at 0 ps, Instance /vhdlunit/ : Warning: 1 = 2
    > > > > > at 0 ps: Error: 1 = 2

    >
    > > > > > ** Failure:1 = 2
    > > > > > User(VHDL) Code Called Simulation Stop
    > > > > > In process VHDLUNIT.vhd:10

    >
    > > > > > INFO: Simulator is stopped.
    > > > > > ISim>

    >
    > > > > > The program doesn't exit even on failure, so there's no error code for
    > > > > > my harness to use. Also, besides reporting or asserting with severity
    > > > > > failure, I can't figure out how to get it to stop. How to I control my
    > > > > > return code?

    >
    > > > > Hi
    > > > > It looks like the simulator DID exit after the failure.
    > > > > Try to put something that the simulator would execute if it didn't exit
    > > > > and see what happens, instead of exiting (or not) on the last instruction.

    >
    > > > > Nicolas

    >
    > > > Actually, the simulator is still on. The iSim> prompt means it didn't
    > > > exit to the Xilinx TCL shell, which has a %> or so prompt. That said,
    > > > it does STOP at the failure. I need it to exit so I can get an exit
    > > > code to tell me whether it failed or not by the launching application/
    > > > script.

    >
    > > As Pontus says, you can use a command in ISim to exit the program
    > > after it ends the simulation run.  But that doesn't tell your program
    > > why it exited.  Is there a simulation result available at the ISim
    > > command line that could be used as an exit condition for the
    > > program?

    >
    > > Rick

    >
    > I'm not quite following - perhaps because I'm not sure what's meant by
    > "simulation result." Is there some object that iSim is aware of or can
    > be made aware of, or is this terminology colloquial? I'm open to
    > making any changes needed - in fact I won't be using ASSERT for this
    > (I'm going to follow the xUnit pattern and make my results openly
    > available at vhdlunit.org).


    That's why I'm asking the question, I don't know either. If there is
    a way to use command line functions to determine the result of the
    simulation... and there is a way to make ISim return an exit code on
    exiting, then you can make this work the way I think you want to do
    it. But I don't know if ISim will do either of those things.

    Would it be easier to have ISim save the log to a file and then read
    the log to get a result?

    Rick
     
    rickman, Dec 5, 2010
    #7
  8. Jonathan Ross

    d_s_klein Guest

    On Dec 5, 2:46 pm, rickman <> wrote:
    >
    > Would it be easier to have ISim save the log to a file and then read
    > the log to get a result?
    >
    > Rick


    Do what Rick says.

    Create a log file. At the very least, have the word PASS or FAIL
    appear in the log file.

    Eventually you will want to have something in your log file to tell
    you why and when the failure was detected.

    Hint: Assertions are your friend.

    RK
     
    d_s_klein, Dec 6, 2010
    #8
  9. On 12/6/2010 6:14 PM, d_s_klein wrote:
    > On Dec 5, 2:46 pm, rickman<> wrote:
    >>
    >> Would it be easier to have ISim save the log to a file and then read
    >> the log to get a result?
    >>
    >> Rick

    >
    > Do what Rick says.
    >
    > Create a log file. At the very least, have the word PASS or FAIL
    > appear in the log file.
    >
    > Eventually you will want to have something in your log file to tell
    > you why and when the failure was detected.
    >
    > Hint: Assertions are your friend.
    >
    > RK


    Maybe I haven't quite well understood the topic, but you maybe
    interested in this thread:

    http://www.velocityreviews.com/forums/t57165-how-to-stop-simulation-in-vhdl.html

    One of the first replies from Mike Treseler point to a broken link which
    you may check here: http://mysite.ncnetwork.net/reszotzl/
    It gives a very good approach to testbench and once there are no
    scheduled transitions any longer (through done_s signal) the simulation
    should stop automatically.

    Hope that helps,

    Al
     
    Alessandro Basili, Dec 7, 2010
    #9
  10. Jonathan Ross

    JimLewis Guest

    turn on VHDL-2008 and use:
    std.env.stop(2) ;

    The intent is that the integer parameter is
    returned to the calling program. Not sure
    how the simulators are implementing it, but
    if it does not work like you expect, submit a
    bug report against it.

    Best,
    Jim
     
    JimLewis, Dec 26, 2010
    #10
  11. JimLewis wrote:

    > turn on VHDL-2008 and use:
    > std.env.stop(2) ;
    >
    > The intent is that the integer parameter is
    > returned to the calling program. Not sure
    > how the simulators are implementing it, but
    > if it does not work like you expect, submit a
    > bug report against it.


    In ModelSim it has been implemented as:

    For both STOP and FINISH the STATUS values are those used
    in the Verilog $finish task
    0 prints nothing
    1 prints simulation time and location
    2 prints simulation time, location, and statistics about
    the memory and CPU times used in simulation

    Other STATUS values are interpreted as 0.

    --
    Paul Uiterlinden
    www.aimvalley.nl
    e-mail addres: remove the not.
     
    Paul Uiterlinden, Dec 28, 2010
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alvin Bruney

    Automated Testing software

    Alvin Bruney, Nov 20, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    393
    Kevin Spencer
    Nov 20, 2003
  2. Tim Smith
    Replies:
    0
    Views:
    420
    Tim Smith
    Feb 20, 2004
  3. Rhino
    Replies:
    0
    Views:
    467
    Rhino
    Mar 12, 2005
  4. Quixote
    Replies:
    2
    Views:
    552
  5. quentin compson

    automated testing of model 1 based app

    quentin compson, Jan 23, 2006, in forum: Java
    Replies:
    0
    Views:
    334
    quentin compson
    Jan 23, 2006
Loading...

Share This Page