Is there a way to make testbenches assume simulation failure as default?

Discussion in 'VHDL' started by py, Nov 5, 2012.

  1. py

    py Guest

    Hi,

    I'm trying to make all my testbench such that when the test finished, they would all indicated the simulation failured (assert, severity error or failure), unless the simulation explicitly arrived at the test vector comparison routine (and passed).

    Proof-positive method would allow me to catch cases where the buses hung, or the sim logic is stuck in a loop.

    What VHDL construct in a testbench would accomplish this? I can't think of anything that would kicks in and check a status flag at the end of simulation (since the testbench itself has no knowledge of the user-defined simulation time). Post-processing would work, but I want to know if there is a wayto keep everything inside the testbench.


    Thanks
     
    py, Nov 5, 2012
    #1
    1. Advertising

  2. py

    Rob Gaddi Guest

    Re: Is there a way to make testbenches assume simulation failure asdefault?

    What about making the last line of your testbench report out "*** SIMULATION COMPLETE ***"? If it gets to that line, then it got to that line.

    Or am I missing something?

    -- Rob


    On Mon, 5 Nov 2012 10:34:07 -0800 (PST)
    py <> wrote:

    > Hi,
    >
    > I'm trying to make all my testbench such that when the test finished, they would all indicated the simulation failured (assert, severity error or failure), unless the simulation explicitly arrived at the test vector comparison routine (and passed).
    >
    > Proof-positive method would allow me to catch cases where the buses hung, or the sim logic is stuck in a loop.
    >
    > What VHDL construct in a testbench would accomplish this? I can't think of anything that would kicks in and check a status flag at the end of simulation (since the testbench itself has no knowledge of the user-defined simulation time). Post-processing would work, but I want to know if there is a way to keep everything inside the testbench.
    >
    >
    > Thanks



    --
    Rob Gaddi, Highland Technology -- www.highlandtechnology.com
    Email address domain is currently out of order. See above to fix.
     
    Rob Gaddi, Nov 5, 2012
    #2
    1. Advertising

  3. py

    py Guest

    quick Google Group test, just post a reply and it didn't show up
     
    py, Nov 5, 2012
    #3
  4. py

    py Guest

    Hi Rob,

    A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail.

    Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since it will not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert.

    If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.


    Thanks,


    On Monday, 5 November 2012 11:30:47 UTC-8, Rob Gaddi wrote:
    > What about making the last line of your testbench report out "*** SIMULATION COMPLETE ***"? If it gets to that line, then it got to that line. Or amI missing something? -- Rob On Mon, 5 Nov 2012 10:34:07 -0800 (PST) py <> wrote: > Hi, > > I'm trying to make all my testbench such that when the test finished, they would all indicated the simulation failured (assert, severity error or failure), unless the simulation explicitly arrived at the test vector comparison routine (and passed). > > Proof-positive method would allow me to catch cases where the buses hung, or the sim logic is stuck in a loop. > > What VHDL construct in a testbench would accomplish this? I can't think of anything that would kicks in and check a status flag at the end of simulation (since the testbench itself has no knowledge of the user-defined simulation time). Post-processing would work, but I want to know if there is a way to keep everything inside the testbench. > > > Thanks -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix.
     
    py, Nov 5, 2012
    #4
  5. py

    Andy Guest

    On Monday, November 5, 2012 1:52:50 PM UTC-6, py wrote:
    > Hi Rob, A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail. Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since itwill not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert. If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.


    OK, so your testbench is capable of knowing whether all tests actually ran and passed (some sort of scoreboard)? If that is the case, can you then (and only then) report success (severity note), and shut down all clocks, using event starvation to stop the simulation?

    Event starvation occurs when there are no more events scheduled on any signals for which there is a sensitive process. There are also events from "wait for..." statements and from waveforms using "after...". that are essentially events scheduled at some simulation time in the future.

    Note, some simulators report different conditions (note, warning, error, failure) upon event starvation, so this may not be any different from what you are getting now. Also event starvation can be difficult to cause if you have various clocks created in different parts of the test bench, and no wayto communicate to them to stop the clock(s).

    Andy
     
    Andy, Nov 5, 2012
    #5
  6. py

    Rob Gaddi Guest

    Re: Is there a way to make testbenches assume simulation failure asdefault?

    On Mon, 5 Nov 2012 11:52:50 -0800 (PST)
    py <> wrote:

    > Hi Rob,
    >
    > A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail.
    >
    > Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since it will not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert.
    >
    > If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.
    >
    >
    > Thanks,
    >


    HANGUP_MONITOR: process
    begin
    wait for 1 ms;
    report "Simulation took too damn long, something's broken." severity failure;
    std.env.stop(0)
    end process HANGUP_MONITOR;

    --
    Rob Gaddi, Highland Technology -- www.highlandtechnology.com
    Email address domain is currently out of order. See above to fix.
     
    Rob Gaddi, Nov 6, 2012
    #6
  7. py

    HT-Lab Guest

    Re: Is there a way to make testbenches assume simulation failureas default?

    On 06/11/2012 01:38, Rob Gaddi wrote:
    > On Mon, 5 Nov 2012 11:52:50 -0800 (PST)
    > py <> wrote:
    >
    >> Hi Rob,
    >>
    >> A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail.
    >>
    >> Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since it will not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert.
    >>
    >> If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.
    >>
    >>
    >> Thanks,
    >>

    >
    > HANGUP_MONITOR: process
    > begin
    > wait for 1 ms;
    > report "Simulation took too damn long, something's broken." severity failure;
    > std.env.stop(0)
    > end process HANGUP_MONITOR;
    >


    Hi Py,

    If you are using Modelsim then you can pass this failure to your "outer
    layers" using the "exit -code xx" command and then take appropriate action.

    For example, if you want a good old DOS batch file to detect you have an
    error then you can use the Modelsim "when" command to trigger a
    breakpoint followed by the "exit" command to pass an error code to your
    batch file.

    Here is some pseudo code:

    VHDL source:

    process
    begin
    end_of_sim <= '1';
    end process;

    Modelsim do/tcl file:

    set dosexit 0
    when -label end_of_simulation {end_of_sim == '1'} {stop ; set dosexit 1}
    run -all

    if { $dosexit } {
    puts "Regression Failed"
    exit -code $dosexit
    } else {
    puts "Regression Passed"
    }
    quit -f

    DOS batch file:

    vsim -c xxx_tb -do "source run.do"
    IF %ERRORLEVEL% EQU 0 GOTO NOERROR:
    ECHO "**** VSIM returned an error ***"
    :NOERROR


    Hans
    www.ht-lab.com
     
    HT-Lab, Nov 6, 2012
    #7
  8. py

    rickman Guest

    Re: Is there a way to make testbenches assume simulation failureas default?

    On 11/5/2012 2:52 PM, py wrote:
    > Hi Rob,
    >
    > A bit more context to my original question. The goal here is to make the simulator itself awares of the failure (simualtor logs fatal asserts), so that an outer layer sim environment can track overall testcase pass and fail.
    >
    > Currently, a test passed is expressed as a lack of fatal assert at the end of the sim time. But this methology has a loophole where since it will not pick up tests that actually hung. So I'm looking for a VHDL construct that operates opposite to fatal assert.
    >
    > If such construct doesn't exist nor standardized, then I would have to parse out "SIMULATION COMPLETE" from the sim console log.


    As others indicate, this will depend on your program. I don't know of
    any way to change the default exit from the simulator.

    But you should be able to find a place in your code, a "safety monitor"
    if you will, that "knows" all of the internal tests have passed and
    flags a NOTE about the "good" exit. Absent the "good" exit the
    simulation monitor could assume the simulation was hung.

    To make sure the safety monitor "knows" about the simulation, it would
    require that every important process be designed for communication with
    the safety monitor.

    Rick
     
    rickman, Nov 6, 2012
    #8
  9. py

    KJ Guest

    On Monday, November 5, 2012 2:52:50 PM UTC-5, py wrote:
    > Hi Rob, A bit more context to my original question. The goal
    > here is to make the simulator itself awares of the failure
    > (simualtor logs fatal asserts), so that an outer layer sim
    > environment can track overall testcase pass and fail. Currently,
    > a test passed is expressed as a lack of fatal assert at the end
    > of the sim time. But this methology has a loophole where since
    > it will not pick up tests that actually hung. So I'm looking for
    > a VHDL construct that operates opposite to fatal assert. If such
    > construct doesn't exist nor standardized, then I would have to
    > parse out "SIMULATION COMPLETE" from the sim console log.


    One method is to inspect your testbench and make sure that each wait statement has a timeout. Rather than this...

    wait until (This = That);
    ....continue on

    Instead you do this...

    wait until (This = That) for 1 us;
    assert (This = That) report "Timeout waiting for this to equal that" severity ERROR;

    This will prevent the problem you describe. The downside can be having to know how long is 'too long' to use for the timeout and having to search through the testbench for all of the wait statements and modify them appropriately. The big plus though is that when the bus hangs, the simulator not only stops but the process that is hung has just asserted so you know exactlywhat condition you were waiting for.

    The other approach is to have some form of external monitoring. This can take the form of monitoring activity on a bus. When activity goes away for 'too long' then the simulator asserts. This can be problematic though if there are busses that are naturally idle during certain tests so figuring out an appropriate timeout value can be a problem. It's maybe less upfront work then the previous approach of adding timeouts/asserts to each wait statement but it is also of less value too. All you know when the hang causes an assertion is that the bus stopped, you don't know which process is actually hung up waiting.

    Kevin Jennings
     
    KJ, Nov 7, 2012
    #9
  10. py

    py Guest

    On Monday, 5 November 2012 10:34:08 UTC-8, py wrote:
    > Hi,
    >
    >
    >
    > I'm trying to make all my testbench such that when the test finished, they would all indicated the simulation failured (assert, severity error or failure), unless the simulation explicitly arrived at the test vector comparison routine (and passed).
    >
    >
    >
    > Proof-positive method would allow me to catch cases where the buses hung,or the sim logic is stuck in a loop.
    >
    >
    >
    > What VHDL construct in a testbench would accomplish this? I can't think of anything that would kicks in and check a status flag at the end of simulation (since the testbench itself has no knowledge of the user-defined simulation time). Post-processing would work, but I want to know if there is a way to keep everything inside the testbench.
    >
    >
    >
    >
    >
    > Thanks


    Thank you all for your helpful suggestions. In the end we found a way in our outer sim environment to detect the difference between exit on end of simtime and exit on user-triggered stop. So that served our usage.

    Rob's HANGUP_MONITOR would also work well.
     
    py, Nov 9, 2012
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. BLF
    Replies:
    4
    Views:
    1,282
    Mike Treseler
    Aug 7, 2004
  2. howa
    Replies:
    7
    Views:
    440
    Alf P. Steinbach
    Dec 3, 2006
  3. linq936
    Replies:
    21
    Views:
    698
    Pete Becker
    Sep 19, 2007
  4. Replies:
    0
    Views:
    475
  5. Josh Sharpe
    Replies:
    1
    Views:
    211
    Brian Candler
    Sep 21, 2010
Loading...

Share This Page