ANNOUNCE: Maia 0.8.2: module-level HDL verification tool

Discussion in 'VHDL' started by Evan Lavelle, Apr 21, 2008.

  1. Evan Lavelle

    Evan Lavelle Guest

    Maia is a new tool which automatically creates HDL(*) testbenches from
    a vector-style description. The trivial test case below, for example,
    is a complete testbench for a 4-bit up-counter with reset.

    The tool compiles a test vector file into HDL output, runs the output
    and the module sources on a specified simulator, and reports the
    results, together with details of any failures. You don't need to know
    or write *any* HDL code to use Maia; if you can write vectors for your
    module, then you can verify it.

    You can download a free compiler and documentation from maia-eda.net.

    (*) Apologies to VHDL'ers; 0.8.2 only creates Verilog output. I hope
    to have VHDL output before too long.

    // ------------------------------------------------------------
    // trivial complete testbench example: 4-bit counter with reset
    DUT {
    module counter(input CLK, RST; output [3:0] Q)
    create_clock CLK // define the clock
    [CLK, RST] -> [Q] // define the test vector format
    }
    main() {
    [.C, 1] -> [0]; // reset
    for(i=1; i <= 16; i++)
    [.C, 0] -> ; // count, with rollover
    }
     
    Evan Lavelle, Apr 21, 2008
    #1
    1. Advertising

  2. Evan Lavelle

    KJ Guest

    On Apr 21, 5:11 am, Evan Lavelle <> wrote:


    > You don't need to know
    > or write *any* HDL code to use Maia; if you can write vectors for your
    > module, then you can verify it.
    >


    Not to be too much of a slug, but I'm kinda missing the point I think
    of this tool....

    Why would I want to write test vectors (Hint: I don't). Using a
    language with control structures is far more concise and easier to
    maintain.

    If I did have to write test vectors I would probably use something to
    generate those test vectors as an output file artifact from a
    simulation run.

    > // trivial complete testbench example: 4-bit counter with reset
    > DUT {
    >   module counter(input CLK, RST; output [3:0] Q)
    >   create_clock CLK         // define the clock
    >   [CLK, RST] -> [Q]        // define the test vector format}
    >
    > main() {
    >   [.C, 1] -> [0];          // reset
    >   for(i=1; i <= 16; i++)
    >     [.C, 0] -> ;        // count, with rollover
    >


    Having flashbacks to generating test vectors in ABEL...something from
    20 years ago.

    KJ
     
    KJ, Apr 21, 2008
    #2
    1. Advertising

  3. > On Apr 21, 5:11 am, Evan Lavelle <> wrote:

    >> You don't need to know
    >> or write *any* HDL code to use Maia; if you can write vectors for your
    >> module, then you can verify it.

    KJ wrote:

    > Not to be too much of a slug, but I'm kinda missing the point I think
    > of this tool....


    Maybe the intended audience is those
    unmotivated to learn vhdl or verilog.
    Sort of a testbench 'wizard' to go with
    wizard generated code.

    Not my cup of tea, but it might get someone started.

    -- Mike Treseler
     
    Mike Treseler, Apr 21, 2008
    #3
  4. Evan Lavelle

    HT-Lab Guest

    "KJ" <> wrote in message
    news:...
    On Apr 21, 5:11 am, Evan Lavelle <> wrote:


    > You don't need to know
    > or write *any* HDL code to use Maia; if you can write vectors for your
    > module, then you can verify it.
    >
    >
    >Not to be too much of a slug, but I'm kinda missing the point I think
    >of this tool....
    >
    >Why would I want to write test vectors (Hint: I don't). Using a
    >language with control structures is far more concise and easier to
    >maintain.
    >
    >If I did have to write test vectors I would probably use something to
    >generate those test vectors as an output file artifact from a
    >simulation run.


    If you look at tools like Mentor's inFact they do exactly that although they
    don't create a vector file but an intelligent testbench instead. The inputs
    are high-level rules and actions. These high-level rules are compiled into
    testbench graphs which are then use to generate optimised vector sets.

    For example to test a UART you write different rules for all the different
    options like word size, parity type, number of stop bits, interrupt
    sequence, read/write sequence etc. The compile will then use this info to
    generate an optimised vector set that test all valid combinations (not 100%
    sure on this one). I believe this is one step up from writing assertions and
    feeding your design with constrained random data.

    This is a trivial example but you can understand that with more rules this
    becomes more and more powerful. From what I understand these tools are very
    complex since they not only need to find the smartest set of testvectors but
    also handle all the different constraints and constraint solvers is one area
    which is still heavily being researched.

    I would say kudos to Evan for attempting to develop a product like this and
    making it freely available to the rest of us,

    Hans
    www.ht-lab.com





    > // trivial complete testbench example: 4-bit counter with reset
    > DUT {
    > module counter(input CLK, RST; output [3:0] Q)
    > create_clock CLK // define the clock
    > [CLK, RST] -> [Q] // define the test vector format}
    >
    > main() {
    > [.C, 1] -> [0]; // reset
    > for(i=1; i <= 16; i++)
    > [.C, 0] -> ; // count, with rollover
    >


    Having flashbacks to generating test vectors in ABEL...something from
    20 years ago.

    KJ
     
    HT-Lab, Apr 21, 2008
    #4
  5. Evan Lavelle

    KJ Guest

    On Apr 21, 12:42 pm, "HT-Lab" <> wrote:
    > "KJ" <> wrote in message
    >
    > >If I did have to write test vectors I would probably use something to
    > >generate those test vectors as an output file artifact from a
    > >simulation run.

    >
    > If you look at tools like Mentor's inFact they do exactly that although they
    > don't create a vector file but an intelligent testbench instead. The inputs
    > are high-level rules and actions. These high-level rules are compiled into
    > testbench graphs which are then use to generate optimised vector sets.
    >


    There is a huge difference between specifying high level rules and
    specifying vectors as the thing that the test creator needs to input.
    In fact it's hard to imagine a lower level form of rules than
    vectors. As a general rule, most skilled people are likely to be much
    more productive working with a higher level of abstraction than with a
    lower one.

    > For example to test a UART you write different rules for all the different
    > options like word size, parity type, number of stop bits, interrupt
    > sequence, read/write sequence etc. The compile will then use this info to
    > generate an optimised vector set that test all valid combinations (not 100%
    > sure on this one). I believe this is one step up from writing assertions and
    > feeding your design with constrained random data.
    >
    > This is a trivial example but you can understand that with more rules this
    > becomes more and more powerful. From what I understand these tools are very
    > complex since they not only need to find the smartest set of testvectors but
    > also handle all the different constraints and constraint solvers is one area
    > which is still heavily being researched.
    >


    Sounds like a pitch for the Mentor stuff, not Evan's.

    > I would say kudos to Evan for attempting to develop a product like this and
    > making it freely available to the rest of us,
    >


    I agree, I wasn't trying to disparage the effort, just questioning the
    rationale for creating a tool that requires the user to input vector
    sets which is a form of test creation popular 20 years ago.

    KJ
     
    KJ, Apr 21, 2008
    #5
  6. Evan Lavelle

    Evan Lavelle Guest

    On Mon, 21 Apr 2008 09:34:32 -0700, Mike Treseler
    <> wrote:

    >Maybe the intended audience is those
    >unmotivated to learn vhdl or verilog.
    >Sort of a testbench 'wizard' to go with
    >wizard generated code.


    Could be, but I hope not. I've noticed a couple of interesting things
    over the years:

    - Most EEs aren't programmers; electronic design and programming are
    two completely different things. Verification is programming; the
    techniques, tools, and mindset required are different from those
    required to do electronic design. And yet, bizzarely, we generally
    expect the same people to use the same language to do both things. The
    reasons for this are, I think, historical and market-related, and have
    nothing to do with the abilities (or otherwise) of Verilog and VHDL.

    - Most EEs don't actually verify their code, mainly for the reason
    above. I know people who don't verify at all, and others who use
    waveform displays. You can get away with this in the FPGA world
    because it's fairly simple (if time-consuming) to fix things, and in
    the ASIC world because you've got a verification department to find
    the problems (or you've actually got your personal verification
    engineer assigned to you; I've been there).

    Given that lot, the target audience is actually competent EEs who
    don't have the time or inclination to start writing complex programs
    in an inappropriate language to test their RTL code. Imagine you've
    spent a day writing your FSM, or bus interface, or whatever. It gets
    to 5 o'clock, and your choice is either (a) to go home and spend the
    next day writing a testbench for it, when you've already forgotten
    what it does, or (b) spend the next half-hour writing "test vectors"
    (yes, bad phrase) for it. I know which one I'd go for.

    There's at least one other target audience. If you can read a spec for
    someone else's module, then you write vectors for it, even if you know
    absolutely no VHDL or Verilog. So, next time your intern's got nothing
    to do, you can get them doing something constructive without having to
    send them on a training course first.

    -Evan
     
    Evan Lavelle, Apr 21, 2008
    #6
  7. Evan Lavelle

    Evan Lavelle Guest

    On Mon, 21 Apr 2008 10:37:52 -0700 (PDT), KJ
    <> wrote:

    >I agree, I wasn't trying to disparage the effort, just questioning the
    >rationale for creating a tool that requires the user to input vector
    >sets which is a form of test creation popular 20 years ago.


    I wouldn't get too carried away by the 'test vector' terminology. When
    you get down to it, most TBs involve only 3 activities:

    1 - generate a set of input data

    2 - make sure that the input data gets applied to the DUT at the right
    time

    3 - wait a bit, check that the output data is as expected, repeat

    That's how the RTL works, so it would be difficult to do it any other
    way. I call that lot a 'test vector'. It's convenient to put
    everything in a "[,,] -> [,,]" format, because it's then obvious
    exactly what expressions have to be evaluated and applied to the DUT,
    and what exactly the DUT outputs have to be tested against. It's
    actually a lot more sophisticated than ABEL - all the inputs and
    outputs can be arbitrary expressions, for example, and there are full
    C-like control structures, so you get reactive TBs, and all the rest
    of it. If you'd prefer, they're not actually "test vectors" - they're
    just groups of input and output expressions, but I couldn't think of a
    word for that. The great thing (if I do say so myself :)) is that you
    have to write absolutely no multi-process code to do this. You don't
    need to know any of the TB-related intricacies of VHDL or Verilog to
    do this; it's trivial.

    It's not a magic bullet, as I say on the website. There are basically
    2 reasons that you can't use it for all your everyday verification
    needs:

    4 - the real smarts in a TB is generating the input data; ie. step 1
    above (I'm ignoring here the additional smarts required for coverage
    stats collection, temporal assertions, and so on). Currently, your
    only options in Maia are essentially table- and algorithm-based. I
    should also have file input before too long. I've already done
    constrained randomisation for another tool, and that should make its
    way in at some point in the future.

    5 - There's currently no way to add additional behavioural code to
    talk to your DUT (a complex bus interface fronting a memory, for
    example). But, again, I've already done this code elsewhere, so it
    should make its way in at some point.

    -Evan
     
    Evan Lavelle, Apr 21, 2008
    #7
  8. KJ wrote:
    >
    > Not to be too much of a slug, but I'm kinda missing the point I think
    > of this tool....
    >
    > Why would I want to write test vectors (Hint: I don't). Using a
    > language with control structures is far more concise and easier to
    > maintain.
    >

    ....
    >
    > Having flashbacks to generating test vectors in ABEL...something from
    > 20 years ago.
    >
    > KJ


    That's a good point. I'm always confused when someone asks me if I have
    "test vectors" for a design. No; I have a testbench. The whole
    concept of test vectors doesn't even seem to apply for any design with
    feedback. I'm having flashbacks to testbenches composed of
    simulator-specific "force" commands. -Kevin
     
    Kevin Neilson, Apr 21, 2008
    #8
  9. Evan Lavelle

    Evan Lavelle Guest

    On Mon, 21 Apr 2008 14:58:26 -0600, Kevin Neilson
    <> wrote:

    >That's a good point. I'm always confused when someone asks me if I have
    > "test vectors" for a design. No; I have a testbench. The whole
    >concept of test vectors doesn't even seem to apply for any design with
    >feedback. I'm having flashbacks to testbenches composed of
    >simulator-specific "force" commands. -Kevin


    Different concept. This is not a traditional device "test vector";
    it's more of a software "test vector". The phrase "test vector" does
    seem to have pressed all the wrong buttons and I've changed it on the
    website, and added an explanation on the front page.

    In short, these "vectors" may contain arbitrary expressions, and can
    be enclosed in standard control constructs (my first posting actually
    showed a simple example of both). You can therefore write arbitrary
    reactive ("feedback") testbenches; it's nothing like a device test
    vector.

    However, those are just details. The fundamental idea behind a
    "vector" is something different; it's a step up in abstraction levels.
    The idea is that you don't have to worry about the details of writing
    explicit multi-process timed code; no clock or reset processes, no
    reader and checker processes, no driving and sampling, no pass/fail
    determination, no error reporting, no process communication, and so
    on. It's all wrapped up, automatically, in the "[,,] -> [,,]"
    ("vector") statement.

    As an engineer testing your module, why would you care about any of
    that stuff? It's just irrelevant busywork, because Verilog and VHDL
    aren't smart enough to handle the details for you. I've expanded all
    this on the front page of maia-eda.net, under "what is a vector?".

    -Evan
     
    Evan Lavelle, Apr 22, 2008
    #9
  10. Evan Lavelle

    Andy Guest

    On Apr 21, 2:35 pm, Evan Lavelle <> wrote:
    > Imagine you've
    > spent a day writing your FSM, or bus interface, or whatever. It gets
    > to 5 o'clock, and your choice is either (a) to go home and spend the
    > next day writing a testbench for it, when you've already forgotten
    > what it does, or (b) spend the next half-hour writing "test vectors"
    > (yes, bad phrase) for it. I know which one I'd go for.


    Any engineer who can't write simple code in his target language to
    test the RTL he wrote, shouldn't be writing either one.

    Secondly, if the vectors can be written in a half-hour, the RTL design
    code shouldn't have taken all day, unless you're testing to what you
    designed, instead of testing to the requirements. My point is that a
    designer should approach developing the test from the same point of
    view as developing the design. What's the likelihood that a designer
    will have spent all day and forgotten a required feature, and a half-
    hour later will have remembered that same feature when writing the
    vectors?

    Andy
     
    Andy, Apr 25, 2008
    #10
  11. Andy wrote:

    > Any engineer who can't write simple code in his target language to
    > test the RTL he wrote, shouldn't be writing either one.


    That may be true, but some get by with trial and error synthesis
    on designs that glue together IP and "known good" modules.
    An fpga guy who is good with logic analyzers and test fixtures
    can debug some fpga designs on the bench. Not my cup of tea,
    but I've seen it done on some successful products.

    > Secondly, if the vectors can be written in a half-hour, the RTL design
    > code shouldn't have taken all day, unless you're testing to what you
    > designed, instead of testing to the requirements.


    For new synthesis code, most of my simulation time is testing
    the design. Working demo units are required to sign off a
    business case, and specifications start out fuzzy.
    Quick turn-around is the key, and a clean, closed loop
    testbench is an advantage.

    > My point is that a
    > designer should approach developing the test from the same point of
    > view as developing the design. What's the likelihood that a designer
    > will have spent all day and forgotten a required feature, and a half-
    > hour later will have remembered that same feature when writing the
    > vectors?


    I agree. I design and test interactively.
    I can't imagine doing otherwise.

    Once I have a testable procedure with
    inputs and outputs, I add stim procedures
    to the testbench and at least some "printf"
    style verification before moving on to
    the next lump of logic.

    -- Mike Treseler
     
    Mike Treseler, Apr 26, 2008
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Symon
    Replies:
    13
    Views:
    831
  2. Topweaver
    Replies:
    0
    Views:
    1,941
    Topweaver
    Mar 22, 2005
  3. Davy
    Replies:
    4
    Views:
    619
  4. pabbu
    Replies:
    8
    Views:
    757
    Marc Boyer
    Nov 7, 2005
  5. Jan Decaluwe

    [Announce] Jan on HDL Design

    Jan Decaluwe, Nov 10, 2009, in forum: VHDL
    Replies:
    4
    Views:
    707
    Jan Decaluwe
    Nov 11, 2009
Loading...

Share This Page