ANNOUNCE: Maia 0.8.2: module-level HDL verification tool

Discussion in 'VHDL' started by Evan Lavelle, Apr 21, 2008.

  1. Evan Lavelle

    Evan Lavelle Guest

    Maia is a new tool which automatically creates HDL(*) testbenches from
    a vector-style description. The trivial test case below, for example,
    is a complete testbench for a 4-bit up-counter with reset.

    The tool compiles a test vector file into HDL output, runs the output
    and the module sources on a specified simulator, and reports the
    results, together with details of any failures. You don't need to know
    or write *any* HDL code to use Maia; if you can write vectors for your
    module, then you can verify it.

    You can download a free compiler and documentation from

    (*) Apologies to VHDL'ers; 0.8.2 only creates Verilog output. I hope
    to have VHDL output before too long.

    // ------------------------------------------------------------
    // trivial complete testbench example: 4-bit counter with reset
    DUT {
    module counter(input CLK, RST; output [3:0] Q)
    create_clock CLK // define the clock
    [CLK, RST] -> [Q] // define the test vector format
    main() {
    [.C, 1] -> [0]; // reset
    for(i=1; i <= 16; i++)
    [.C, 0] -> ; // count, with rollover
    Evan Lavelle, Apr 21, 2008
    1. Advertisements

  2. Evan Lavelle

    KJ Guest

    Not to be too much of a slug, but I'm kinda missing the point I think
    of this tool....

    Why would I want to write test vectors (Hint: I don't). Using a
    language with control structures is far more concise and easier to

    If I did have to write test vectors I would probably use something to
    generate those test vectors as an output file artifact from a
    simulation run.

    Having flashbacks to generating test vectors in ABEL...something from
    20 years ago.

    KJ, Apr 21, 2008
    1. Advertisements

  3. Maybe the intended audience is those
    unmotivated to learn vhdl or verilog.
    Sort of a testbench 'wizard' to go with
    wizard generated code.

    Not my cup of tea, but it might get someone started.

    -- Mike Treseler
    Mike Treseler, Apr 21, 2008
  4. Evan Lavelle

    HT-Lab Guest

    If you look at tools like Mentor's inFact they do exactly that although they
    don't create a vector file but an intelligent testbench instead. The inputs
    are high-level rules and actions. These high-level rules are compiled into
    testbench graphs which are then use to generate optimised vector sets.

    For example to test a UART you write different rules for all the different
    options like word size, parity type, number of stop bits, interrupt
    sequence, read/write sequence etc. The compile will then use this info to
    generate an optimised vector set that test all valid combinations (not 100%
    sure on this one). I believe this is one step up from writing assertions and
    feeding your design with constrained random data.

    This is a trivial example but you can understand that with more rules this
    becomes more and more powerful. From what I understand these tools are very
    complex since they not only need to find the smartest set of testvectors but
    also handle all the different constraints and constraint solvers is one area
    which is still heavily being researched.

    I would say kudos to Evan for attempting to develop a product like this and
    making it freely available to the rest of us,


    Having flashbacks to generating test vectors in ABEL...something from
    20 years ago.

    HT-Lab, Apr 21, 2008
  5. Evan Lavelle

    KJ Guest

    There is a huge difference between specifying high level rules and
    specifying vectors as the thing that the test creator needs to input.
    In fact it's hard to imagine a lower level form of rules than
    vectors. As a general rule, most skilled people are likely to be much
    more productive working with a higher level of abstraction than with a
    lower one.
    Sounds like a pitch for the Mentor stuff, not Evan's.
    I agree, I wasn't trying to disparage the effort, just questioning the
    rationale for creating a tool that requires the user to input vector
    sets which is a form of test creation popular 20 years ago.

    KJ, Apr 21, 2008
  6. Evan Lavelle

    Evan Lavelle Guest

    Could be, but I hope not. I've noticed a couple of interesting things
    over the years:

    - Most EEs aren't programmers; electronic design and programming are
    two completely different things. Verification is programming; the
    techniques, tools, and mindset required are different from those
    required to do electronic design. And yet, bizzarely, we generally
    expect the same people to use the same language to do both things. The
    reasons for this are, I think, historical and market-related, and have
    nothing to do with the abilities (or otherwise) of Verilog and VHDL.

    - Most EEs don't actually verify their code, mainly for the reason
    above. I know people who don't verify at all, and others who use
    waveform displays. You can get away with this in the FPGA world
    because it's fairly simple (if time-consuming) to fix things, and in
    the ASIC world because you've got a verification department to find
    the problems (or you've actually got your personal verification
    engineer assigned to you; I've been there).

    Given that lot, the target audience is actually competent EEs who
    don't have the time or inclination to start writing complex programs
    in an inappropriate language to test their RTL code. Imagine you've
    spent a day writing your FSM, or bus interface, or whatever. It gets
    to 5 o'clock, and your choice is either (a) to go home and spend the
    next day writing a testbench for it, when you've already forgotten
    what it does, or (b) spend the next half-hour writing "test vectors"
    (yes, bad phrase) for it. I know which one I'd go for.

    There's at least one other target audience. If you can read a spec for
    someone else's module, then you write vectors for it, even if you know
    absolutely no VHDL or Verilog. So, next time your intern's got nothing
    to do, you can get them doing something constructive without having to
    send them on a training course first.

    Evan Lavelle, Apr 21, 2008
  7. Evan Lavelle

    Evan Lavelle Guest

    I wouldn't get too carried away by the 'test vector' terminology. When
    you get down to it, most TBs involve only 3 activities:

    1 - generate a set of input data

    2 - make sure that the input data gets applied to the DUT at the right

    3 - wait a bit, check that the output data is as expected, repeat

    That's how the RTL works, so it would be difficult to do it any other
    way. I call that lot a 'test vector'. It's convenient to put
    everything in a "[,,] -> [,,]" format, because it's then obvious
    exactly what expressions have to be evaluated and applied to the DUT,
    and what exactly the DUT outputs have to be tested against. It's
    actually a lot more sophisticated than ABEL - all the inputs and
    outputs can be arbitrary expressions, for example, and there are full
    C-like control structures, so you get reactive TBs, and all the rest
    of it. If you'd prefer, they're not actually "test vectors" - they're
    just groups of input and output expressions, but I couldn't think of a
    word for that. The great thing (if I do say so myself :)) is that you
    have to write absolutely no multi-process code to do this. You don't
    need to know any of the TB-related intricacies of VHDL or Verilog to
    do this; it's trivial.

    It's not a magic bullet, as I say on the website. There are basically
    2 reasons that you can't use it for all your everyday verification

    4 - the real smarts in a TB is generating the input data; ie. step 1
    above (I'm ignoring here the additional smarts required for coverage
    stats collection, temporal assertions, and so on). Currently, your
    only options in Maia are essentially table- and algorithm-based. I
    should also have file input before too long. I've already done
    constrained randomisation for another tool, and that should make its
    way in at some point in the future.

    5 - There's currently no way to add additional behavioural code to
    talk to your DUT (a complex bus interface fronting a memory, for
    example). But, again, I've already done this code elsewhere, so it
    should make its way in at some point.

    Evan Lavelle, Apr 21, 2008
  8. That's a good point. I'm always confused when someone asks me if I have
    "test vectors" for a design. No; I have a testbench. The whole
    concept of test vectors doesn't even seem to apply for any design with
    feedback. I'm having flashbacks to testbenches composed of
    simulator-specific "force" commands. -Kevin
    Kevin Neilson, Apr 21, 2008
  9. Evan Lavelle

    Evan Lavelle Guest

    Different concept. This is not a traditional device "test vector";
    it's more of a software "test vector". The phrase "test vector" does
    seem to have pressed all the wrong buttons and I've changed it on the
    website, and added an explanation on the front page.

    In short, these "vectors" may contain arbitrary expressions, and can
    be enclosed in standard control constructs (my first posting actually
    showed a simple example of both). You can therefore write arbitrary
    reactive ("feedback") testbenches; it's nothing like a device test

    However, those are just details. The fundamental idea behind a
    "vector" is something different; it's a step up in abstraction levels.
    The idea is that you don't have to worry about the details of writing
    explicit multi-process timed code; no clock or reset processes, no
    reader and checker processes, no driving and sampling, no pass/fail
    determination, no error reporting, no process communication, and so
    on. It's all wrapped up, automatically, in the "[,,] -> [,,]"
    ("vector") statement.

    As an engineer testing your module, why would you care about any of
    that stuff? It's just irrelevant busywork, because Verilog and VHDL
    aren't smart enough to handle the details for you. I've expanded all
    this on the front page of, under "what is a vector?".

    Evan Lavelle, Apr 22, 2008
  10. Evan Lavelle

    Andy Guest

    Any engineer who can't write simple code in his target language to
    test the RTL he wrote, shouldn't be writing either one.

    Secondly, if the vectors can be written in a half-hour, the RTL design
    code shouldn't have taken all day, unless you're testing to what you
    designed, instead of testing to the requirements. My point is that a
    designer should approach developing the test from the same point of
    view as developing the design. What's the likelihood that a designer
    will have spent all day and forgotten a required feature, and a half-
    hour later will have remembered that same feature when writing the

    Andy, Apr 25, 2008
  11. That may be true, but some get by with trial and error synthesis
    on designs that glue together IP and "known good" modules.
    An fpga guy who is good with logic analyzers and test fixtures
    can debug some fpga designs on the bench. Not my cup of tea,
    but I've seen it done on some successful products.
    For new synthesis code, most of my simulation time is testing
    the design. Working demo units are required to sign off a
    business case, and specifications start out fuzzy.
    Quick turn-around is the key, and a clean, closed loop
    testbench is an advantage.
    I agree. I design and test interactively.
    I can't imagine doing otherwise.

    Once I have a testable procedure with
    inputs and outputs, I add stim procedures
    to the testbench and at least some "printf"
    style verification before moving on to
    the next lump of logic.

    -- Mike Treseler
    Mike Treseler, Apr 26, 2008
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.