Coding complex VHDL testbenches

[Eli Bendersky]

Hello all,

Which method do you use in your projects to code complex VHDL
testbenches. By complex I mean not "write some, simulate some"
testbenches that are fine for initial development and very simple
models, but rather automatic suites that exercise the design and
report results, reading parameters from and writing results to text
files.

Here are a few options I can think of:

1) We don't use any complex testbenches. All verification is done by
writing some code, running the simulator and watching the waves.

2) We write the testbenches in pure VHDL, including input generation,
result comparison, etc.

3) We write in VHDL and some other language, like Perl, to drive the
simulation, generate inputs and analyze results. If this is the case,
can you elaborate on what techniques you use ?

4) We use a completely non-VHDL system for the testbenches, writing
them in some FLI-tied C++ code, or SystemC.

Thanks in advance
Eli

 

[Andy]

#2, and unless the stimulus is generated by external models (matlab,
hardware in the lab, etc.), I don't use text io either. All reporting
handled via assert & report statements.

Andy

 

[Jim Lewis]

Eli,
We use (and teach) transaction based bus functional models
with directed test cases for algorithmic, numeric tests,
and corner cases. We use procedural based randomization
techniques for generating a constrained random environment.
All of our test environments use self-checking.

One of the missing items in VHDL is class based randomization.
With class base randomization, several items, such as a
transaction can be randomized simultaneously. This gives a
greater capability to provide a uniform randomization
across a transaction.

The procedural based randomization environment that we use in
VHDL, will sometimes require a little more thinking about
how the constraints on each value will interact so as to
ensure that good uniformity of values and hence test cases
is achieved.

Currently the Accellera VHDL TSC is working on adding class
based randomization and the initial proposal of this work
should be done by the end of the week (I am the one working
on it).

Cheers,
Jim
SynthWorks VHDL Training

P.S.

> 2) We write the testbenches in pure VHDL, including input generation,
> result comparison, etc.

Keep in mind that every verification problem is unique.
For some problems, such as video images, it is more effect to do
the results comparison off-line (after simulation completes).

 

[Ajeetha (www.noveldv.com)]

Hi Eli,

Hello all,

Which method do you use in your projects to code complex VHDL
testbenches. By complex I mean not "write some, simulate some"
testbenches that are fine for initial development and very simple
models, but rather automatic suites that exercise the design and
report results, reading parameters from and writing results to text
files.

Here are a few options I can think of:

1) We don't use any complex testbenches. All verification is done by
writing some code, running the simulator and watching the waves.

I wish not, though I do see this at times with few customers and
more so with any open source examples around. It is quite
disheartening to see this.

2) We write the testbenches in pure VHDL, including input generation,
result comparison, etc.

This is certainly doable as Jim also mentioned, though the ROI is
little questionable. Especially the TEXTIO is quite powerful, though
will require lot of coding and fine tuning etc. After all that, the
simulation performance is not as much as what you would like it to be
- especially if you do clock-by-clock file reading.

I usually recommend simulator's memory -read kind of an approach to
initialize a mem array, but enabling such TCL interface also means
reduction in sim performance. Ofcourse these things matter only if you
have long running sims.

3) We write in VHDL and some other language, like Perl, to drive the
simulation, generate inputs and analyze results. If this is the case,
can you elaborate on what techniques you use ?

This I would say a thing of past as of today. (See below)

4) We use a completely non-VHDL system for the testbenches, writing
them in some FLI-tied C++ code, or SystemC.

We do this in SystemVerilog - has classes & OOP for maintainable
TBs, assertions (SVA), coverage, randomization - and SV + VHDL works
in atleast 2 simulators today!

We at CVC have a paper in works to demonstarte all the advanced
Verification capabilities as SystemVerilog provides as applied to VHDL/
Mixed DUT. It will be some work to get it to a publishing shape, but
we can present it to a small audience if there is interest. Contact me
via email if interested.

Regards
Ajeetha, CVC
www.noveldv.com

 

[Jim Lewis]

Ajeetha,

This is certainly doable as Jim also mentioned, though the ROI is
little questionable. Especially the TEXTIO is quite powerful, though
will require lot of coding and fine tuning etc. After all that, the
simulation performance is not as much as what you would like it to be
- especially if you do clock-by-clock file reading.

As you mention, doing a file reading, clock-by-clock testbench is
an effort of slow performance and fine tuning (aka futility), so I
understand why you question ROI, however ...

Transaction based testbenches and randomization are not
limited to the SystemVerilog domain and are currently being used
in VHDL testbenches quite effectively. As I mentioned in my
previous post, for now randomization is a procedural rather than
class based, however, this will change as VHDL is enhanced.

> We do this in SystemVerilog - has classes & OOP for maintainable
> TBs, assertions (SVA), coverage, randomization -

Assertions (via PSL) are in the current version of VHDL (Accellera 3.0).

Classes/OOP, randomization (class based and additional language constructs),
and coverage is currently being worked on in committee.

The rationale for SystemVerilog in the first place was so that
the design and verification engineers could use the same language.
From a VHDL perspective, SystemVerilog is a foreign language and
VHDL design and verification engineers will benefit more from
having a verification constructs in VHDL.

Cheers,
Jim

 

[Eli Bendersky]

Eli,
We use (and teach) transaction based bus functional models
with directed test cases for algorithmic, numeric tests,
and corner cases. We use procedural based randomization
techniques for generating a constrained random environment.
All of our test environments use self-checking.

Are your environments pure VHDL, or do you involve some third party
languages / tools to help with generation / randomization / self-
checking ?

Also, can you please point to a good resource on transaction based
BFMs online ?

Thanks
Eli

 

[Jim Lewis]

Eli,

Are your environments pure VHDL, or do you involve some third party
languages / tools to help with generation / randomization / self-
checking ?

Pure VHDL. It is a matter of managing the layering. See the paper
referenced below.

To do self-checking with randomization, you will need some type
of a scoreboard. This can be done either in an entity or a
protected type (which is a little like a class without extensibility).
For randomization I use the function is ieee.math_real.uniform, but
embed it also in a protected type to encapsulate the seed values with
the randomization.

Also, can you please point to a good resource on transaction based
BFMs online ?

While its focus is not on transaction based BFMs, towards the end
of the following paper there is good example of a transaction based BFM:
"Accelerating Verification Through Pre-Use of System-Level Testbench Components"

It is posted at: http://www.synthworks.com/papers/index.htm
It is down the page a little under "DesignCon 2003".

Cheers,
Jim

 

[Ajeetha (www.noveldv.com)]

Jim,

Ajeetha,


Transaction based testbenches and randomization are not
limited to the SystemVerilog domain and are currently being used
in VHDL testbenches quite effectively. As I mentioned in my
previous post, for now randomization is a procedural rather than
class based, however, this will change as VHDL is enhanced.

Agreed, but Jim - from a practical view, it will be few years before
this even becomes reality as the standard itself is being designed,
then we pray to our EDA Vendors to implement it etc. Whereas SV is
available today. I don't intend to trigger a SV vs. VHDL war here, I
will (hopefully) make more business if there is VHDL-classes etc. So
I'm all for it.

Assertions (via PSL) are in the current version of VHDL (Accellera 3.0).

Oh, sure. Sorry I didn't mention that. Just FYI - I'm owning the
task of delivering OVL-PSL-VHDL version to Accellera so I'm bullish on
that too. I'm promoting both PSL and SVA equally here locally as well.

Classes/OOP, randomization (class based and additional language constructs),
and coverage is currently being worked on in committee.

The rationale for SystemVerilog in the first place was so that
the design and verification engineers could use the same language.
From a VHDL perspective, SystemVerilog is a foreign language and
VHDL design and verification engineers will benefit more from
having a verification constructs in VHDL.

True, but only if they can afford to wait for another couple of
years (maybe 201X?). Again - please don't take my comments in -ve
sense, I really applaud the contributions you are doing for the
standards. Even with a thing like OVL-PSL I'm finding it hard to
commit extra time away from consulting, so sure your efforts are
great. I wish there is enough EDA backing to make this successful.

Regards
Ajeetha, CVC
www.noveldv.com

 

[Jim Lewis]

Ajeetha,

Agreed, but Jim - from a practical view, it will be few years before
this even becomes reality as the standard itself is being designed,
then we pray to our EDA Vendors to implement it etc. Whereas SV is
available today. I don't intend to trigger a SV vs. VHDL war here, I
will (hopefully) make more business if there is VHDL-classes etc. So
I'm all for it.

We will remove the need to war as we will copy the good stuff from
SV. As a result, I reckon that it will be similar enough to SV that
it will not be too much trouble to implement it - especially after
they have already implemented SV.

True, but only if they can afford to wait for another couple of
years (maybe 201X?). Again - please don't take my comments in -ve
sense, I really applaud the contributions you are doing for the
standards. Even with a thing like OVL-PSL I'm finding it hard to
commit extra time away from consulting, so sure your efforts are
great. I wish there is enough EDA backing to make this successful.

EDA vendors generally do what their users ask them to do.
If people ask for VHDL, then the vendors will do it.
I would recommend to start asking sooner (like now) rather than
wait.

Beyond that, the language already has the ground work for classes
in the form of protected types. So integrating verification features
into VHDL should take somewhat less than what they did for SystemVerilog.

BTW, using just protected types one can already do a pretty good
job at procedural randomization. Granted it takes a little more
creativity than it will once we build the constructs directly into
the language - but none the less it works well and gets the job done.

It will be an interesting next couple of years. Since these are
verification features, if one simulation vendor were to do an early
implementation of the VHDL verification features, they could easily
get a leg up over the others and as a result capture the VHDL market.

Cheers,
Jim

 

[chriskoh]

Hi guys,
couldn't resist but to add in into the fun.

3) We write in VHDL and some other language, like Perl, to drive the
simulation, generate inputs and analyze results. If this is the case,
can you elaborate on what techniques you use ?


4) We use a completely non-VHDL system for the testbenches, writing
them in some FLI-tied C++ code, or SystemC.


point (3) is the one i would like to elaborate on. one more popular
verification language. they call it the E language, and it is used
together with a tool called specman. Typically a verification guy
would code the testbench in VHDL and then use E to generate
randomization in the stimulus as well as another portion to do
scoreboarding.
during runtime, the simulator (ie..modelsim, or nc-affirma) links
to specman via this pli thing. E and specman is currently used in
house aggresively to verify chips from rtl flow to chip level
simulations.
E by the way is the so-called competitor to system verilog. I will
not go into that though. I guess it will spawn another thread
altogether. I think system verilog is very nicely packaged though, as
you do not need to completely relearn a new language and learn how to
use a new co-simulator tool to achieve your end. just using one
simulator like modelsim for instance which supports system verilog
already is sufficient. all you have to do is to merely brush up on
your sv language.
perl is also good to use specially in regression test cases, and
if you have many licenses. Just run, use perl to generate a few
testbenches at a go, compile, and call a few simulators to run
multiple test cases at a go. After that, just grep all the data from
the logfiles and join them up to get a nice neat report.
We use a combination of E and perl and VHDL to achieve our ends.

Chris

 

[Jim Lewis]

Chris,
Any compelling capability that you like/use in E/perl
that you do not see in SystemVerilog?

Not looking for a flame fest. Rather, I do alot of work in
the VHDL standards community and we are currently extending
VHDL to add OO, randomization, and coverage capabilities.
Just want to make sure we don't miss things.

You can answer me privately if you wish.

Cheers,
Jim

 

[Eli Bendersky]

Chris,
Any compelling capability that you like/use in E/perl
that you do not see in SystemVerilog?

Not looking for a flame fest. Rather, I do alot of work in
the VHDL standards community and we are currently extending
VHDL to add OO, randomization, and coverage capabilities.
Just want to make sure we don't miss things.

You can answer me privately if you wish.

Cheers,
Jim

Jim,

A layman's question - when you say "randomization" in the context of
testbench coding, what exactly do you mean ? Advanced abilities to
generate various random stimuli ?
Is this something that can't be coded as a set of VHDL functions ?

Thanks

 

[chriskoh]

Chris,
Any compelling capability that you like/use in E/perl
that you do not see in SystemVerilog?

Not looking for a flame fest. Rather, I do alot of work in
the VHDL standards community and we are currently extending
VHDL to add OO, randomization, and coverage capabilities.
Just want to make sure we don't miss things.

You can answer me privately if you wish.

Cheers,
Jim








- Show quoted text -

Hi Jim,
honestly, no reason why we have not used system verilog. the
projects here are running full steam on E at this point. we have
started looking at system verilog here but not switching yet at this
point. alot of legacy issues, like testbenches being coded already in
e...using developed eVCs etc from other e users or getting licensed
eVCs. not re-inventing the wheel unless the absolute advantage yet. I
am really no historian here, but E was introduced here first seemingly
before system verilog.
personally, I dun like VHDL cos of the compilation order thingy.
also, if VHDL can implement 00, randomization it would be interesting.
but for coverage, i think simulators like modelsim can easily do code
coverage...not quite sure what you mean here.

Chris

 

[chriskoh]

Jim,

A layman's question - when you say "randomization" in the context of
testbench coding, what exactly do you mean ? Advanced abilities to
generate various random stimuli ?
Is this something that can't be coded as a set of VHDL functions ?

Thanks

Hi Eli,
randomization.... refers to something like if you were testing a
router (the standard specman example). you would like your packets
to have the relevant headers and packet structure, yet randomise its
generation when you are generating many many packets right? VHDL is
kinda weak in this..how you do that in VHDL if there is any. it is a
so called advanced ability to generate various random stimuli.
using specman, we can randomise variables in their generation and
yet also set constraints for the randomization. the constraints are
used in our example, when you specify the kind of packet headers
(ie..what kind of modes allowed) etc...

Chris

 

[Eli Bendersky]

randomization.... refers to something like if you were testing a
router (the standard specman example). you would like your packets
to have the relevant headers and packet structure, yet randomise its
generation when you are generating many many packets right? VHDL is
kinda weak in this..how you do that in VHDL if there is any. it is a
so called advanced ability to generate various random stimuli.
using specman, we can randomise variables in their generation and
yet also set constraints for the randomization. the constraints are
used in our example, when you specify the kind of packet headers
(ie..what kind of modes allowed) etc...

Chris

The kind of randomization capabilities you describe here appear
trivial to implement in VHDL. Everything stems from the basic uniform
PRNG, and weighted choice from an array can be easily implemented.

 

[chriskoh]

The kind of randomization capabilities you describe here appear
trivial to implement in VHDL. Everything stems from the basic uniform
PRNG, and weighted choice from an array can be easily implemented.

Hi Eli,
you could spend time doing that. A PRNG with weighted choice from
array is possible. no one is stopping you from doing that. When you
are using E or specman, u do not need the additional effort to do
that.
in the industry, time is $$. time to market is everything. If you
have a tool that saves say some engineer man hours, there will surely
be a customer. We need little tools to do good design. surely, but
with more tools, you can do bigger designs faster.
It just gets worse when your design becomes really huge and you are
spending time building PRNGs to randomise your testing behaviour.


Chris

 

[Pieter Hulshoff]

Which method do you use in your projects to code complex VHDL
testbenches. By complex I mean not "write some, simulate some"
testbenches that are fine for initial development and very simple
models, but rather automatic suites that exercise the design and
report results, reading parameters from and writing results to text
files.

2) We write the testbenches in pure VHDL, including input generation,
result comparison, etc.

We use method 2. Over the years we have written an extensive set of VHDL
libraries with driver/monitors for the verification of our designs. We can
create stimuli for these driver/monitors through command files or via VHDL
commands. The monitors will generate errors in a log file, and expected errors
can be flagged as such to prevent them from being reported as an error. At the
end of the simulation, a message will be printed in the log file if the test was
successful and if not how many errors were detected.

Regards,

Pieter Hulshoff

 

[Marcus Harnisch]

The kind of randomization capabilities you describe here appear
trivial to implement in VHDL. Everything stems from the basic uniform
PRNG, and weighted choice from an array can be easily implemented.

Chris has already pointed out that it is much easier to use existing
language features than to roll your own.

To add to that, I am afraid that there is a potential problem with the
reliability of a home grown solution. Especially when it comes to
combining many constraints, sorting out conflicts may not be easy and
is certainly not optimized for simulation speed. Even if you managed
to work around that, debugging those conflicts will be a pain. Specman
has quite useful constraint debugging capabilities and SystemVerilog
will get there one day, too.

Dynamically turning on/off (SV) and overwriting/resetting (/e/)
specific constraints is another feature provided by HVLs.

Yes you could most of all that, of course. But you will end up
polluting your actual verification code with all these different
functions that are specific to your enviroment (more likely: just
you!), thereby obstructing the original intention of the code. Let's
keep in mind D.E. Knuth's recommendation (from his book Literate
Programming):

"Let us change our traditional attitude to the construction of
programs: Instead of imagining that our main task is to instruct a
computer what to do, let us concentrate rather on explaining to
human beings what we want a computer to do."

It is much more concise to write something like

,----
| class C;
| rand int a, b;
|
| constraint a_less_than_b {
| a < b;
| }
| endclass
`----

than hand coding the randomization mechanism in an HDL and using an
obscure functional (as opposed to declarative) approach to
constraints.

Last not least for the money needed to dedicate engineers to implement
all this from scratch, verify it and document it properly you'd easily
get a few licenses of your favorite HVL.

Regards,
Marcus

 

[Jim Lewis]

Chris,

you could spend time doing that. A PRNG with weighted choice from
array is possible. no one is stopping you from doing that. When you
are using E or specman, u do not need the additional effort to do
that.

I agree, it would be easier having syntax built-in, however, it is
not as difficult as one would think without it. The following
are a few I use (taken from our training class examples):

variable RV : RandomPType ;
type slva is array (integer range <>) of std_logic_vector (3 downto 0);
constant VAL : slva(0 to 3) := ("0001","1001","1011","0011");
variable RanVal : std_logic_vector(3 downto 0) ;
.. . .

-- Uniform randomization of SLV values within a range (specified by integers)
RanVal := RV.RandSlv(7, 288) ;

-- Uniform randomization, General case with a sparse set of values.
RanVal := VAL( RV.RandInt(VAL'low, VAL'high) );

-- Weighted randomization, General case with a sparse set of values.
-- Weights done positionally relative to array indicies
-- value: 0001 weight 1, 1001 weight 1, 1011 weight 4, 0011 weight 4
RanVal := VAL( RV.RandDist( (1, 1, 4, 4) );

With weights, relative order of occurrence = weight / (sum of all weights).
As a result, 0001 occurs 10 % of the time, 1001 occurs 10%,
1011 occurs 40% and 0011 occurs 40%


My opinion is that some of this is more work that it needs to be and
we should enhance VHDL language to do this. If you agree, please
reply to my separate post: "RFC: Enhancing VHDL for OO, Randomization,
Functional Coverage".

Cheers,
Jim

 

[Jim Lewis]

Marcus,

> It is much more concise to write something like
>
> ,----
> | class C;
> | rand int a, b;
> |
> | constraint a_less_than_b {
> | a < b;
> | }
> | endclass
> `----
>
> than hand coding the randomization mechanism in an HDL and using an
> obscure functional (as opposed to declarative) approach to
> constraints.

I have read much of the literature and agree that it is important to
have this functionality, however, it is not that impossible to cope
without it:

variable RV : RandomPType ;
variable A, B : integer ;
.. . .

A := RV.RandInt(0, 254) ; -- Min, Max, Length
B := RV.RandInt(A+1, 255) ;

The limitation to this case is the distribution of values is not
uniform. The value(A=254, B=255) happens 1 in 255 times where as
the value (A=0, B=255) happens 1 in (255*255) times. As a result,
adjustments would have to be made to some problems (however many
are fine just as they are).

With class based randomization (Marcus's example), the intent is
that the solver gives a uniform distribution across possible values.
Hence, (254, 255) occurs with the same frequency as (0,255).
While this is nice in theory, actual solver behavior will vary as
demonstrated in Chris Spear's book, SystemVerilog for Verification,
example 6-18 and results in table 6-3 - ironically when I reported
this as eratta, it was explained to me that this is valid behavior
of the solver and the user had a choice of the fast solver or the
slow and accurate solver.

To add to that, I am afraid that there is a potential problem with the
reliability of a home grown solution. Especially when it comes to
combining many constraints, sorting out conflicts may not be easy and
is certainly not optimized for simulation speed. Even if you managed
to work around that, debugging those conflicts will be a pain. Specman
has quite useful constraint debugging capabilities and SystemVerilog
will get there one day, too.

Dynamically turning on/off (SV) and overwriting/resetting (/e/)
specific constraints is another feature provided by HVLs.

Sounds like this is a general problem that also occurs with class based
randomization. You can specify some constraints in the class, then
revise them by extending the class, then revise them in the call to
randomize as well as mix code in with this. So what does the test
actually do? It is specified by the class + extensions + code.
Of course, you do have functional coverage to answer this.
Is the test readable and maintainable and by whom? Better have a
good methodology - I suppose this is a universal truth.

With VHDL in its current state, I would not try to write declarative
constraints. Instead I write them procedurally - in a process or
subprogram. For the time being, this has done a good enough job for
many problems.


I see the importance of class based randomization. I think VHDL
needs to be enhanced to do this. If you agree, please reply to
my separate post: "RFC: Enhancing VHDL for OO, Randomization,
Functional Coverage".

Cheers,
Jim