[OT] Simulation Results Managment

Discussion in 'Python' started by moogyd@yahoo.co.uk, Jul 14, 2012.

  1. Guest

    Hi,
    This is a general question, loosely related to python since it will be the implementation language.
    I would like some suggestions as to manage simulation results data from my ASIC design.

    For my design,
    - I have a number of simulations testcases (TEST_XX_YY_ZZ), and within each of these test cases we have:
    - a number of properties (P_AA_BB_CC)
    - For each property, the following information is given
    - Property name (P_NAME)
    - Number of times it was checked (within the testcase) N_CHECKED
    - Number of times if failed (within the testcase) N_FAILED
    - A simulation runs a testcase with a set of parameters.
    - Simple example, SLOW_CLOCK, FAST_CLOCK, etc
    - For the design, I will run regression every night (at least), so I will have results from multiple timestamps
    We have < 1000 TESTCASES, and < 1000 PROPERTIES.

    At the moment, I have a script that extracts property information from simulation logfile, and provides single PASS/FAIL and all logfiles stored in a directory structure with timestamps/testnames and other parameters embedded in paths

    I would like to be easily look at (visualize) the data and answer the questions
    - When did this property last fail, and how many times was it checked
    - Is this property checked in this test case.

    Initial question: How to organize the data within python?
    For a single testcase, I could use a dict. Key P_NAME, data in N_CHECKED, N_FAILED
    I then have to store multiple instances of testcase based on date (and simulation parameters.

    Any comments, suggestions?
    Thanks,
    Steven
     
    , Jul 14, 2012
    #1
    1. Advertising

  2. Neal Becker Guest

    wrote:

    > Hi,
    > This is a general question, loosely related to python since it will be the
    > implementation language. I would like some suggestions as to manage simulation
    > results data from my ASIC design.
    >
    > For my design,
    > - I have a number of simulations testcases (TEST_XX_YY_ZZ), and within each of
    > these test cases we have:
    > - a number of properties (P_AA_BB_CC)
    > - For each property, the following information is given
    > - Property name (P_NAME)
    > - Number of times it was checked (within the testcase) N_CHECKED
    > - Number of times if failed (within the testcase) N_FAILED
    > - A simulation runs a testcase with a set of parameters.
    > - Simple example, SLOW_CLOCK, FAST_CLOCK, etc
    > - For the design, I will run regression every night (at least), so I will have
    > results from multiple timestamps We have < 1000 TESTCASES, and < 1000
    > PROPERTIES.
    >
    > At the moment, I have a script that extracts property information from
    > simulation logfile, and provides single PASS/FAIL and all logfiles stored in a
    > directory structure with timestamps/testnames and other parameters embedded in
    > paths
    >
    > I would like to be easily look at (visualize) the data and answer the
    > questions - When did this property last fail, and how many times was it
    > checked - Is this property checked in this test case.
    >
    > Initial question: How to organize the data within python?
    > For a single testcase, I could use a dict. Key P_NAME, data in N_CHECKED,
    > N_FAILED I then have to store multiple instances of testcase based on date
    > (and simulation parameters.
    >
    > Any comments, suggestions?
    > Thanks,
    > Steven


    One small suggestion,
    I used to store test conditions and results in log files, and then write parsers
    to read the results. The formats kept changing (add more conditions/results!)
    and maintenance was a pain.

    Now, in addition to a text log file, I write a file in pickle format containing
    a dict of all test conditions and results. Much more convenient.
     
    Neal Becker, Jul 15, 2012
    #2
    1. Advertising

  3. rusi Guest

    Re: Simulation Results Managment

    On Jul 14, 10:50 am, wrote:
    > Hi,
    > This is a general question, loosely related to python since it will be the implementation language.
    > I would like some suggestions as to manage simulation results data from my ASIC design.
    >
    > For my design,
    > - I have a number of simulations testcases (TEST_XX_YY_ZZ), and within each of these test cases we have:
    >   - a number of properties (P_AA_BB_CC)
    >   - For each property, the following information is given
    >     - Property name (P_NAME)
    >     - Number of times it was checked (within the testcase) N_CHECKED
    >     - Number of times if failed (within the testcase) N_FAILED
    > - A simulation runs a testcase with a set of parameters.
    >   - Simple example, SLOW_CLOCK, FAST_CLOCK, etc
    > - For the design, I will run regression every night (at least), so I willhave results from multiple timestamps
    > We have < 1000 TESTCASES, and < 1000 PROPERTIES.
    >
    > At the moment, I have a script that extracts property information from simulation logfile, and provides single PASS/FAIL and all logfiles stored in a directory structure with timestamps/testnames and other parameters embedded in paths
    >
    > I would like to be easily look at (visualize) the data and answer the questions
    > - When did this property last fail, and how many times was it checked
    > - Is this property checked in this test case.
    >
    > Initial question: How to organize the data within python?
    > For a single testcase, I could use a dict. Key P_NAME, data in N_CHECKED,N_FAILED
    > I then have to store multiple instances of testcase based on date (and simulation parameters.
    >
    > Any comments, suggestions?
    > Thanks,
    > Steven


    Not sure if you are asking about:
    1. Python data structure organization
    or
    2. Organization of data outside python for conveniently getting in and
    out of python

    For 2. if the data is modestly sized and is naturally managed with
    builtin python types -- lists and dictionaries -- yaml gives a nice
    fit. I used pyyaml some years ago, today I guess json which is
    similar, is the way to go.

    For 1, you need to say what are your questions/issues.
     
    rusi, Jul 15, 2012
    #3
  4. Guest

    Re: Simulation Results Managment

    On Sunday, July 15, 2012 5:25:14 AM UTC+2, rusi wrote:
    > On Jul 14, 10:50 am, wrote:
    > &gt; Hi,
    > &gt; This is a general question, loosely related to python since it will be the implementation language.
    > &gt; I would like some suggestions as to manage simulation results data from my ASIC design.
    > &gt;
    > &gt; For my design,
    > &gt; - I have a number of simulations testcases (TEST_XX_YY_ZZ), and within each of these test cases we have:
    > &gt;   - a number of properties (P_AA_BB_CC)
    > &gt;   - For each property, the following information is given
    > &gt;     - Property name (P_NAME)
    > &gt;     - Number of times it was checked (within the testcase) N_CHECKED
    > &gt;     - Number of times if failed (within the testcase) N_FAILED
    > &gt; - A simulation runs a testcase with a set of parameters.
    > &gt;   - Simple example, SLOW_CLOCK, FAST_CLOCK, etc
    > &gt; - For the design, I will run regression every night (at least), so Iwill have results from multiple timestamps
    > &gt; We have &lt; 1000 TESTCASES, and &lt; 1000 PROPERTIES.
    > &gt;
    > &gt; At the moment, I have a script that extracts property information from simulation logfile, and provides single PASS/FAIL and all logfiles stored in a directory structure with timestamps/testnames and other parameters embedded in paths
    > &gt;
    > &gt; I would like to be easily look at (visualize) the data and answer the questions
    > &gt; - When did this property last fail, and how many times was it checked
    > &gt; - Is this property checked in this test case.
    > &gt;
    > &gt; Initial question: How to organize the data within python?
    > &gt; For a single testcase, I could use a dict. Key P_NAME, data in N_CHECKED, N_FAILED
    > &gt; I then have to store multiple instances of testcase based on date (and simulation parameters.
    > &gt;
    > &gt; Any comments, suggestions?
    > &gt; Thanks,
    > &gt; Steven
    >
    > Not sure if you are asking about:
    > 1. Python data structure organization
    > or
    > 2. Organization of data outside python for conveniently getting in and
    > out of python
    >
    > For 2. if the data is modestly sized and is naturally managed with
    > builtin python types -- lists and dictionaries -- yaml gives a nice
    > fit. I used pyyaml some years ago, today I guess json which is
    > similar, is the way to go.
    >
    > For 1, you need to say what are your questions/issues.


    Hi Rusi,

    For (1), I guess that the only question I had was how to handle regression results. But I think that the most logical way for string this data is as adict with key = datestamp, and entries being list of testcases/results.

    For (2), I will look at both these.

    Thanks for the help.

    Steven
     
    , Jul 15, 2012
    #4
  5. Guest

    On Sunday, July 15, 2012 2:42:39 AM UTC+2, Neal Becker wrote:
    > me wrote:
    >
    > &gt; Hi,
    > &gt; This is a general question, loosely related to python since it will be the
    > &gt; implementation language. I would like some suggestions as to manage simulation
    > &gt; results data from my ASIC design.
    > &gt;
    > &gt; For my design,
    > &gt; - I have a number of simulations testcases (TEST_XX_YY_ZZ), and within each of
    > &gt; these test cases we have:
    > &gt; - a number of properties (P_AA_BB_CC)
    > &gt; - For each property, the following information is given
    > &gt; - Property name (P_NAME)
    > &gt; - Number of times it was checked (within the testcase) N_CHECKED
    > &gt; - Number of times if failed (within the testcase) N_FAILED
    > &gt; - A simulation runs a testcase with a set of parameters.
    > &gt; - Simple example, SLOW_CLOCK, FAST_CLOCK, etc
    > &gt; - For the design, I will run regression every night (at least), so I will have
    > &gt; results from multiple timestamps We have &lt; 1000 TESTCASES, and &lt; 1000
    > &gt; PROPERTIES.
    > &gt;
    > &gt; At the moment, I have a script that extracts property information from
    > &gt; simulation logfile, and provides single PASS/FAIL and all logfiles stored in a
    > &gt; directory structure with timestamps/testnames and other parameters embedded in
    > &gt; paths
    > &gt;
    > &gt; I would like to be easily look at (visualize) the data and answer the
    > &gt; questions - When did this property last fail, and how many times was it
    > &gt; checked - Is this property checked in this test case.
    > &gt;
    > &gt; Initial question: How to organize the data within python?
    > &gt; For a single testcase, I could use a dict. Key P_NAME, data in N_CHECKED,
    > &gt; N_FAILED I then have to store multiple instances of testcase based on date
    > &gt; (and simulation parameters.
    > &gt;
    > &gt; Any comments, suggestions?
    > &gt; Thanks,
    > &gt; Steven
    >
    > One small suggestion,
    > I used to store test conditions and results in log files, and then write parsers
    > to read the results. The formats kept changing (add more conditions/results!)
    > and maintenance was a pain.
    >
    > Now, in addition to a text log file, I write a file in pickle format containing
    > a dict of all test conditions and results. Much more convenient.


    Hi Neal,
    We already store the original log files.
    Does pickle have any advantages over json/yaml?
    Thanks,
    Steven
     
    , Jul 15, 2012
    #5
  6. Guest

    On Sunday, July 15, 2012 2:42:39 AM UTC+2, Neal Becker wrote:
    > me wrote:
    >
    > &gt; Hi,
    > &gt; This is a general question, loosely related to python since it will be the
    > &gt; implementation language. I would like some suggestions as to manage simulation
    > &gt; results data from my ASIC design.
    > &gt;
    > &gt; For my design,
    > &gt; - I have a number of simulations testcases (TEST_XX_YY_ZZ), and within each of
    > &gt; these test cases we have:
    > &gt; - a number of properties (P_AA_BB_CC)
    > &gt; - For each property, the following information is given
    > &gt; - Property name (P_NAME)
    > &gt; - Number of times it was checked (within the testcase) N_CHECKED
    > &gt; - Number of times if failed (within the testcase) N_FAILED
    > &gt; - A simulation runs a testcase with a set of parameters.
    > &gt; - Simple example, SLOW_CLOCK, FAST_CLOCK, etc
    > &gt; - For the design, I will run regression every night (at least), so I will have
    > &gt; results from multiple timestamps We have &lt; 1000 TESTCASES, and &lt; 1000
    > &gt; PROPERTIES.
    > &gt;
    > &gt; At the moment, I have a script that extracts property information from
    > &gt; simulation logfile, and provides single PASS/FAIL and all logfiles stored in a
    > &gt; directory structure with timestamps/testnames and other parameters embedded in
    > &gt; paths
    > &gt;
    > &gt; I would like to be easily look at (visualize) the data and answer the
    > &gt; questions - When did this property last fail, and how many times was it
    > &gt; checked - Is this property checked in this test case.
    > &gt;
    > &gt; Initial question: How to organize the data within python?
    > &gt; For a single testcase, I could use a dict. Key P_NAME, data in N_CHECKED,
    > &gt; N_FAILED I then have to store multiple instances of testcase based on date
    > &gt; (and simulation parameters.
    > &gt;
    > &gt; Any comments, suggestions?
    > &gt; Thanks,
    > &gt; Steven
    >
    > One small suggestion,
    > I used to store test conditions and results in log files, and then write parsers
    > to read the results. The formats kept changing (add more conditions/results!)
    > and maintenance was a pain.
    >
    > Now, in addition to a text log file, I write a file in pickle format containing
    > a dict of all test conditions and results. Much more convenient.


    Hi Neal,
    We already store the original log files.
    Does pickle have any advantages over json/yaml?
    Thanks,
    Steven
     
    , Jul 15, 2012
    #6
  7. writes:
    > ...
    > Does pickle have any advantages over json/yaml?


    It can store and retrieve almost any Python object with almost no effort.

    Up to you whether you see it as an advantage to be able to store
    objects rather than (almost) pure data with a rather limited type set.


    Of course, "pickle" is a proprietary Python format. Not so easy to
    decode it with something else than Python. In addition, when
    you store objects, the retrieving application must know the classes
    of those objects -- and its knowledge should not be too different
    from how those classes looked when the objects have been stored.


    I like very much to work with objects (rather than with pure data).
    Therefore, I use "pickle" when I know that the storing and retrieving
    applications all use Python. I use pure (and restricted) data formats
    when non Python applications come into play.
     
    Dieter Maurer, Jul 15, 2012
    #7
  8. Neal Becker Guest

    Dieter Maurer wrote:

    > writes:
    >> ...
    >> Does pickle have any advantages over json/yaml?

    >
    > It can store and retrieve almost any Python object with almost no effort.
    >
    > Up to you whether you see it as an advantage to be able to store
    > objects rather than (almost) pure data with a rather limited type set.
    >
    >
    > Of course, "pickle" is a proprietary Python format. Not so easy to
    > decode it with something else than Python. In addition, when
    > you store objects, the retrieving application must know the classes
    > of those objects -- and its knowledge should not be too different
    > from how those classes looked when the objects have been stored.
    >
    >
    > I like very much to work with objects (rather than with pure data).
    > Therefore, I use "pickle" when I know that the storing and retrieving
    > applications all use Python. I use pure (and restricted) data formats
    > when non Python applications come into play.


    Typically what I want to do is post-process (e.g. plot) results using python
    scripts, so using pickle is great for that.
     
    Neal Becker, Jul 15, 2012
    #8
  9. rusi Guest

    Re: Simulation Results Managment

    On Jul 15, 11:35 am, Dieter Maurer <> wrote:
    > writes:
    > > ...
    > > Does pickle have any advantages over json/yaml?

    >
    > It can store and retrieve almost any Python object with almost no effort.
    >
    > Up to you whether you see it as an advantage to be able to store
    > objects rather than (almost) pure data with a rather limited type set.
    >
    > Of course, "pickle" is a proprietary Python format. Not so easy to
    > decode it with something else than Python. In addition, when
    > you store objects, the retrieving application must know the classes
    > of those objects -- and its knowledge should not be too different
    > from how those classes looked when the objects have been stored.
    >
    > I like very much to work with objects (rather than with pure data).
    > Therefore, I use "pickle" when I know that the storing and retrieving
    > applications all use Python. I use pure (and restricted) data formats
    > when non Python applications come into play.


    Pickle -> JSON -> Yaml
    are roughly in increasing order of human-friendliness and decreasing
    order of machine friendliness (where machine means python 'machine')

    This means that
    - Pickle is most efficient, Yaml least
    - Pickle comes with python from as far back as I know
    Json started coming somewhere round 2.5 (I think)
    (py)yaml needs to be installed separately
    - reading pickled data will spoil your eyes whereas yaml is pleasant
    to read (just like python)
     
    rusi, Jul 15, 2012
    #9
  10. Guest

    Re: Simulation Results Managment

    On Sunday, July 15, 2012 6:20:34 PM UTC+2, rusi wrote:
    > On Jul 15, 11:35 am, Dieter Maurer &lt;&gt; wrote:
    > &gt; writes:
    > &gt; &gt; ...
    > &gt; &gt; Does pickle have any advantages over json/yaml?
    > &gt;
    > &gt; It can store and retrieve almost any Python object with almost no effort.
    > &gt;
    > &gt; Up to you whether you see it as an advantage to be able to store
    > &gt; objects rather than (almost) pure data with a rather limited type set.
    > &gt;
    > &gt; Of course, &quot;pickle&quot; is a proprietary Python format. Not soeasy to
    > &gt; decode it with something else than Python. In addition, when
    > &gt; you store objects, the retrieving application must know the classes
    > &gt; of those objects -- and its knowledge should not be too different
    > &gt; from how those classes looked when the objects have been stored.
    > &gt;
    > &gt; I like very much to work with objects (rather than with pure data).
    > &gt; Therefore, I use &quot;pickle&quot; when I know that the storing andretrieving
    > &gt; applications all use Python. I use pure (and restricted) data formats
    > &gt; when non Python applications come into play.
    >
    > Pickle -&gt; JSON -&gt; Yaml
    > are roughly in increasing order of human-friendliness and decreasing
    > order of machine friendliness (where machine means python 'machine')
    >
    > This means that
    > - Pickle is most efficient, Yaml least
    > - Pickle comes with python from as far back as I know
    > Json started coming somewhere round 2.5 (I think)
    > (py)yaml needs to be installed separately
    > - reading pickled data will spoil your eyes whereas yaml is pleasant
    > to read (just like python)


    Hi Everyone,
    Thanks for the feedback. For now, I store the data using Pickle.
    Steven
     
    , Jul 17, 2012
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Vik

    State managment

    Vik, Feb 3, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    406
    Tommy
    Feb 4, 2004
  2. Mike
    Replies:
    1
    Views:
    345
    Greg Huber
    Apr 14, 2004
  3. Paul Fi
    Replies:
    0
    Views:
    678
    Paul Fi
    Sep 10, 2004
  4. Eirik Eldorsen

    ad managment

    Eirik Eldorsen, Feb 7, 2005, in forum: ASP .Net
    Replies:
    3
    Views:
    418
    Scott Allen
    Feb 7, 2005
  5. jasperng
    Replies:
    0
    Views:
    1,349
    jasperng
    Nov 27, 2008
Loading...

Share This Page