Standardized Functional Verification- P15 docx

5.3 Factors to consider in using Convergence 125 Fig. 5.2. Stratification of convergence Another factor to consider is which RTL-based coverage measure to use in computing the test generator’s power using the convergence technique. The greater the granularity in the measure (such as using expression coverage instead of line coverage), the better indicator the convergence is likely to be of the test generator’s power. Investigating the empirical results for a given project or multiple projects will provide insight in the best way to estimate power with convergence. Fig. 5.3. Determining convergence of test generator against a target 126 Chapter 5 – Normalizing Data Another effect might require some adjustments in how one calculates convergence: an initial step function in coverage from running a handful of tests. In such a case, one might consider the level achieved by the initial step function as the baseline from which the exponential growth curve rises. 5.4 Complexity of a Target Complexity Z is proportional to the amount of logic to be exercised in the target. Many different measures have been proposed and used to model the complexity of computer code (such as lines of uncommented code), but for the purposes of functional verification of an IC, it’s more useful to consider a more reasonable measure: a particular count of the number of gates in the target. A common practice in the IC design community is the division of logic into datapath logic and control logic. Datapath logic usually consists of uniform arrays of cells, such as bits in a register file, slices in an adder, and so on. The remaining logic is regarded as control logic (because it, of course, exerts control over the datapath, telling it what to do). It’s widely known that the vast majority of bugs in a design are associated with the control logic, rather than the datapath logic. It stands to reason (hence, our reasonable measure) that some function of the number of gates of control logic has something to do with the complexity (and, therefore, the potential buggy-ness) of a target. Consider a network of 2 2-input NAND gates. Each gate could connect to the input of either gate, for a total of 2 possible connection destinations for each of the two gates. A bug could dwell in any of 4 possible miscon- nections. With 3 such 2-input NAND gates, we have 9 possible miscon- nections. Reasoning along these lines (inductively speaking) we can state that from the standpoint of finding bugs, an estimate for the complexity of our target is the square of the number of 2-input NAND gates required to implement the control logic. This analysis for an estimate of complexity is analogous to that of a general machine made of N parts. Considering all possible interactions among parts, the resulting complexity is O(N 2 ). We can refine this estimate by recognizing that within our control logic are other regular logical structures that we can choose (usefully and with sound reason) to regard as free of bugs. These regular logical structures are readily recognized as the multiplexors and adders and other MSI-like (medium scale integration) elements. Memory elements (flip- flops) are also excluded from control logic because they don’t participate 5.4 Complexity of a Target 127 in the connectivity of 2-input NAND gates (they are simply inserted be- tween the output of a NAND gate and the inputs of the gate or gates to which it is connected). One further refinement to this estimate is to acknowledge that verifying 2 exact copies of a given body of logic is no more work than verifying the logic on its own. That is, multiple instances (code instances, not instances of connectivity) of a single module do not contribute directly to the complexity of the target. However, the control logic connected to the 2 instances does contribute to the complexity. Synthesizing our target’s RTL using a library of only 2-input NAND (or NOR) gates and MSI-like elements and excluding multiple instances of any module, and then tallying up the number of resulting 2-input NAND gates. This number C of control gates can be used to compute a reasonable estimate for the target’s complexity as Z C ∝ C 2 . (5.1) The subscript distinguishes this estimate from other estimates for complexity. This estimate Z C is based on control gates, but other estimates are possible. One of these other estimates will be discussed later in this chapter. We can then estimate the complexity Z G for each clock domain by using a commercial synthesizer with a particular library of components (datapath elements and 2-input NANDs only) and excluding repeated sub-networks of multiple instances of any module instantiated more than once. Timing is unconstrained because we only want to obtain a logical mapping of RTL to logic elements. Filtering the output of this special synthesis run to exclude the “zero-risk” datapath elements and to count up remaining 2-input NAND gates in each clock domain. Squaring each sum gives the estimated (and unit-less) measure for complexity of each clock domain. 1 1 A whitepaper from Design and Reuse arrives at a similar conclusion, stating that “as designs double in size, the verification effort can easily quadruple.” 128 Chapter 5 – Normalizing Data Fig. 5.4. Complexity rises rapidly with gate count Estimating Z by analyzing control logic will be found to correlate strongly with the count of arcs in the target’s VTG. The size |E(G)| of the edge set of the VTG is defined as this count of arcs. More specifically, the count of arcs in the condensed functional space is likely to correlate strongly with Z C as computed from the control logic. It is the control logic, after all, that steers the action along arcs from point to point in the functional space. Mathematically we can state that Z V ∝ E(G). (5.2) The subscript V indicates that this estimate for complexity is based on the VTG rather than on control gates using the synthesis method. An estimate for complexity based on knowing values of standard vari- ables means that we can make this estimate soon after conducting interpre- tation of the specifications rather than waiting until synthesis results are available from RTL at FC done (see Fig. 5.5). 5.5 Scaling Regression using Convergence 129 Fig. 5.5. Getting an early estimate for complexity Z This estimate Z, whether based on control gates or on VTG arcs, can be a useful tool for planning and managing the resources needed to verify an RTL database for tape-out. In particular, by normalizing other measures with respect to Z, we can begin to compare results from differing targets. This is fundamental to the risk assessment that will be discussed in chapter 6. Just as code coverage is a normalized quantity (expressed as a percent- age of the lines of code, etc.), functional coverage can be normalized, too. 5.5 Scaling Regression using Convergence How “big” must a regression suite be to be regarded as thorough? And, for that matter, what does it mean to have a “random regression suite”? A regression suite that thoroughly exercises a target in a pseudo-random manner can be said to bombard the target with novelty - new sequences of stimuli under various operational conditions and in various contexts, each different from the other - i.e., novelty (or “newness”). A target that can survive overwhelming novelty is regarded as free from bugs (at a given level of risk) and is protected from the introduction of bugs during en- hancement by the same overwhelming novelty. Define N R as the number of clock cycles needed for a thorough regression of the target. In general N R will be found to be proportional to the complexity of the target: 130 Chapter 5 – Normalizing Data N R ∝ Z . (5.3) This number of clock cycles is proportional not only to complexity of the target as indicated by examining control logic, but also as indicated by examining the VTG corresponding to the target. After all, during a single clock cycle the target is performing a single computation–that of determining the next point that will be reached in traversing the graph. Eq. 3.5 (repeated here) expresses mathematically what happens within the target: r (n +1) = f ( r (n),e(n),s(n),c(n),a(n),k(n)) (3.5) It is the control logic that determines which of the many arcs departing from a function point at time n will be traversed to the function point at time n+1. It should be no surprise that the number of arcs in the condensed VTG correlates strongly with complexity as determined by analyzing the control logic. The value for N R is determined empirically, unless one already has results from prior projects, in which case that project’s results can be used to forecast requirements for the current project. Consider two projects: prior project X and current project Y. Project X went well, producing an IC with few bugs at first silicon. Project Y is to produce an even larger IC. How much pseudo-random testing is needed to bring the risk at tape-out to the same level as for project X? The complexity of the target in project X at its completion was Z X and at FC (final coding) the complexity of the target in project Y was Z Y . The manager had observed during project X that the complexity at tape-out was within 10% of the value at FC and that the complexity was always within 10% of the FC value so it was regarded as a good estimate of the final complexity. The test generator in project X had good convergence but at FC in project Y the convergence is not yet as good as for project X. However, project Y appeared to have the same convergence gap as project X, i.e. α Y = α X . A graph of the convergence for these 2 projects is illustrated in Fig. 5.6. 5.5 Scaling Regression using Convergence 131 Fig. 5.6. Forecasting project Y from project Y To achieve the same level of risk at tape-out for project Y, the relative complexity of the 2 targets and the relative power of the test generator on each target are used to determine the number of cycles that target Y must survive without any faulty behavior. Both the complexity of a target and the power of the test generator contribute to the resulting convergence of the generator on the target. So, how must the number of cycles be scaled to achieve the same level of risk? Determining this scaling factor requires a bit of math but the result is the pleasingly simple ratio of beta cycles for each target. In general, the coverage achieved by running N cycles of tests is: coverage = (1 − α )(1 − e − β N ). (5.4) The variabe is simply the scaling factor that determines the rise time of the curve. β 132 Chapter 5 – Normalizing Data (1 − α )⋅ 2e −1 = (1 − α )⋅ (1 − e − β N β ) 2e −1 =1− e − β N β e − β N β =1− 2e −1 − β N β = log(1− 2e −1 ) β =− log(1− 2e −1 ) N β (5.5) We have derived this general closed-form expression for β so that we can use it below. Because the coverage at the beta cycles for each target is the same, we can find the value of beta cycles for Y from the value for X as follows: coverage( N R Y ) = coverage( N R X ) (1 − α )(1− e − β Y N R Y ) = (1 − α )(1− e − β X N R X ) β Y N R Y = β X N R X N R Y = β X β Y N R X = −log(1− 2e −1 ) N β X −log(1− 2e −1 ) N β Y N R X = N β Y N β X N R X (5.6) So, the scaling of regression cycles for Y is straightforward: N R Y = N β Y N β x N R X (5.7) After N β cycles the coverage will be (1 − α ) ⋅ 2e − 1 so 5.6 Normalizing Cycles Counts with Complexity 133 5.6 Normalizing Cycle Counts with Complexity By scaling the number of regression cycles by the complexity of the target we can compare cycle counts needed for regression of different targets. This normalized cycle count Q R is defined as: Q R = N R Z (5.8) This constant Q R based on empirical data is the quantity of pseudo- random regression that takes into account both the power of the test generator and the complexity of the target. Of course, the same estimate of complexity must be used for generating normalized values to be used to- gether. This normalized measure of clock cycles can be used to compare di- verse projects, and then to allocate resources according to the complexity of the targets and the power of the test generator with respect to the target. An organization that has accumulated data over a number of projects K can compute a mean value for Q R as the geometric mean of individual values for Q R : Q R = ( Q R i i = 1 K ∏ ) 1 K (5.9) A design organization with an empirically determined Q R can then forecast the amount of regression N for some new design by estimating its complexity Z and running sufficient pseudo-random simulations to estimate Ν β . Then a suitable estimate for the number of regression cycles for this new design is simply. N R = Q R ⋅ Z. (5.10) When managing development of multiple devices, allocating resources suitably across projects can be made more effective by understanding the value of Q for your organization. Similarly, as the organization makes changes (tools used, staffing levels, etc.) the effectiveness of these changes is reflected in a change in Q (lower is better). 134 Chapter 5 – Normalizing Data But, what if the convergence gap α is not the same from project to project? The foregoing analysis would be complicated by differing convergence gaps, but in practice this gap can be disregarded in either of 2 situations: • make the convergence gap as near to zero as possible with suitable adjustments and modifications to the test generator, or • determine that uncovered logic in the target is redundant or unreachable and simply disregard it in computing coverage. 5.7 Using Normalized Cycles in Risk Assessment Q can be a useful measure to determine the risk of a bug in the device after tape-out. Consider the example of 10 tape-outs of a number of different devices. Plotting the number of bugs exposed in the device against the Q for regression prior to tape-out can enable the manager to establish the resources needed to achieve a sufficiently low level of risk (bugs in the device). Fig. 5.7. Post tape-out bugs as a function of Q . the complexity of computer code (such as lines of uncommented code), but for the purposes of functional verification of an IC, it’s more useful to consider a more reasonable measure: a particular. the VTG is defined as this count of arcs. More specifically, the count of arcs in the condensed functional space is likely to correlate strongly with Z C as computed from the control logic code coverage is a normalized quantity (expressed as a percent- age of the lines of code, etc.), functional coverage can be normalized, too. 5.5 Scaling Regression using Convergence How “big”