Standardized Functional Verification- P6 pdf

2.14 Special Cases 35 <’ struct gen_crc { crc: int; crc = compute_crc(data); keep impose_error == TRUE => crc == crc+1; // Adding 1 to computed value yields a crc error. } ‘> 2.13 Generating Excitement Choosing the values of the external conditional variables and the values of stimulus variables determines the excitation of an instance within its con- text. Pseudo-random excitation is currently the province of commercially available pseudo-random test generators. 2.14 Special Cases The foregoing categories of variables (connectivity, activation, conditions, and stimuli and response) form the basis for pseudo-randomly exercising the target. However, certain aspects of a target’s functionality may not be readily exercised in a pseudo-random manner, but instead require special treatment. These are best defined as special cases, and are often exercised with deterministic tests, or with deterministic test sequences within a pseudo- random sequence. Most special cases will be associated with stimuli or responses, but there may also be other things to verify that just don’t seem to fit neatly into any subset. This should not impede progress in interpreting the design specifi- cations – just write them down as for later classification. Eventually, with time and increasing familiarity with the design, special cases in this latter category will be recast as variables of suitable type with their respective ranges. Treating them as special cases early in the project is a practical convenience that enables progress in spite of a few “loose ends”. Interpreting functionality related to reset often accumulates a number of special cases. For example, checking the state of the target immediately af- ter de-assertion of reset (whether hard reset or soft reset) is more readily accomplished with a deterministic test and merits description as one or 36 Chapter 2 – Analytical Foundation more special cases. Similarly, checking for state preservation upon receiv- ing a soft reset constitutes another batch of special cases. On the other hand, these are better regarded as variables of response whose values are dependent on the type of reset. If indirect conditions are defined for a target, it is important that the dy- namic transition from one value to another during the application of stim- uli under given conditions be recorded as special cases to be exercised and observed. Other external activity such as the connection or disconnection of a ca- ble, entering or leaving power-save states, and entering or leaving test modes (morphing the target) can also be recorded as special cases during early stages of a project. Each of the special cases discussed so far fall into the category fre- quently referred to as scenarios. As will be seen later, these scenarios ac- tually correspond to transitions among values of standard variables, but for the time being they will simply remain on the list as special cases. The second category of special cases lists properties 5 of the target. These describe higher-level (or transcendental) behaviors of the target that are usually not verifiable via CRV, such as the performance (in MIPS) of a processor. Verifying performance is likely to need additional testing above and beyond CRV, which tends to focuses on correctness and completeness rather than speediness. Properties often encompass behavior of multiple clock-domains and are often expressed with inexact requirements. Passing a compliance test (such as for USB or PCI-Express) is also a property of the target. 6 Other properties may be based on aesthetic evalua- tions, such as the quality of an image encoded and decoded by the target, or the sound fidelity produced by the target. Cybernetic properties such as responsiveness to human interaction (in controlling a commercial aircraft, for example) would be listed here as well. 5 Note that the term properties is also used in the context of model checking where a given property of the RTL is to be checked. Properties listed as special cases are not based on RTL, but are based only on the requirements stated in the de- sign specifications. 6 Passing a compliance test does not mean that the risk of a functional bug is zero. Examples abound of products (including IP) that pass compliance tests but still have bugs. 2.15 Summary 37 2.14.1 Example – Special Case A classic example of a property as a special case comes to us from the specifications for Ethernet (IEEE 802.3) that states requirements on the distribution of the back-off slot times. Ethernet uses a collision-detect multiple-access algorithm on a shared medium, the wire carrying the electrical signals. When two (or more) de- vices happen to drive the wire simultaneously, both (or all) devices must stop driving the wire and wait some specified period of time until attempt- ing to drive the wire again. The time each must wait is referred to as the back-off slot time, where a slot corresponds to a specific number of bit times. Testing for this requirement is achieved within a network of compliant, commercially available devices. The back-off slot times used by the device being tested are recorded and then tested for a probabilistic uniform distri- bution. Verifying this particular requirement doesn’t lend itself to CRV testing and isn’t readily verified in simulation. Similarly, functional requirements related to liveness, fairness, or performance of a target may require treat- ment as a special case outside the context of CRV. 2.15 Summary The taxonomy defined in this chapter provides a consistent and compre- hensive nomenclature for all aspects of functional verification regardless of the size or complexity of the digital system to be verified. The entire function space can be characterized completely and exhaustively by a well-defined set of standard variables using the linear algebra for digital systems. The premise stated at the beginning of this chapter bears repeat- ing here: From the standpoint of functional verification, the func- tionality of a digital system, including both the target of verification as well as its surrounding environment, is characterized completely by the variability inherent in the definition of the system. Different teams will interpret specification documents to yield different standard variables, surely differing in name but also differing in definition, 38 Chapter 2 – Analytical Foundation but the sets of variables will be equivalent, each forming orthogonal basis sets. Any non-equivalence indicates one or “bugs” in a document because it is not sufficiently clear or ambiguous in its interpretation. Table 2.1 summarizes this “loosely orthogonal system of basis variables.” Table 2.1. Relations among variables for a target of verification Choose values for these variables: To establish: Internal connectivity Instance External connectivity Context Power, clocking, and reset Activation Morph-related conditions and stimuli Morph Internal conditions Operation External conditions Stimuli Errors Excitation The variables discussed in this chapter already exist in testbenches around the world. However, they simply have not been partitioned into the standard framework. And, without this standard partitioning, it is ex- tremely difficult to compare different verification projects or to use these projects to assess risk for future projects. With this set of standard variables it is now possible to determine the size of the functional space in terms of its points and arcs and, therefore, determine what is required to achieve functional closure, that is, exhaus- tive functional coverage. This will be the topic of the next chapter. References Bailey B, Martin G, and Anderson T (ed.) (2005) Taxonomies for the Develop- ment and Verification of Digital Systems. Springer. Lachish O, Marcus E, Ur S, Ziv A (2002) Hole Analysis for Functional Coverage Data, Proceedings of the 39th conference on Design Automation, ACM Press. Verisity Design Inc. (2002) e Language Reference Manual www.ieee1647.org. Verisity Design Inc. (2005) Spec-Based Verification, whitepaper Chapter 3 – Exploring Functional Space The previous chapter described a method of interpretation of natural- language specification documents into a standard set of variables and their ranges that is applicable to all possible digital systems. Each unique com- bination of values of these variables determines a single function point within the functional space. These function points determine the functional space in a mathematically precise manner, allowing us to determine the size of this space and, consequently, determine what it means to achieve functional closure. Value transition graphs provide a technique for visual- izing this space in a useful way. 3.1 Functional Closure Much recent discussion in the industry has focused on “functional closure” but without providing a solid, substantive definition for this highly desir- able outcome. Intuition tells us that functional closure must exist because, after all, the functionality can be expressed with a finite number of transis- tors. What does it really mean to achieve functional closure? This term is borrowed from timing domain analysis where closure is a genuine outcome. To reach timing closure is to adjust the placement of transistors and the routing of interconnecting wires and polysilicon such that each and every path propagates signals within a predetermined inter- val of time so that setup times at destinations are satisfied. Software tools make exhaustive lists of signal paths and then determine whether all paths have no negative slack. When a signal has positive slack, it means that the signal reaches its destination with time to spare, whether a few picosec- onds or many nanoseconds. A signal with negative slack reaches its desti- nation too late. When all paths have no negative slack, nothing arrives late. Some texts refer to “functional closure” or “coverage closure,” but while also claiming that the size of the space over which this “closure” is reached cannot be determined. This is not correct, as we will see in the analysis in this chapter. A survey of current literature finds many technical papers and articles that tell us, “A good test plan should list all the interesting test cases to 40 Chapter 3 – Exploring Functional Space verify the design.” However, there is no objective way to separate “inter- esting” from “uninteresting” test cases. If so, there would be a software program that could accept as its input these “test cases” in some yet to be defined form and then decide algorithmically whether it is interesting or uninteresting. Clearly, this approach does not seem promising. Similar remarks appear in the current literature regarding coverage, in- forming the reader to “decide on the interesting data fields/registers … im- portant state machines, … interesting interactions between some of the above states or data.” Again, we have no objective way to make these de- cisions regarding interestingness or importance. Formal verification (such as theorem provers of some sort) might some day be able to make such de- cisions. With a complete set of variables and their ranges as described in chapter 2, it is possible to arrive at an understanding of the overall size of the func- tional space of a given target. Because these variables imply all of the mechanisms (elements) that use them, they may be said to represent the functional space completely. This means that it is now possible to gain a complete characterization of the functional space of the target. 3.2 Counting Function Points The previous chapter explained that each unique combination of values of standard variables determines a single function point within the functional space. In other words, a function point is represented by a tuple corre- sponding to the location of a point in some large multi-dimensional space, providing us with the coordinates of the point in this space. It is possible to arrive at an exact count P of these points by considering each category of standard variable in turn and how it relates to the other categories of vari- ables. Analysis begins with connectivity. 3.2.1 Variables of Connectivity Variables of connectivity are separated into two categories, variables of in- ternal connectivity and variables of external connectivity. Each variable has a finite range of discrete values that are countable. This is true because these ranges are present in computer programs (verification software tools). In many designs the ranges of variables of connectivity are independent such that every value of some variable of connectivity k A can be associated with each and every value of some other variable of connectivity k B . So, if 3.2 Counting Function Points 41 for example a design has these two variables of internal connectivity k A and k B , then the number of points determined by only these two variables is simply the product of the number of values in the range of each variable. However, for many designs the range of values of certain variables are dependent upon the values of one or more other variables. Some ranges are not simple finite sets of discrete values, but rather must be determined by methodically analyzing dependencies among the variables. Consider the specification for a microprocessor that includes an optional cache control- ler capable of handling caches of varying size. This defines two variables of internal topology: CACHE_PRESENT and CACHE_SIZE. The range of CACHE_PRESENT is [YES,NO] and the range of CACHE_SIZE is (for our example) [4K,8K,16K]. The number of function points is not simply 2 ⋅ 3 = 6 due to the dependency of CACHE_SIZE on CACHE_PRESENT. When CACHE_PRESENT is equal to NO, then there is no related value for CACHE_SIZE. For that matter, one might argue that the variable CACHE_SIZE does not even exist when CACHE_PRESENT is equal to NO. For notational convenience, we will use the value of ∅ (the null value) in the tuple when the value of the corresponding variable is undefined. So, in this example there are actually only 4 function points whose co- ordinates are: • (NO,∅), • (YES,4K), • (YES,8K), and • (YES,16K). One further example illustrates how the count of function points grows quickly even for a design with relatively few variables of connectivity. Consider a simple design with 3 variables of internal connectivity (k i1 , k i2 , and k i3 ) and 2 variables of external connectivity (k e1 and k e2 ). Assume that all 5 variables are mutually independent. Let’s examine first how many different instances of the target can be built. Let k i1 have 2 values in its range, let k i2 have 3 values in its range, and let k i3 have 4 values in its range. Because all 3 variables are mutually independent, there are at most 2 ⋅ 3 ⋅ 4 = 24 different instances that can be built. Now consider how many different external contexts can surround each of these 24 instances. Let k e1 have 5 values in its range and let k e2 have 6 values in its range. Again, because the variables are independent, there are 5 ⋅ 6 = 30 possible different contexts. 42 Chapter 3 – Exploring Functional Space Combining our 24 possible instances with our 30 possible contexts, we find that the subspace of connectivity contains 24 ⋅ 30 = 720 distinct func- tion points in our simple example. In other words, it is possible to build 720 distinctly different systems. Now let’s consider the general case. Consider the set X containing all of the values from the range of a vari- able of connectivity (or any other category of variable, for that matter) and also the null value ∅. The zero norm of X, denoted X , is the number of non-zero elements of X. This zero norm is also known as the Hamming weight of X. For the purposes of analyzing functional spaces, X is de- fined as the number of non-null elements of X. Assume that there are M different variables in k i (variables of internal connectivity) and N different variables in k e (variables of external connec- tivity). To arrive at the maximum number of function points P k in the space of connectivity, we count the number of values in the range of each variable and then take the product of all of these counts. Mathematically this is stated as: P k i = number of instances = X i m m=1 M ∏ and P k e = number of contexts = X i n n=1 N ∏ . (3.1) Then, the number of function points P k in the subspace of connectivity of our design is at most the product of these two products: P k ≤ P k i ⋅ P k e . (3.2) Any dependencies between ranges of variables will reduce the overall number of possible instance-context pairs, so P k may usually be less than this upper bound. To illustrate how these dependencies reduce the number of function points, consider the first example in previous chapter that de- scribed the standard variables of external connectivity. In that example we had a simple central bus on which 3 different types of agent could reside: from 1 to 8 processors, from 1 to 4 memory control- lers, and from 1 to 14 I/O controllers for a total of 16 agents. The simple product of the counts of values in the ranges of the respective variables 3.2 Counting Function Points 43 yields a total of 448 function points (448 = 8 ⋅ 4 ⋅ 14). However, this count includes systems that violate the no-more-than-16-agents requirement. Consider a system with one processor and one memory controller. This particular system can include any of 1 to 14 I/O controllers for a total of 14 distinct contexts. A system with one processor and two memory controllers can include any of 1 to 13 I/O controllers for a total of 13 distinct systems. Of course, it’s unlikely that actual systems corresponding to each and every one of these systems would be interesting in a commercial sense, but the point is that such systems are theoretically allowed within the ranges defined by the requirements. Table 3.1. Number of possible contexts based on example in Fig. 2.2 proc_agents mem_agents io_agents Possible contexts 1 [1 14] 14 2 [1 13] 13 3 [1 12] 12 1 4 [1 11] 11 1 [1 13] 13 2 [1 12] 12 3 [1 11] 11 2 4 [1 10] 10 M M M M 1 [1 8] 8 2 [1 7] 7 3 [1 6] 6 7 4 [1 5] 5 1 [1 7] 7 2 [1 6] 6 3 [1 5] 5 8 4 [1 4] 4 Total: 288 Continuing in this manner (see Table 3.1) we see that 50 (50 = 14 + 13 + 12 + 11) distinct single-processor systems can be assembled when only one processor is included. Accounting for the dependency on the range for I/O controllers for the remaining possible N-processor systems, we see that only 288 possible systems can be assembled. This 288 is the actual number of function points in the space of connec- tivity, significantly fewer than the 448 calculated as the simple product, but still a large number of possible contexts in which the target of verifica- tion must operate correctly. This is the challenge for the IP developer. 44 Chapter 3 – Exploring Functional Space In general it is not possible to rely on simple closed-form algebraic ex- pressions (such as in Eq. 3.1 and Eq. 3.2) to determine the number of func- tion points in the subspace of connectivity (or any other subspace, for that matter), but rather these function points must be counted algorithmically. This is accomplished in a straightforward manner with nested for loops that examines each tuple of values of variable of internal connectivity (i.e., each instance) in turn and determining which tuples of values of variables of external connectivity (i.e., contexts) are valid, just as was done to create Table 3.1. 3.2.2 Variables of Activation (and other Time-variant Variables) Variables of activation are categorized into three subsets, namely power, clocking, and reset. For any valid system there may be multiple ways that the system can be activated, each described not by a single tuple of values of these variables, but rather by sequences in time of changing values of these variables. These sequences will be analyzed at length later in this chapter, but some preliminary analysis is in order here. Consider the ex- ample of the multi-processor design that we have been using so far. In a system of multiple processors one of the processors is typically des- ignated as the monarch processor with the remaining processors designated as serfs. Activation of the monarch precedes activation of the serfs, per- haps by holding each serf in its reset state until system resources such as memory are sufficiently initialized for the serfs to begin fetching and exe- cuting instructions on their own. Also, some serf processors might not even be energized until they are needed. And, a given serf might be de- activated to conserve power when it is not needed. Each of these scenarios constitutes a different sequence of values of variables of activation, each sequence composed of multiple function points. Function points corresponding to time-variant variables such as those of activation are not counted as simply as time-invariant variables of connec- tivity. Constructing the value transition graphs (defined later in this chap- ter) is necessary before the corresponding function points can be counted. Accounting for the 4 possible instances of the processor as determined by CACHE_PRESENT and CACHE_SIZE, this yields 1,152 distinct sys- tems, all of which must be verified to function correctly. At first glance this may appear to a rather dire situation for the verification engineer, but, as will be seen later in this chapter, condensation in the functional space can provide significant practical advantages. . determine the size of the functional space in terms of its points and arcs and, therefore, determine what is required to achieve functional closure, that is, exhaus- tive functional coverage. This. functional closure. Value transition graphs provide a technique for visual- izing this space in a useful way. 3.1 Functional Closure Much recent discussion in the industry has focused on functional. Intuition tells us that functional closure must exist because, after all, the functionality can be expressed with a finite number of transis- tors. What does it really mean to achieve functional closure?