Introduction This purpose of this document is to define a set of test cases that systems implementing the OpenADR 2.0b profile must successfully pass in order to obtain certification. Scope OpenADR 2.0b is an application layer message exchange specification used for twoway communication of Demand Response (DR) , price, and Distributed Energy Resource (DER) signals between the electricity service provider and its customers. OpenADR 2.0b is defined in the following documents: • The OpenADR 2.0b Profile Specification, Revision 1.1 • The OpenADR 2.0b schema The conformance requirements that are not defined by the schema are contained in a set of conformance rules in the OpenADR 2.0b Profile Specification. These rules are referenced in both the OpenADR 2.0 Protocol Implementation Conformance Statement (PICS) and this document. Appendix F provides a mapping between each of these conformance rules and the test cases defined in the test specification. The other narrative sections of the OpenADR B Profile specification are considered informational and the Test Harness assumes that all the testable requirements have been captured in the conformance rules.
OpenADR 2.0b Certification Test Specification Version 1.1.0 Revisions: Version 1.0.0 1.0.1 • • 1.0.2 1.0.3 • • • 1.0.4 1.0.5 • • • • • • • • • • 1.0.6 • • • • • • Changes Released Version Revised test case Rx_3170 to change metadata template to Request_005 Revisions to test cases for updates Revised test case N1_0080 and N0_5080 to expect no response or 463 error in oadrResponse Modified the scope of testing to use test one randomly selected test case and all security test cases while testing multiple transports A_E3_0680 is now optional S0-303 Diagram has been updated Removed RegisterReport from Diagram S0-304 Displaying Push sequence in Diagram S1-812 Updated Diagram S0-311 and S1-811 to use oadRegisterReport Changed to VTN Ignore oadrCreatedEvent Modified N0_5080 to also accept appropriate responses Test case A_E3_0680_TH_VEN has been modified to use oadrRequest and not oadrPoll The following test cases have been made optional o R1_3050_TH_VEN, R0_8050_TH_VEN_1, R1_3060_TH_VEN, R0_8060_TH_VEN, R1_3120_TH_VTN_1, R0_8120_TH_VTN, R1_3070_TH_VTN_1, R0_8070_TH_VTN_1, R1_3080_TH_VTN_1, R0_8080_TH_VTN_1, R1_3090_TH_VTN_1,R0_8090_TH_VTN_1, R1_3100_TH_VTN_1, R0_8100_TH_VTN_1 Corrected typo in N1_0015, N0_5015, N1_0070 and N0_5070 Added pulseCount to Metadata_001 Removed second prompt A_Ex_0484 Updated prompt in test case A_E3_0655_TH_VEN, E1_1040_TH_VEN, E0_6040_TH_VEN Removed Test Case A_E0_0295_TH_VTN and A_E0_0295_DUT_VEN Updated A_E3_0420_TH_VEN to use oadrRequestEvent Updated Prompt_034 text Date/Editor 06/28/13 - JZ 07/09/13 - JZ 08/07/13 – SK 08/12/13 - EP 08/20/13 - EP 09/06/13 - BD, EP 2/11/2014 – EP, BD • • • • 1.0.7 • • • • • • • • • • • • • • • Modified E2_0530/E3_0530 prompt Expected result for Test case R1_3130_ and R0_8130_ has been modified Also Prompt_043 has been modified for the same Scenario for test case A_E2_0527 has been modified Added test case E1_1055 & E0_6055 to test Multiple Signals in an Event Split G0_9010 into G0_9005 for RSA and G0_9010 for ECC Updated A_E3_0430_TH_VEN/A_E3_0435_TH_VEN for the number of intervals Removed all references to startBefore Mantis 139: Added adjacent event tests case E1_1090 and E0_6090 Mantis 144: Test G1_4012 added using event followed by poll for empty queue Mantis 154: Expected result for Test case R1_3130_ and R0_8130_ has been modified.Mantis 157: Conformance rule check modified to flag use of oadrPoll in http push Mantis 163: Added test cases G1_4030 and G0_9030 (both TH_VEN and TH_VTN) Mantis 164: Deleted Test Case A_E0_0325/A_E1_0325 MarketContext can have a wild card, so can't negative test Mantis 165: Modified test case E1_1065 and E0_6065 (both TH_VEN and TH_VTN) Mantis 166: Updated conformance rule check to verify dateTime is the same for all events in a oadrDistributeEvent payload Mantis 167: Modified conformance rule check to verify that "Z" is at the end of every dateTime string Mantis 173: Conformance rule check modified to validate ordering of intervals in reports Mantis 174: Confirmance rule check modified to validate ordering of dtstart relative to intervals in reports Mantis 183: update conformance rule CreateReport_GranularityValid_Check to confirm that granularity is less than reportBackDuration Mantis 186: Added tests for expired certificates: G1_4007_TH_VTN and G1_4007_TH_VEN • • Mantis 190: Updated tests N1_0020 and N0_5020 to expect oadrRequestEvent to be sent following the exchange of metadata reports during registration Revised tests A_E1_0285, A_E0_0285, A_E2_0510, 1.1.0 • • • A_E3_0510 and deleted tests A_E2_0520, A_E3_0520, A_E2_530, and A_E3_0530 as the result of changes to rule regarding overlapping events Added support for wildcard deviceClass enumerations in conformance rule check Added explicit signalType check for “setpoint” in test cases E1_1025_TH_VEN and E0_6025_THV_VEN Added checks to N1_0020_TH_VTN and N0_5020_TH_VTN to verify that the reports required by the conformance rules are returned by the VEN 2/22/16 - JZ Contents Introduction This purpose of this document is to define a set of test cases that systems implementing the OpenADR 2.0b profile must successfully pass in order to obtain certification Terms and Acronyms The following terms and abbreviations are used in this document: Term DR DUT DUT_VEN DUT_VTN TH TH_VEN TH_VTN VEN VTN Description Demand Response Device Under Test – This is used to refer to the software implementation of OpenADR and any supporting hardware necessary to execute the software Device Under Test Virtual End Node (VEN) Device Under Test Virtual Top Node (VTN) Test Harness - A software implementation of OpenADR playing the role of a VEN or VTN for the purpose of running test cases Test harness playing the role of the VEN Test harness playing the role of the VTN Virtual End Node Virtual Top Node Scope OpenADR 2.0b is an application layer message exchange specification used for two-way communication of Demand Response (DR) , price, and Distributed Energy Resource (DER) signals between the electricity service provider and its customers OpenADR 2.0b is defined in the following documents: • • The OpenADR 2.0b Profile Specification, Revision 1.1 The OpenADR 2.0b schema The conformance requirements that are not defined by the schema are contained in a set of conformance rules in the OpenADR 2.0b Profile Specification These rules are referenced in both the OpenADR 2.0 Protocol Implementation Conformance Statement (PICS) and this document Appendix F provides a mapping between each of these conformance rules and the test cases defined in the test specification The other narrative sections of the OpenADR B Profile specification are considered informational and the Test Harness assumes that all the testable requirements have been captured in the conformance rules The scope of the test cases will focus on validating the successful exchange of messages between VTNs and VENs, validating that any exceptions are handled gracefully and without service interruptions and that security mechanism are properly implemented Validating the functional behavior of the DUT in response to payload content is out of scope for the certification tests For instance, the test cases will test that a VEN acknowledges the receipt of a DR Event, but it will not validate that the VEN actually sheds load as a result of this message For product certification, the certification test suite will be will be run over top of the specific transport protocols supported by the DUT If multiple transports are supported by the DUT, the full conformance test will be run over one transport and a representative set of test cases will be run over the remaining transports The representative set shall consist of one randomly selected test case from each service plus the one security test case Conformance testing of the supported transports is out of scope for the certification tests However, limited testing will be done to validate OpenADR specific requirements for implementation of the transports, such as the inclusion of specific http headers The certification test suite will be run utilizing each of the security mechanisms supported by the DUT Conformance testing of the security mechanisms is out of scope for the certification tests However, limited testing will be done to validate OpenADR specific configuration requirements for implementation of the security mechanisms, such as use of x509 certificates, TLS versions, and cipher suites Exception handling tests will be limited to a small set of representative real-world scenarios The intent of these test cases will not be to exhaustively test all possible exception scenarios, but to validate that the DUT has sound exception handing capabilities For instance, a test might validate that the DUT triggers an error if it is sent a payload with an unknown VEN or VTN ID Boundary and limit testing will be limited and will utilize values that are deemed to be practical from a test execution perspective and typical from an implementation perspective, as specific implementation limits have not been defined by the OpenADR 2.0b Profile Specification For instance, testing that includes a responseDescription string would use, at most, a few hundred characters, not tens of thousands Optional elements will be tested with some payloads using all optional elements and other payloads excluding optional elements Note that optional elements not need to be included in the payload, but if they are, the VEN or VTN receiving the payload must understand and act upon those optional elements Test Methodology The test harness will play the role of the opposite party (VEN or VTN) in interactions with the DUT Each test case will define a set of prerequisites, a test scenario consisting of a sequence of VEN/VTN message exchanges, and an expected result Execution of that test scenario will result in payload exchanges between the DUT and the Test Harness to be analyzed The analysis will consist of the following: • • • • • The message interaction patterns are as expected This includes… o Correct response payloads For instance, receiving an oadrCreatedEvent payload in response to an oadrPoll would not be correct o Appropriate request payloads For instance a request containing oadrCreatedOpt payload would not be correct That payloads contain well-formed XML That payloads conform to the OpenADR 2.0b schema That the specific conformance rules defined in the OpenADR Profile Specification are followed That the intent of the test case is met The test case may expect the VEN to send an optType of “optOut” and if this is not received, the test case will fail Note that some of the OpenADR conformance rules can be validated by simply analyzing the payloads of all message exchanges that occur during execution of the certification test suites These rules are summarized in Appendix F Other conformance rules will require specific test scenarios to validate A few test cases will require the test engineer executing the test to observe the behavior of the device under test in order to determine the pass fail status Test Case Organization The test suite is organized as follows: • testcases o pull event • • • vtn general opt • • ven vtn • • ven • • ven vtn vtn registerparty report • • o selftest_a_ported ven ven vtn Push Test Case Numbering Test case numbering uses the following pattern: XY_ZZZZ_TH_WWW_1 X = A letter designating the service or special test type • • • • • E= EiEvent Service Tests P=EiOpt Service Tests R=EiReport Service Tests N=EiRegisterParty Tests G=General Tests (includes non-service specific test such as security) Y = A digit indicating the type of implementation: • • 1= Pull Test = Push Test ZZZZ = A numeric sequence number for the test case with initial increments of 10 For a given Push or Pull test scenario, each VEN test case must have a matching VTN test case with the same digit value TH_WWW = The "TH" indicates Test Harness and the "WWW" indicates the role, either "VEN" or "VTN" This helps the user keep track of the role that the test harness is playing in the exchange _1 = An optional postfix to indicate that the test case should be started before the device under test So, an example test case would be R1_1223_TH_VEN_1 This would be an EiReport Pull Test case where the test case is playing role of the VEN and this test case should be started first Test case definitions reference sequence diagrams in Appendix G These sequence diagrams will follow a similar syntax staring with an S, followed by a push/pull indicator, a dash, then a digit number Example: S0-123 B Profile Test Case Template Each test case definition results in set two sets of mirrored test routines; one set each for push and pull exchange models, and within each set a VEN and VTN test routine The result is that each test case definition will result in separate test routines, one each for the following entities: VEN Push, VTN Push, VEN Pull, VTN Pull Test cases can be run against their counterpart DUT or against each other in a self test scenario Each test case is based on a use case scenario defined by a sequence diagram in Appendix G Unless otherwise noted, the test case will use the exact sequence of payload exchanges defined in the use case sequence diagram The use cases also provide guidance as to the location in the sequence for user prompts Test Case Name Pull Test Case Push Test Case Objective Prerequisites Scenario Details Expected Results Test Case Root ID xx_xxxx_ xx_xxxx_ Use Case Scenario Name of use case scenario for pull Name of use case scenario for push TH_VEN A sentence indicating the goal of the test case with respect to the DUT VTN Any prerequisite settings or conditions Payload-specific data for the test scenario and any necessary prompts The expected behavior of the DUT VTN TH_VTN A sentence indicating the goal of the test case with respect to the DUT VEN Any prerequisite settings or conditions Payload-specific data for the test scenario and any necessary prompts The expected behavior of the DUT VEN Note that test case definitions will reference a set of standardized resources defined in the appendices as follows: • • • • Appendix A - Standard prompts referenced in the test cases as Prompt_xxx Appendix B - Standard events referenced in the test cases as Event_xxx Appendix C - Standard opt schedules referenced in the test cases as OptSchedule_xxx Appendix D - Standard reports referenced as Metadata_xxx, Request_xxx, Update_xxx Ported A Profile Test Cases Test cases ported from the A Profile Test Harness will retain the same test case definition format as used in the A profile test specification as well as the same numbering, with the addition of an "A_" prefix These test cases are functionally identical to the A profile tests with the exception that the pull tests use oadrPoll rather than oadrRequestEvent Also note that while new B profile tests can be run against their matching counterparts (VEN or VTN) , the A profile test cases use a self test routine which will be stored in a package named "selftest_a_ported" Test case definition will use the following format: Test Case Name (Ported A Test) Pull Test Case Push Test Case Test Case Number A_xx_xxxx_TH_VEN A_xx_xxxx_TH_VEN Self Test Number A_xx_xxxx_DUT_VEN A_xx_xxxx_DUT_VEN TH_VTN Prerequisites - DUT_VEN has no pending or active events Scenario {DUT_VEN: oadrPoll } TH_VTN: oadrDistributeEvent – Dispatch Event {DUT_VEN: Transport Acknowledgement } DUT_VEN: oadrCreatedEvent TH_VTN: oadrResponse , success 1) Common Pass/Fail verdicts 2) Received oadrCreatedEvent with optType equal to optIn -A basic Dispatch event transaction Expected Results Notes Test Engineer Observations Each of the test cases above show a conditional oadrPoll as the first operation in the sequence which applies to the pull scenario only and a conditional transport acknowledgement which applies to the push scenario only Unless otherwise specified, the following defaults will be used for payloads generated by the test system: oadrDistributeEvent: • • • • • • • • • • • • Mandatory elements unless otherwise specified in the test description marketContext as configured in the test system oadrResponseRequired set to always eiTarget with no sub-elements dtstart set to currentTime +2 minutes x-eiNotification set to a duration of minute signalID set to a placeholder value Overall duration for CPP Event set to minutes Interval durations for CPP Event set to minute Interval signalPayload set to a repeating sequence of 1, 2, 3, 2, eiResponse element with mandatory elements included if responding to an oadrPoll requestID = unique value 10 Diagram: S1-703 VEN Opt - Mutliple Opt Scheduled with Cancel 203 Diagram: S1-800 VEN Reporting - One Shot Report 204 Diagram: S1-801 VEN Reporting - Periodic Report 205 Diagram: S1-802 VEN Reporting - Multiple Report Request Payloads 206 Diagram: S1-803 VEN Reporting - Cancel Report 207 Diagram: S1-804 VEN Reporrting - piggy back Cancellation 208 Diagram: S1-805 VEN Reporting - piggy back request 209 Diagram: S1-806 VTN Reporting -One Shot Report 210 Diagram: S1-807 VTN Reporting - Periodic Report 211 Diagram: S1-808 VTN Reporting - Multiple Report Request Payloads 212 Diagram: S1-809 VTN Reporting - Cancel Report 213 Diagram: S1-810 VTN Reporting - piggy back Cancellation 214 Diagram: S1-811 VTN Reporting - piggy back request 215 Diagram: S1-812 VTN Reporting - Metadata Report 216 Diagram: S1-813 VEN Reporting - Metadata Report 217 ... service provider and its customers OpenADR 2.0b is defined in the following documents: • • The OpenADR 2.0b Profile Specification, Revision 1.1 The OpenADR 2.0b schema The conformance requirements... the test cases defined in the test specification The other narrative sections of the OpenADR B Profile specification are considered informational and the Test Harness assumes that all the testable... profile test cases use a self test routine which will be stored in a package named "selftest_a_ported" Test case definition will use the following format: Test Case Name (Ported A Test) Pull Test