Quick Start The following steps will get the test harness running in a localhost environment in just a few minutes: Step 1 Install a version of Eclipse on your computer (See Eclipse Configuration on page 9). Step 2 Unzip the Eclipse project OpenADR20bCertTest_v1_x_x into your eclipse working directory. Import the project into your Eclipse environment. Step 3 Start Eclipse and expand the tree in Package Explorer srctestcasespushregisterParty Step 4 Run a VEN registration test case against a VTN test in a self test mode over http as follows (Shortcut: run SelfTest_Register in util): • Run venN0_5020_TH_VTN_1 • Click Resume on first two prompts • Run vtnN0_5020_TH_VEN • Click Resume on prompt • Watch the console windows as test cases execute Step 5 To view a detailed transaction log (starting with TraceLog_xxx), you can look in the log folder off the Eclipse root (you might need to refresh the folder) or point a browser at the logfile.html file in the same folder for a prettier interface. (Shortcut: run LogFiles_View in util) Step 6 Now lets run the same tests over XMPP. Unzip openfire_Windows_xxxxx or openfire_Linux_xxxxx file into a system folder. Step 7 Start Openfire from the bin directory. On Windows use openfire.exe or OpenADR_Start.Bat if you encounter issues. On Linux use.openfire start. Optional Step 8 You can start the Openfire console at 127.0.0.1:9090 from a browser. Enter username as admin and password as password. Ignore warning about RSA cert not matching server domain. Step 9 Modify the selftest properties located in the Eclipse Package Tree at srcproperties vendor.selftest.openadrconfig.properties as follows: Transport_Type=XMPP. Step 10 Repeat steps 3, 4, and 5 to run the same registration test case over XMPP. You have now successfully run test cases across both transports Suggested next steps: • Review Appendix B to get a feel for how the test harness is configured. • Skim the balance of this user manual so you know what information is available. • Set up your device, configure the test harness, and start running tests
OpenADR 2.0b Certification Test Harness User's Manual Version 1.1.1 Revisions: Version 1.0.0 1.0.1 1.0.2 1.0.3 1.0.4 1.0.6 1.0.7 • • • • • • • • • • 1.1.0 1.1.1 • • 1.0.5 Changes Initial Release Updates Updates Corrected typo in property disable_MarketContext_Check Added utility class VTNPollQueue_Clear to clear the VTN poll queue maxSocketTimeout property has been added Updated Appendix D for 1.0.5 release Deleted Appendix E and open issues documented here have been resolved Updated Appendix D for 1.0.6 release ECC certificate support is added SHA1 removal Many other edits Version Change HTML Logfile Added Date/Editor 06/28/13 – JZ 07/25/13 - SK 08/06/13 – SK 08/06/13 08/22/13 09/11/13 - JZ 03/19/14 - SK 11/1/15 – JZ 2/22/16 – JZ 6/26/16 - JZ Table of Contents Quick Start The following steps will get the test harness running in a localhost environment in just a few minutes: Step - Install a version of Eclipse on your computer (See Eclipse Configuration on page 9) Step - Unzip the Eclipse project OpenADR20bCertTest_v1_x_x into your eclipse working directory Import the project into your Eclipse environment Step - Start Eclipse and expand the tree in Package Explorer src/testcases/push/registerParty/ Step - Run a VEN registration test case against a VTN test in a self test mode over http as follows (Shortcut: run SelfTest_Register in util): • Run ven/N0_5020_TH_VTN_1 • Click Resume on first two prompts • Run vtn/N0_5020_TH_VEN • Click Resume on prompt • Watch the console windows as test cases execute Step - To view a detailed transaction log (starting with TraceLog_xxx), you can look in the log folder off the Eclipse root (you might need to refresh the folder) or point a browser at the logfile.html file in the same folder for a prettier interface (Shortcut: run LogFiles_View in util) Step - Now let's run the same tests over XMPP Unzip openfire_Windows_xxxxx or openfire_Linux_xxxxx file into a system folder Step - Start Openfire from the bin directory On Windows use openfire.exe or "OpenADR_Start.Bat" if you encounter issues On Linux use"./openfire start" Optional Step - You can start the Openfire console at 127.0.0.1:9090 from a browser Enter username as admin and password as password Ignore warning about RSA cert not matching server domain Step - Modify the selftest properties located in the Eclipse Package Tree at src/properties/ vendor.selftest.openadrconfig.properties as follows: Transport_Type=XMPP Step 10 - Repeat steps 3, 4, and to run the same registration test case over XMPP You have now successfully run test cases across both transports! Suggested next steps: • • • Review Appendix B to get a feel for how the test harness is configured Skim the balance of this user manual so you know what information is available Set up your device, configure the test harness, and start running tests! Overview OpenADR 2.0b is an application layer message exchange protocol used for two-way communication of Demand Response (DR), price, and Distributed Energy Resource (DER) signals between the electricity service provider and its customers The technical requirements for the protocol are defined in the OpenADR 2.0b Profile Specification Devices that implement OpenADR 2.0b must pass a certification test The requirements for the certification test are spelled out in the OpenADR 2.0b Certification Test Specification Each test case defined in the test specification is implemented in the OpenADR 2.0b Certification Test Harness All OpenADR 2.0b interactions are between a Virtual End Node (VEN) and a Virtual Top Node (VTN) using either a push or pull exchange model The functionality of OpenADR 2.0b is split across services which include EiEvent, EiOpt, EiReport, EiRegisterPart, and OadrPoll When a test is run, the test harness plays the role opposite that of the device under test For instance, when testing a VEN Push implementation, the test harness will play the role of a VTN Push implementation This document describes how to use the OpenADR 2.0b Certification Test Harness to execute one or more sets of tests, the sum of which represents the OpenADR certification requirements Test Cases contained in the test harness are organized in a hierarchy as follows: • src o testcases Exchange Model (push or pull) • Service (event, report, registerparty, opt, or general) o Role (VEN or VTN) Test case Test case Etc Test cases listed under "DR_PRogram_Guide" are not part of the message exchange certification and they may be ignored The specific sets of tests required to pass certification are outlined in the OpenADR 2.0b PICS document In order to successfully pass each of the certification tests, the device under test (DUT) must have certain capabilities as outlined in the following sections of the OpenADR 2.0b Test Specification: • • DUT Configuration Requirements DUT Implementation Limits The Test Harness is built on top of the open source Eclipse integrated development environment The test harness consists of a large body of code, referred to as the Test Framework that implements the OpenADR 2.0 protocols, and a series of small test case programs that implement the test case-specific logic Running a test case is as simple as selecting the test case from the Test Harness “Package Explorer” and clicking the green Run toolbar button, watching the text case execute in the Console window, and reviewing the test results in a browser The following figure shows the major functional areas of the Test Harness GUI The run button is used to execute a test case This is the test code which the user should not need to alter Test Cases appear in the Package Explorer Figure – Eclipse IDE During test execution progress indications will appear in the console window Once a test case has completed execution, a pass/fail indication is shown in the Console window Detailed transaction logs are recorded for each test execution, and these transaction logs can be used to isolate the cause of a test case failure Figure shows the Test Result Summary screen When a tracelog link is clicked, a detailed transaction log will be displayed in a text editor as show in Figure Figure – Test Result Summary Figure – Transaction Log Installation Java The security test cases require the test harness be run on top of Java 1.7 This can be obtained at the following site: http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html Eclipse IDE for Java Developers The test harness uses the Eclipse Juno IDE for Java Developers, which can be downloaded from the following site: http://www.eclipse.org/downloads/packages/eclipse-ide-java-developers/junosr2 Note that the test code is not sensitive to the version of Eclipse that is used as long as it is Juno or later The narrative in this user’s guide presumes the Juno version of Eclipse is being used Eclipse Configuration Two changes need to be made to the Eclipse environment (Fig 4a and Fig 4b) From the Main Menu select Window, Preferences, Java Expand Compiler, and then select the Errors/Warning option On the right panel under Deprecated API, change the Forbidden reference to Warning Also from Window, Preferences, select General, Workspace, and check Refresh using native hooks or polling Figure 4a – Eclipse Configuration Setting Figure 4b – Eclipse Configuration Setting Test Harness installation The OpenADR 2.0b Test Harness is distributed as an Eclipse project zip file The process for installing the Test Harness in the Eclipse workspace is as follows: • • • • Extract the project into a temporary directory Rename the project folder to something unique, and then copy the project directory into the Eclipse workspace directory Note that the project folder name will become the project name inside Eclipse Use the Eclipse File, Import, General, "Existing Project into Workspace" option You can then move over any property settings from the previous release and then delete the previous release if desired It is easier to traverse the test cases when running tests if you use the drop down menu in the Package Explorer to change the package presentation to hierarchical (Fig 5) Figure – Package Presentation Openfire Installation When testing a VEN over XMPP, the test harness VTN configuration includes an XMPP server called Openfire, and this server must also be installed Note that when testing a VTN, the test harness will be communicating with the XMPP server that is part of the device under test Create a folder on your local system and unzip one of the following Openfire packages to that folder: • • Windows = openfire.zip Linux = openfire.tar.gz Openfire can be started from the Openfire bin directory using openfire.exe (if you encounter issues use "OpenADR_Start.Bat" for Windows) or on Linux with "./openfire start" The Windows OpenFire, if started with the batch file, will run in a DOS window, as a special startup script was needed to get it to run under Java 1.7 on some systems Configuration In order for the test suite to work with a device under test, a number of properties need to be set in the test harness, Openfire server, and the device under test In general the following types of settings need to be configured: • Information necessary to connect to test harness to the device under test, including IP addresses, port numbers, user names, passwords, and security configuration 10 Error when launching browser or log - The following error appears in the console log of some systems when the browser or log viewer is launched programmatically We have not been able to isolate the cause of this error, but it appears to no harm ******** ERROR ******************************************************** This program is improperly using Base::Config by not initializing config with the Load functions See the documentation in Base/Dev/index.html for usage instructions At the minimum, LoadCommandLine should be use with a dummy command line consisting of the program name ************************************************************************ 20 Appendix A – Configuration Tables These following tables use the following addresses to illustrate the configuration options: • • • • • • • • • • Test Harness and Openfire installed on 192.168.5.100 Test Harness VTN http listener port for HTTP: 8080 Test harness VEN http listener port for HTTP: 8081 Test harness XMPP Port 5222 Device under test IP address 192.168.5.120 (http or XMPP, VEN or VTN) Device under test HTTP port 8082 Device under test XMPP port 5222 User names configured on XMPP server (Openfire or DUT VEN): o VEN: 111111111111 o VTN: 222222222222 Devices support SHA256 ciphers The well known service http postfix is not shown in examples Pull VEN Configuration Characteristic ven ID of the DUT vtnID of the test harness Market Context Associated with VEN Resource Supported If the DUT requires a special string to invoke a test event, add it to properties HTTP Configuration Test Harness (VTN) Property VEN_ID=DUT_VEN DUT VEN Configuration DUT_VEN VTN_ID=TH_VTN TH_VTN DR_MarketContext_1_Name= http://MarketContext1 DR_MarketContext_2_Name= http://MarketContext2 ResourceIDs= resource1, resource2,resource3,resource4 http://MarketContext1 http://MarketContext2 Device_Class=Thermostat,Strip_Heater TestEventString=Test Event Thermostat,Strip_Heater Transport_Type=HTTP HTTP_VtnURL_Port=8080 HTTP_Security= HTTP_SHA256_Security URL used to talk to VTN https://192.168.5.100:8080 21 resource1, resource2, resource3, resource4 Push VEN Configuration Characteristic venID of the DUT Test Harness (VTN) Property VEN_ID=DUT_VEN DUT VEN Configuration DUT_VEN venID of the test harness VTN_ID=TH_VTN TH_VTN Market Context Associated with VEN DR_MarketContext_1_Name= http://MarketContext1 DR_MarketContext_2_Name= http://MarketContext2 http://MarketContext1 http://MarketContext2 ResourceIDs= resource1, resource2,resource3,resource4 resource1, resource2, resource3, resource4 Device_Class=Thermostat,Strip_Heater Thermostat,Strip_Heater Resource Supported If the DUT requires a special string to invoke a test event, add it to properties HTTP Configuration TestEventString=This is a test Event Transport_Type=HTTP HTTP_VtnURL_Port=8080 HTTP_Security= HTTP_SHA256_Security URL used to talk to VTN https://192.168.5.100:8080 Listen on 8082 Transport Address sent in registration: https://192.168.5.120:8082 XMPP Test Harness Configuration Transport_Type=XMPP XMPP_VTN_Username=222222222222 TH_XMPP_Address=192.168.5.100 TH_XMPP_Port=5222 XMPP_Security=XMPP_SHA256_Security XMPP Username = 111111111111 XMPP Server = 192.168.5.100 XMPP Port = 5222 Transport Address sent in registration: 111111111111@192.168.5.100/client Client Cert CN Field = 111111111111 XMPP Openfire Server System Properties openadr2.vtn.jid = 222222222222@192.168.5.100 xmpp.domain = 192.168.5.100 User names configured on XMPP Server: 111111111111, 222222222222 Run OpenADR_XMPP_SHA256_Security.Bat 22 Pull VTN Configuration Characteristic venID of the test harness venID of the device under test Market Context Associated with VTN Resource Supported HTTP Configuration Test Harness (VEN) Property VEN_ID=DUT_VEN DUT VTN Configuration DUT_VEN VTN_ID=TH_VTN TH_VTN DR_MarketContext_1_Name= http://MarketContext1 DR_MarketContext_2_Name= http://MarketContext2 http://MarketContext1 http://MarketContext2 ResourceIDs= resource1, resource2,resource3,resource4 resource1, resource2, resource3, resource4 Device_Class=Thermostat,Strip_Heater Thermostat,Strip_Heater Transport_Type=HTTP Listen on port 8082 HTTP_VTN_BaseURL= http://192.168.5.120:8082 HTTP_Security= HTTP_SHA256_Security 23 Push VTN Configuration Characteristic venID of the test harness Test Harness (VEN) Property VEN_ID=DUT_VEN DUT VTN Configuration DUT_VEN venID of the device under test Market Context Associated with VTN VTN_ID=TH_VTN TH_VTN DR_MarketContext_1_Name= http://MarketContext1 DR_MarketContext_2_Name= http://MarketContext2 http://MarketContext1 http://MarketContext2 ResourceIDs= resource1, resource2,resource3,resource4 resource1, resource2, resource3, resource4 Device_Class=Thermostat,Strip_Heater Transport_Type=HTTP Thermostat,Strip_Heater Resource Supported HTTP Configuration Listen on 8082 HTTP_VenURL_Port=8080 HTTP_Security= HTTP_SHA256_Security HTTP_VTN_BaseURL= http://192.168.5.120:8082 HTTP_VEN_BaseURL= http://192.168.5.100:8081 XMPP Test Harness Configuration Transport_Type=XMPP XMPP_VEN_Username=111111111111 DUT_XMPP_Address=192.168.5.120 DUT_XMPP_Port=5222 XMPP_Security=XMPP_SHA256_Security Transport Address sent in registration: 111111111111@192.168.5.120/client 24 XMPP Username = vtn XMPP Server = 192.168.5.120 XMPP Port = 5222 User names configured on XMPP Server: 111111111111, vtn Security must match Test Harness Appendix B – Test Harness Configuration Properties ################################################################### # Vendor Specific Properties # To use this file set Properties_File key in # in openadconfig.properties to point to this file name #################################################################### ######################################### ######### Change Transport ############## ######################################### # Switches between HTTP and XMPP transports Valid values are HTTP or XMPP Transport_Type=HTTP ######################################### ######### Change Security ############## ######################################### # Note that the security settings below control which security configuration file is read when a # test case is run The security configuration files are located in scr.properties.security # ********** HTTP Security ************* # HTTP_Basic_Or_No_Security, HTTP_SHA256_Security, HTTP_SHA256_Expired HTTP_Security=HTTP_SHA256_Security # ********** XMPP Security ************* # XMPP_Plain_Security, XMPP_SHA256_Security XMPP_Security=XMPP_SHA256_Security # ********** Security Debug ************* # This controls whether a detailed TLS handshake transaction log is output to the console during message exchange Security_Debug=false ######################################### ######### HTTP Configuration ############ ######################################### # These values point to the base URL of the the DUT and the TH based on there roles # Example: If the DUT is a VEN, set HTTP_VEN_BaseURL to point to the DUT base URL # Example: If the TH is a VEN, set HTTP_VTN_BaseURL to point to the TH base URL # Do not use https as this is automatically set when Security_Enabled=true HTTP_VEN_BaseURL=http://127.0.0.1:8081 HTTP_VTN_BaseURL=http://127.0.0.1:8080 # This is the port that the test harness listens on depending on the role it is playing # If the test harness is a VTN, set vtnURL_Port HTTP_VtnURL_Port=8080 HTTP_VenURL_Port=8081 # Setting a name and password will cause basic authentication to be used for http # If left blank, basic authentication key values will not be put in the http header # Basic authentication is NOT part of the OpenADR requirements, so leave these values blank unless you have a special need Login_Name= Login_Password= 25 ######################################### ######### XMPP Configuration ############ ######################################### # These values point to the XMPP server and port used by the DUT and test harness for XMPP # When acting as a VEN, the test harness will talk to the DUT_XMPP_Address # When acting as a VTN, the test harness will talk to the TH_XMPP_Address # IMPORTANT NOTE: The XMPP address used to communicate with the server must be the same address # the server is configured for in the Openfire system configuration properties: openadr2.vtn.jid and xmpp.domain TH_XMPP_Address=127.0.0.1 TH_XMPP_Port=5222 # DUT XMPP Server DUT_XMPP_Address=127.0.0.1 DUT_XMPP_Port=5222 # Use XMPP Service Discovery to determine the VTN JIDs # If set to false the VTN account below is used as the service endpoint XMPP_Service_Discovery=true # VEN account when test harness is playing the role of the VEN # Note that the transport address provided to the VTN is a concatination of the XMPP_VEN_Username below and DUT_XMPP_ADDRESS # The password is used only for Plain Authentication, which is NOT part of the OpenADR standard # Note that the VEN Username MUST match the username configured on the Openfire server and the x.509 cert XMPP_VEN_Username=111111111111 XMPP_VEN_Password=password XMPP_VEN_ResourceName=client # VTN account used when the test harness is playing the role of the VTN # The password is used only if XMPP_SASL_External is false # Note that the VTN Username MUST match the username configured on the Openfire server and the x.509 cert XMPP_VTN_Username=222222222222 XMPP_VTN_Password=password XMPP_VTN_ResourceName=client # Enable Smack debug window when running test cases # Note that will will have to manually kill the test case process with this enabled XMPP_Debug=false ################################################# ######### OpenADR Test Case Settings ############ ################################################# # This VTN ID used by the test harness when playing the role of the VTN # DUT VEN's should be configured to expect this venID from the VEN VTN_ID=TH_VTN # Market context values for events If sending an event, DR_MarketContext_1_Name is included in event # and if receiving an event DR_MarketContext_1_Name is expected in the event DR_MarketContext_1_Name=http://MarketContext1 DR_MarketContext_2_Name=http://MarketContext2 # These characteristics are used to populate the oadrCreatePartyRegistration payload CreatePartyRegistration_ProfileName=2.0b CreatePartyRegistration_ReportOnly=false CreatePartyRegistration_XmlSignature=false 26 CreatePartyRegistration_oadrVenName=Test_VEN_Name #Polling frequency returned by TH_VTN in oadrcreatedPartyRegistration CreatedPartyRegistration_RequestedOadrPollFreq=PT10S # These characteristics are used to populate the TH oadrCreatedPartyRegistraion payload Profile_A=2.0a Profile_A_Transport=SIMPLE_HTTP Profile_B=2.0b Profile_B_Transport=SIMPLE_HTTP,XMPP # Used for populating targeting information in outbound payloads and validating targeting information on inbound payloads ResourceIDs=resource1,resource2,resource3,resource4 # Device classes Device_Class=Thermostat,Strip_Heater # Used to populate the test event string in events TestEventString=This is a test Event # For Test Case A_E0_0200 and A_E1_0200 # These configuration values are used for specific test cases and, as a general rule, should not be changed MaxLen_VTN_ID=01234567890123456789012345678901234567890123456789 MaxLen_MarketContext=http://A890123456789012345678901234567890123456789012345678901234 56789012345678901234567890123456789 # This controls dynamic generation of venID's during registration # If set to true, the test harness will validate inbound VEN traffic against this VEN_ID # and will use this value to populate the venID of outbound VEN Traffic UseStaticVENID=false VEN_ID=TH_VEN ######################################## ######### Misc Properties ############## ######################################## # An overall timeout for the test case in milliseconds During prompts, this timeout is suspended asyncResponseTimeout=200000 # The time to wait in milliseconds for all the expected oadrCreatedEvent responses to be received createdEventAsynchTimeout=95000 # Buffer time to handle any residue request or response once the test case is done testCaseBufferTimeout=3000 # Maximum socket timeout in milliseconds (Applies only for HTTPS security) maxSocketTimeout=60000 # Info for Test log Report Header Report_Header_Info_1=Name of Implementation Report_Header_Info_2=Version Number ##################################################### ######### Conformance Rule Check Control ############ ##################################################### # Each payload exchanged is checked for a variety of conformance rules These rules can be # disabled on a global basis if a particular bug in an implementation is preventing test cases from running # These checks can also be disabled on a test case specific basis (see helpful hints in user guide) 27 #******* General Conformance Rules ******** disable_MarketContext_Check=false disable_UIDValid_Check=false disable_VtnIDValid_Check=false disable_AtLeastOneEiTargetMatch_ValidCheck=false disable_VenIDValid_Check=false disableHTTPHeader_Valid_Check=false disablePoll_PUSH_Check=false disableResponseEvent_ResponseCodeSuccess_Check=false disableRequestIDValid_Check=false disableSchemaVersionValid_Check=false disableDistributeEvent_OnePayloadFloatPresentInEachInterval_Check=false disableDeviceClass_Check=false disable_isDateTimeinZulu_Check=false #******* Event Service Conformance Rules ******** disableDistributeEvent_RequestIDPresent_Check=false disableDistributeEvent_MultipleVENIDFound_Check=false disableDistributeEvent_ResponseCodeSuccess_Check=false disableDistributeEvent_EventIDsUnique_ValidCheck=false disableDistributeEvent_EventStatusValid_ValidCheck=false disable_isDateTimeinZulu_Check=false disable_CreatedDateTimeWithinExpectedWindow_Check=false disableDistributeEvent_EventOrderValid_Check=false disableDistributeEvent_ActiveCurrentValueMatchCurrentPayload_Check=false disableDistributeEvent_NonActiveCurrentValueValid_Check=false disableDistributeEvent_PayloadFloatLimitValid_Check=false disableDistributeEvent_ActivePeriodDurationValid_Check=false disableDistributeEvent_EIResponsePresent_Check=false disableDistributeEvent_AllPreviousEvntPresent_Check=false disableCreatedEvent_ResponseCodeSuccess_Check=false disableCreatedEvent_EIResponsesValid_Check=false disable_EventIDValid_Check=false disableRequestEvent_isAllPrevCreatedEvntsRecvd_Check=false disableDistributeEvent_DTStartNotInEventSignalsOrEventBaselines_Check=false disableDistributeEvent_SignalTypeAndUnitValidForSignalName_Check=false disableDistributeEvent_SignalIDUnique_Check=false disableDistributeEvent_EventBaselineDurationValid_Check=false disableDistributeEvent_EINotificationSubElementOfActivePeriod_Check=false #******* Opt Service Conformance Rules ******** disableCreateOpt_isOptIDPresent_Check=false disableCancelOpt_isOptIDValid_Check=false disableCanceledOpt_ResponseCodeSuccess_Check=false disableCreatedOpt_ResponseCodeSuccess_Check=false disableCreateOpt_VavailabilityExcludedProperties_Check=false disableCreateOpt_VavailabilityMarketContextNotPresentIfEventIDPresent_Check=false #******* Report Service Conformance Rules ******** disableCreateReport_isRIDValid_Check=false disableUpdateReport_isRIDValid_Check=false disableRegister_Create_reportRequestIDZero_Check=false disableRegisterReport_DurationValid_Check=false disableUpdateReport_RequiredElementsAndReportName_Check=false disableHistoryUsageGreenbuttonGranularityReportBackDurationZero_Check=false disableCreateReport_reportRequestIDUnique_Check=false disableRegisterReport_ReportSpecifierIDAndRIDUnique_Check=false disable_UpdateReportDtstartValid_Check=false disable_UpdateReportDurationValid_Check=false disableRegisterReport_SamplingRateFoundInTelemertyMetadata_Check=false disableRegisterReport_ReportNameReportTypeItemBaseReadingTypeCombination_Check=false disableCreateReportReportSpecifierNotPresent_Check=false 28 disableCreatedPartyRegistration_ResponseCodeSuccess_Check=false disableCreatedReport_ResponseCodeSuccess_Check=false disableRegisteredReport_ResponseCodeSuccess_Check=false disableUpdatedReport_ResponseCodeSuccess_Check=false disable_UpdateReportLoadControlStateConsistentForAllIntervals_Check=false disableCreateReport_ReadingTypeValid_Check=false disableCreateReport_GranularityValid_Check=false disableRegisterReport_VenReportNameValid_Check=false disableRegisterReport_RequestIDMatchPreviousCreateReport_Check=false disableCreateReport_DoesNotContainExcludedReportIntervalProperties_Check=false disable_UpdateReportTelemetryUsageReportIntervalDuration_Check=false #******* Registration Service Conformance Rules ******** disable_ActiveRegistration_Check=false disableOadrCreatePartyRegistration_isRegIDAndVENIDMatch_Check=false disableOadrCreatePartyRegistration_PUSHTransportAddress_Check=false disableCancelPartyRegistration_IsRegistrationValid_Check=false #While disableOadrCreatePartyRegistration_PUSHTransportAddress_Check checks based on the PUSH or PULL setting in the properties file THe following property looks into the payload disableCreatePartyRegistration_HttpPullModelTrueFalse_Check=false disableOadrCreatedPartyRegistration_PULLPollFreq_Check=false disableCanceledPartyRegistration_ResponseCodeSuccess_Check=false disableCanceledReport_ResponseCodeSuccess_Check=false ########################################## ######### Internal Settings ############## ########################################## # Do not change SchemaFile=oadr_20b.xsd # Legacy settings for A profile Party_ID=Party_123 GroupID=Group_123 ########################################## ######### Self Test Control ############## ########################################## #For controlling self test automation #Not useful for running tests against real devices #Bypasses prompts if false Test_Prompt=true #Launches text editor with report(s) if true upon test completion Test_OpenLog=false #Auto runs self test counterpart if true Must have Default VM Argument set set -Dth.test=selftest for flag to work #Also displays console output in two DOS boxes while running tests Test_Runner=false #If set to true while Test_Runner is true uses larger DOS boxes that don't overlap for console output #Need to set correct path in runtest.bat shortcuts for this to work Test_RunnerShortcuts=false #BatchTestRunner in core.internal can be be used to run the test cases lists in the runner folder #Use instructions are in the comments of BatchTestRunner Test_Runner must be true for batch self test to work 29 #For full automation REGRESSION_MODE=true in Batch runner, set Test_Prompt=false comment out pause in runtest.bat, skip A_E1_0100_TH_VTN_1, prompt_050 answer is y then n Appendix C – Security Configuration Properties ########################################## ######## XMPP_SHA256_Security ############ ########################################## # Global settings for security behavior for all test cases # Controlled locally in security test cases # Use client certificates to login to the XMPP Server # If set to true SSL is used regardless of the security_Enabled property setting # If set to false, plain authentication is used XMPP_SASL_External=true #If set to true, all test cases will run with SSL Security_Enabled=true # Selection of TLS version TLSv1.2 (currently this only accepted cipher) TLS_Version=TLSv1.2 # OpenADR Alliance supported ciphers - Do not change Client_TLS12_Ciphers=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC _SHA256 Server_TLS12_Ciphers=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CB C_SHA256 # Controls the test harness verification of the servers X.509 CN field Recommend leave set to false vtnClientVerifyHostName=false venClientVerifyHostName=false # X.509 certificates # Paths are relative to the TH directory # IMPORTANT NOTE: Openfire has its own set of trust and keystores, as well as cipher settings VTN_trustStorePath=certs/NetworkFX/SHA256/truststore.jks VTN_truststorePassword=password VTN_needClientAuth=true VTN_keystorePath_HTTP= VTN_keystorePath_XMPP=certs/NetworkFX/SHA256/keystore_ven_222222222222.jks VTN_keystorePassword=password VTN_keystoreType=JKS VEN_trustStorePath=certs/NetworkFX/SHA256/truststore.jks VEN_truststorePassword=password VEN_needClientAuth=true VEN_keystorePath_HTTP= VEN_keystorePath_XMPP=certs/NetworkFX/SHA256/keystore_ven_111111111111.jks VEN_keystorePassword=password VEN_keystoreType=JKS 30 Appendix D – Updated Tests V1.1.0 - The following tests were updated to either correct or enhance the behavior: • • • Added support for wildcard deviceClass enumerations in conformance rule check Added explicit signalType check for “setpoint” in test cases E1_1025_TH_VEN and E0_6025_THV_VEN Added checks to N1_0020_TH_VTN and N0_5020_TH_VTN to verify that the reports required by the conformance rules are returned by the VEN V1.0.7 - The following tests were updated to either correct or enhance the behavior: • • SHA1, TLS 1.0 removed from all tests R1_3160…/R0_8160… were updated to place the intervals dStart time in ascending order The associated conformance rule for checking the report time against the first interval was updated to conform to the new check which looks for the first or last interval dStart time to match the report time Updated A_E3_0430_TH_VEN/A_E3_0435_TH_VEN for the number of intervals Added adjacent event tests case E1_1090 and E0_6090 Test G1_4012 added using event followed by poll for empty queue Expected result for Test case R1_3130_ and R0_8130_ has been modified Conformance rule check modified to flag use of oadrPoll in http push Added test cases G1_4030 and G0_9030 (both TH_VEN and TH_VTN) Deleted Test Case A_E0_0325/A_E1_0325 - MarketContext can have a wild card, so can't negative test • Modified test case E1_1065 and E0_6065 (both TH_VEN and TH_VTN) • Updated conformance rule check to verify dateTime is the same for all events in a oadrDistributeEvent payload • Modified conformance rule check to verify that "Z" is at the end of every dateTime string • Conformance rule check modified to validate ordering of intervals in reports • Confirmance rule check modified to validate ordering of dtstart relative to intervals in reports • Updated conformance rule CreateReport_GranularityValid_Check to confirm that granularity is less than reportBackDuration • dded tests for expired certificates: G1_4007_TH_VTN and G1_4007_TH_VEN • Updated tests N1_0020 and N0_5020 to expect oadrRequestEvent to be sent following the exchange of metadata reports during registration V1.0.6 - The following tests were updated to either correct or enhance the behavior: • • • • • • • • • • • • ECC certificates are now supported in the A and B Profile security tests isRIDValid_Check needs to check the RIDs match between oadrCreateReport and oadrUpdateReport oadrDistributeEvent push test cases contain an optional eiResponse which is not needed G0_9010/G1_4010 need to switch ciphers including RSA and ECC ciphers Fix warnings that are generated in Eclipse when compiling the JAVA source code 31 • • • • • • • • • • • • • • • • • • • • • • • • • • • • In the test specification, A_E3_0420_TH_VEN > TH_VEN showing oadrPoll It should be oadrRequestEvent Polling prompt for N1_0015_TH_VTN_1 test case needs to be updated Wrong resources for eiTarget element are being used for E1_1060_TH_VEN, E0_6060_TH_VEN_1 and E0_6065_TH_VEN_1 Prompts for test cases for overlapping events in same market contexts for A and B profile tests (A_E2_0530, A_E3_0530) need to be updated R1_3030_TH_VTN_1 Not Waiting for Update Report Developed a new test case to check VEN DUTs ability to handle multiple signals in same event using Event_003 Add tests E1_1055_TH_VTN_1.java, E1_1055_TH_VEN.java, E0_6055_TH_VTN.java and E0_6055_TH_VEN_1.java G0_9010_TH_VEN_1 needs to check sending of createEvent1 instead of createEvent2 A_Ex_0484 : Remove unused prompt in test specification and openadrresource_en_us.properties A_E2_0525 in Profile A and Profile B: modify to send one oadrEvent or two based on what's received isVENIDValid should consider veniD element not present and value not present the same E0_6040_TH_VEN_1 - Modify to handle VTNs that sends an event as soon as it is added A_E3_0655_TH_VEN prompt to be modified to give buffer time for user event entry Typos in PICS document need to be corrected N1_0060_TH_VTN_1 and N1_0065_TH_VTN_1 : control adding oadrRegisterReport to poll E1_1040 and E0_6040 : Modify prompts in test spec and code R1_3130_TH_VEN should check for presence of one of the following three datapoints: EnergyReal, PowerReal, and Pulsecount Typo in failure message of A_E3_0680_TH_VEN needs to be corrected R1_3150_TH_VEN Modify prompt and the test specification need to be updated to not include “Do not include report interval as this is a one shot report.” A_E2_0685_TH_VEN_1 second prompt has a typo that needs to be corrected R1_3020_TH_VEN Second Update Report should be sent after a minute A_E2_0527_TH_VEN_1 profile B and Profile A test case E2_0527 case definition and code needs to be updated to first send the event as mentioned in the prompt and then add another event after a minute Overlapping event conformance rule update – if an event ends and another starts at the same time, it should be considered non-overlapping event For the Test case A_E1_0200_TH_VTN, the SignalName cannot have 100 characters A_E3_0520_TH_VEN calls the prompt TestCase_0510_UIUserPromptText() Push Test Case: A_E0_0295_TH_VTN is to be removed from profile B Granularity in periodic report requests needs to consider value of samplingRate P1_2030_TH_VTN_1 does not Prerequisite verification, such as eiTargettype, marketContext and deviceClass Revised tests A_E1_0285, A_E0_0285, A_E2_0510, A_E3_0510 and deleted tests A_E2_0520, A_E3_0520, A_E2_530, and A_E3_0530 as the result of changes to rule regarding overlapping events V1.0.5 - The following tests were updated to either correct or enhance the behavior: • Numerous report test cases were made optional to implement the following changes to the B profile specification requirements (Refer to PICS for details): 32 VTNs and VENs are not required to demonstrate piggyback report requests or report cancellations o VENs are not required to request reports from the VTN The data points that a VEN must offer in its metadata report for usage reports has been relaxed Now only one data point must be supported from the set of energyReal, powerReal, or pulse Count Many report test cases were updated as follows: o Where the test harness acts as the VEN, all three data points are offered o Where the test harness acts as the VTN, report request are dynamically constructed based upon the data points offered by the VEN Prior test case versions assumed the presence of energyReal and PowerReal in the metadata repor Improvements were made to the database locking thanks to a contribution from Ulrich at Fujitsu Various minor changes to prompts and timeout settings in test cases Changes made to base test case to enable testCaseBufferTimeout in properties N0_5080 and N1_0080 updated to allow error code to be returned in payloads other than oadrResponse R0_8030 and R1_3030 we updated to remove the timeout to allow the initial report to be delivered a minute or so later A_E3_0680_TH_VEN modified to use oadrRequestEvent rather than oadrPoll E1_1040_TH_VEN modified to remove redundant optIn in the oadrCreatedEvent payload R1_3180 was fixed to generate unique requestID in second payload R1_3040_TH_VEN_1 and R1_3030 were revised to reflect the correct number of pending reports o • • • • • • • • • • V1.0.4 - The following tests were updated to either correct or enhance the behavior: • • • • • • • • • • • • Flush the queue after each test in a poll environment R?_?170: only require that the metadata reportRequestID appear in oadrCreatedReport list of pending reports R1_6060 has an error message that references oadrCreatedEvent instead of oadrCreateOpt Improve test logs to indicate when payloads are sent to incorrect service R0_8027/R1_3027: Allow or oadrUpdateReports R0_8030_TH_VTN conversion and delays adjustments A_E3_0680: change specifications to make this optional Pull test case N1_0080: log/console different from the test specification Add a property setting for jetty socket timeout values R1_3025_TH_VTN_1: the prompt_054 does not have a Yes/No buttons S0-303 VEN Reporting: update the sequence diagram in the test specification R1_3070: prompt_025 should be present in the TH_VTN V1.0.2 - The following tests were updated to either correct or enhance the behavior: • • • • R0_8170_TH_VTN fails because the test harness is only checking the st pending report instead of looping through all pending reports E0_6080_TH_VEN_1: Clicking Yes if real device causes it to hang – add a prompt to tell the DUT to send a DistributeEvent to start the expected sequence A_E0_0392_TH_VTN_1 – update the test code to deal with timing issues A_E0_280_TH_VTN - allow either oadrCreatedEvent or oadrCreateOpt 33 • • • • • • • • • • N0_5060_TH_VTN_1 – updated to only check for an empty registration ID as an indication to initiating a new registration HTTP Header ContentLength check is case sensitive – update test harness to check for these header fields case insensitive E1_1080/E0_6080: Correct response when includes invalid VTN ID – now deals with both tests correctly so they both pass R1_3160_TH_VEN – updated to deal with successful completion R1_3150_TH_ VEN – updated to deal with successful completion R1_3120_TH_VEN – added a prompt to notify the user what is expected A_E0_0280_TH_VTN_1 – update the test code to deal with timing issues E0_0190_TH_VTN_1 – update the test code to deal with timing issues A_E0_0180_TH_VTN_1 – update the test code to deal with timing issues E1_1040/e0_6040 – now supports or CreatedEvents V1.0.1 - The following tests were updated to either correct or enhance the behavior: • • • • • • • • • • • • • • • • • • • • • • R*_3170 test case VEN reporting - metadata report request needs update to reflect latest spec change – receipt of a metadata report no longer implicitly cancels requests for a periodic metadata report R*3180 test case does not handle multiple oadrupdateReport payloads R*3140: Test harness emits invalid oadrReportRequest if the DUT generates an incorrect metadata report R*3150: Test harness emits invalid oadrCreateReport if the DUT sends an incorrect TELEMETRY_STATUS Use of directory names with blanks in Linux causes FileNotfoundException E0_0240_revised_TH_VTN - verify its package location – should be in package testcases.push.event.ven E?_?060 Prompt fix – VTN Prompt_13 should be displayed before CreateOpt G0_9010 Missing Prompt – update test to display Prompt_010 in the correct location Responses to invalid oadrCancelRegistation should be handled If content-type is missing in the HTTP 200 header, the test will fail Update to check for this condition Enhance default handler for oadrregisterreport to not add oadrreportRequest element in Oadrregisteredreport default payload A_E2_0680 should have size == check (line number 88 in java) N1_0015 missing poll – move the prompt from VEN to VTN Registration check on report test cases – add active registration check on top of each report test case E?_?040 support for payload – add support to handle or CreatedEvents N?_?060 Error Message Update to remove "validation error" E?_?070 460 check – include the error in eventResponse instead of eiResponse A_E0_0360_TH_VTN_1.java has failure condition flipped A_E0_0240_TH_VTN_1.java - move the VTNService.StartVTNService(); before post E0_6070 Improve error message – provide more informative message E0_6060 Improve error message – provide more informative message G0_9010_th_vtn listener timing – start listening for requests earlier 34 ... the OpenADR 2.0b Profile Specification Devices that implement OpenADR 2.0b must pass a certification test The requirements for the certification test are spelled out in the OpenADR 2.0b Certification. .. OpenADR 2.0b Certification Test Specification Each test case defined in the test specification is implemented in the OpenADR 2.0b Certification Test Harness All OpenADR 2.0b interactions are between... test For instance, when testing a VEN Push implementation, the test harness will play the role of a VTN Push implementation This document describes how to use the OpenADR 2.0b Certification Test