|Document Name: |Abstract Test Suite and Implementation eXtra Information for Test |
|Date: |2024 March 1 |
|Abstract: |Abstract Test Suite and Implementation eXtra Information for Test consists of :<br/>- Definition of the Abstract Protocol Tester (APT)<br/>- Definition of TTCN-3 test architecture<br/>- Development of TTCN-3 test suite, e.g. naming conventions, code documentation, test case structure.<br/>- IXIT proforma;<br/> |
|Template Version:23 February 2015 (Do not modify) |Template Version:23 February 2015 (Do not modify) |
This Specification is provided for future development work within oneM2M only. The Partners accept no liability for any use of this Specification.
<br/>This Specification is provided for future development work within oneM2M only. The Partners accept no liability for any use of this Specification.
The present document has not been subject to any approval process by the oneM2M Partners Type 1. Published oneM2M specifications and reports for implementation should be obtained via the oneM2M Partners' Publications Offices.
...
...
@@ -43,7 +47,7 @@ More information about oneM2M may be found at: http//www.oneM2M.org
[5.4.4.1 Usage for Automated AE Testing](#5441-usage-for-automated-ae-testing)
[5.4.4.2 Upper Tester Control Primitives](#5442-upper-tester-control-primitives)
[5.4.4.2.2 UtTrigger and UtTriggerAck Primitives](#54422-uttrigger-and-uttriggerack-primitives)
[5.4.4.2.3 Control Communication Protocol](#54423-control-communication-protocol)
[5.4.4.2.4 Control Message Serialization](#54424-control-message-serialization)
[7.2.1 General guidelines](#721-general-guidelines)
[7.2.2 oneM2M specific TTCN-3 naming conventions](#722-onem2m-specific-ttcn-3-naming-conventions)
[7.2.3 Usage of Log statements](#723-usage-of-log-statements)
[7.2.4 Test Case (TC) identifier](#724-test-case-tc-identifier)
[7.3 IXIT](#73-ixit)
[8 TTCN-3 Verifications](#8-ttcn-3-verifications)
[Annex A (normative): TTCN-3 library modules](#annex-a-normative-ttcn-3-library-modules)
[A.1 Electronic annex, zip file with TTCN-3 code](#a1-electronic-annex-zip-file-with-ttcn-3-code)
[Annex B (informative): Bibliography](#annex-b-informative-bibliography)
[History](#history)
# 1 Scope
The present document contains the Abstract Test Suite (ATS) for oneM2M as defined in oneM2M TS-0001 <ahref="#_ref_1">[1]</a> and oneM2M TS-0004 <ahref="#_ref_2">[2]</a> in compliance with the relevant requirements and in accordance with the relevant guidance given in ISO/IEC 96467 <ahref="#_ref_5">[5]</a>.
The present document contains the Abstract Test Suite (ATS) for oneM2M as defined in oneM2M TS-0001 <ahref="#_ref_1">[1]</a> and oneM2M TS-0004 <ahref="#_ref_2">[2]</a> in compliance with the relevant requirements and in accordance with the relevant guidance given in ISO/IEC 9646-7 <ahref="#_ref_i_6">[i.6]</a>.
The objective of the present document is to provide a basis for conformance tests for oneM2M products giving a high probability of interoperability between different manufacturers' equipment.
The ISO standard for the methodology of conformance testing (ISO/IEC 96461 <ahref="#_ref_3">[3]</a> and ISO/IEC 96462 <ahref="#_ref_4">[4]</a>) as well as oneM2M TS-0015 Testing Framework <ahref="#_ref_i.2">[i.2]</a> are used as a basis for the test methodology.
The ISO standard for the methodology of conformance testing (ISO/IEC 9646-1 <ahref="#_ref_i_4">[i.4]</a> and ISO/IEC 96462 <ahref="#_ref_i_5">[i.5]</a>) as well as oneM2M TS-0015 <ahref="#_ref_i.2">[i.2]</a> are used as a basis for the test methodology.
# 2 References
...
...
@@ -134,13 +83,10 @@ The following referenced documents are necessary for the application of the pres
-<aname="_ref_3">[3]</a> ISO/IEC 9646-1 (1994): "Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 1: General concepts".
-<aname="_ref_4">[4]</a> ISO/IEC 9646-2 (1994): "Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 2: Abstract Test Suite specification".
-<aname="_ref_5">[5]</a> ISO/IEC 9646-7 (1995): "Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 7: Implementation Conformance Statements".
-<aname="_ref_6">[6]</a> ETSI ES 201 873-1 (V4.5.1): "Methods for Testing and Specification (MTS); The Testing and Test Control Notation version 3; Part 1: TTCN-3 Core Language".
-<aname="_ref_7">[7]</a> oneM2M TS-0018: "Test Suite Structure and Test Purposes".
-<aname="_ref_3">[3]</a> ETSI ES 201 873-1 (V4.5.1): "Methods for Testing and Specification (MTS); The Testing and Test Control Notation version 3; Part 1: TTCN-3 Core Language".
-<aname="_ref_4">[4]</a> oneM2M TS-0018: "Test Suite Structure and Test Purposes".
-<aname="_ref_8">[8]</a>3GPP TS 29.122; T8 reference point for Northbound APIs (Release 15)
-<aname="_ref_5">[5]</a> 3GPP TS 29.122; T8 reference point for Northbound APIs (Release 15)
## 2.2 Informative references
...
...
@@ -153,13 +99,16 @@ The following referenced documents are not necessary for the application of the
> NOTE: Available at [http://www.onem2m.org/images/files/oneM2M-Drafting-Rules.pdf](http://www.onem2m.org/images/files/oneM2M-Drafting-Rules.pdf).
-<aname="_ref_i.3">[i.3]</a> oneM2M TS-0025: "Definition of product profiles".
-<aname="_ref_i_4">[i.4]</a> ISO/IEC 9646-1 (1994): "Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 1: General concepts".
-<aname="_ref_i_5">[i.5]</a> ISO/IEC 9646-2 (1994): "Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 2: Abstract Test Suite specification".
-<aname="_ref_i_6">[i.6]</a> ISO/IEC 9646-7 (1995): "Information technology - Open Systems Interconnection - Conformance testing methodology and framework - Part 7: Implementation Conformance Statements".
# 3 Definition of terms, symbols and abbreviations
## 3.1 Terms
For the purposes of the present document, the terms given in ISO/IEC 96461 <ahref="#_ref_3">[3]</a>, ISO/IEC 96467 <ahref="#_ref_5">[5]</a> and oneM2M TS-0015 <ahref="#_ref_i.2">[i.2]</a> apply.
For the purposes of the present document, the terms given in ISO/IEC 9646-1 <ahref="#_ref_i_4">[i_4]</a>, ISO/IEC 9646-7 <ahref="#_ref_i_6">[i.6]</a> and oneM2M TS-0015 <ahref="#_ref_i.2">[i.2]</a> apply.
## 3.2 Symbols
...
...
@@ -171,7 +120,8 @@ For the purposes of the present document, the following abbreviations apply:
`AE Application Entity`
`APT Abstract Protocol Tester`
`APT Abstract Protocol Tester`
`ASP Abstract Service Primitives`
`ATM Abstract Test Method`
`ATS Abstract Test Suite`
`CoAP Constrained Application Protocol`
...
...
@@ -180,13 +130,14 @@ For the purposes of the present document, the following abbreviations apply:
@@ -206,7 +157,7 @@ The key words "Shall", "Shall not", "May", "Need not", "Should", "Should not" in
# 5 Abstract Test Method (ATM)
## 5.1 Abstract protocol tester
## 5.1 Abstract protocol tester (APT)
An abstract protocol tester (APT) is a process that provides behaviours for testing an IUT by emulating a peer IUT at the same layer, and enabling to address a single test objective.
APTs used by the oneM2M test suite are described in figure 5.1-1. The test system will simulate valid and invalid protocol behaviour, and will analyse the reaction of the IUT.
...
...
@@ -243,14 +194,13 @@ Following this recommendation the oneM2M tester architecture comprises a non-pla
> NOTE: However, it can be implemented in a semi-independent manner, which will minimize the dependency to those elements.
>
> NOTE: However, it can be implemented in a semi-independent manner, which will minimize the dependency to those elements.
**Figure 5.3-1: High level oneM2M Test Architecture**
-**oneM2M****TTCN****-3****Abstract****Test Suite:** the test suite is platform independent, and it is the cornerstone of the architecture. It allows a complete decoupling between the test suite and the rest of the test system. The test suite is composed of a complete set of test cases covering oneM2M requirements specified by oneM2M TS0001 <ahref="#_ref_1">[1]</a> and oneM2M TS-0004 <ahref="#_ref_2">[2]</a>.
-**oneM2M System Adaptor****:** this is the platform dependent part that includes adaptors and codecs (out of the scope of the present document). This part of the architecture definition depends on the specific platform (e.g. Windows or Linux) and test tool on which the tester is going to run.
-**oneM2M System Adaptor****:** this is the platform dependent part that includes adaptors and codecs (out of the scope of the present document). This part of the architecture definition depends on the specific platform (e.g. Windows® or Linux®) and test tool on which the tester is going to run.
Figure 5.3-2 shows the oneM2M TTCN-3 test architecture design used for the oneM2M ATS. The Test Suite needs to interact with the System Adaptor to implement the collection of TTCN-3 test cases that are intended to be used to test the oneM2M IUTs.
...
...
@@ -310,11 +260,11 @@ Two primitives are currently defined for these ports indicated as table 5.4.1-1:
1. The M2MRequestPrimitive - to send or receive oneM2M messages to/from the IUT. Depending on the IUT to be tested:
1. If the IUT is an AE, these messages are either received or sent by the tester which is associated with the CSE role through the mcaPortIn or the mcaPort respectively.
1. If the IUT is a CSE, these messages are either sent or received by the tester when it plays the AE role through the mcaPort or the mcaPortIn respectively, or sent or received by the tester when it plays the CSE role through the mccPort or the mccPortIn respectively.
- If the IUT is an AE, these messages are either received or sent by the tester which is associated with the CSE role through the mcaPortIn or the mcaPort respectively.
- If the IUT is a CSE, these messages are either sent or received by the tester when it plays the AE role through the mcaPort or the mcaPortIn respectively, or sent or received by the tester when it plays the CSE role through the mccPort or the mccPortIn respectively.
1. The M2MResponsePrimitive - to send or receive oneM2M messages to/from the IUT. Depending on the IUT to be tested:
1. If the IUT is an AE, these messages are either sent or received by the tester which is associated with the CSE role through the mcaPortIn or the mcaPort respectively.
1. If the IUT is a CSE, these messages are either sent or received by the tester when it plays the CSE role through the mccPortIn or the mccPort respectively, sent or received by the tester when it plays the AE role through the mcaPortIn or mcaPort respectively.
- If the IUT is an AE, these messages are either sent or received by the tester which is associated with the CSE role through the mcaPortIn or the mcaPort respectively.
- If the IUT is a CSE, these messages are either sent or received by the tester when it plays the CSE role through the mccPortIn or the mccPort respectively, sent or received by the tester when it plays the AE role through the mcaPortIn or mcaPort respectively.
Both primitives contain another parameters that permits to dynamically configure the test adaptor for every single sending. These parameters are:
...
...
@@ -327,16 +277,17 @@ Both primitives contain another parameters that permits to dynamically configure
**Table 5.4.1-1: Mapping of TTCN-3 Primitives to oneM2M Service Primitives**<aname="table_5.4.1-1"></a>
@@ -428,20 +379,83 @@ Table 5.4.4.2.2-1 defines UtTrigger and UtTriggerAck primitives including oneM2M
**Table 5.4.4.2.2-1: UtTrigger and UtTriggerAck Primitive**<aname="table_5.4.4.2.2-1"></a>
|Ut Control Primitive |Mapping to oneM2M data types |Description |Reference |Triggering Message |HTTP message |
|-|-|-|-|-|-|
|_UtTrigger_ _Primitive_ |_requestPrimitive_ |ONLY essential parameters included for certain test case<br/><br/>See note 1 |oneM2M<br/>TS-0004 <ahref="#_ref_2">[2]</a> |**EXAMPLE****1:**<br/>If the test objective is to test **_"Test System triggers_****_IUT_****_to execute a test case for creation of <_****_AE_****_> with labels attribute under a_****_CSEBase_****_resource_** ", then the triggering message would be serialized as following. |**EXAMPLE****1:**<br/>If the test objective is to test **_"Test System triggers_****_IUT_****_to execute a test case for creation of <_****_AE_****_> with labels attribute under a_****_CSEBase_****_resource_** ", then the triggering message would be serialized as following. |
|_UtTrigger_ _Primitive_ |_requestPrimitive_ |ONLY essential parameters included for certain test case<br/><br/>See note 1 |oneM2M<br/>TS-0004 <ahref="#_ref_2">[2]</a> |**Request**<br/>{<br/>"m2m:rqp" :{<br/> "op": 1, //indicate CREATE operation<br/> "ty": 2, //indicate AE resource type<br/> "to": {TEST\_SYSTEM\_ADDRESS},<br/> "pc": {<br/> "m2m:ae": {<br/> "lbl":"UNINITIALIZED" //indicate that attribute labels needs to be included<br/> },<br/> }<br/> "rvi": "2a"<br/>}<br/>} |**Request**<br/>**POST** /{SUT\_UT\_APPLICATION\_URL} HTTP/1.1<br/>**Host** : {SUT\_IP\_ADDRESS:PORT}<br/>**Content-Length** : {PAYLOAD\_LENGTH}<br/>**Content-Type** : **application/****json**<br/><br/>{<br/>"m2m:rqp" :{<br/> "op": 1, //indicate CREATE operation<br/> "ty": 2, //indicate AE resource type<br/> "to": {TEST\_SYSTEM\_ADDRESS},<br/> "pc": {<br/> "m2m:ae": {<br/> "lbl":"UNINITIALIZED" //indicate that attribute labels needs to be included<br/> }<br/> },<br/> "rvi": "2a"<br/>}<br/>} |
|_UtTrigger_ _Primitive_ |_requestPrimitive_ |ONLY essential parameters included for certain test case<br/><br/>See note 1 |oneM2M<br/>TS-0004 <ahref="#_ref_2">[2]</a> |**EXAMPLE****2:**<br/>If the test objective is to test "**_Test System triggers_****_IUT_****_to execute a test case for delete of a <_****_AE_****_> resource._** ", then the triggering message would be serialized as following. |**EXAMPLE****2:**<br/>If the test objective is to test "**_Test System triggers_****_IUT_****_to execute a test case for delete of a <_****_AE_****_> resource._** ", then the triggering message would be serialized as following. |
|_UtTrigger_ _Ack_ _Primitive_ |_responsePrimitive_ |ONLY responseStatusCode attribute included<br/><br/>See note 2 |oneM2M TS-0004 <ahref="#_ref_2">[2]</a> |**Response**<br/>{<br/> "m2m:rsp": {<br/> "rsc": 2000<br/> }<br/><br/>}<br/>For any triggering response, it only contains a response status code, and the response status code for the triggering operation can only be set to either 2000 (OK) or 4000 (BAD\_REQUEST) according to the rules for triggering operations. |**Response**<br/>HTTP/1.1 200 OK<br/>X-M2M-RSC: 2000 |
|NOTE 1: Additional rules defined in table 5.4.4.2.2-3 are also applied.<br/>NOTE 2: Attribute response status code is defined at table 5.4.4.2.2-3. |NOTE 1: Additional rules defined in table 5.4.4.2.2-3 are also applied.<br/>NOTE 2: Attribute response status code is defined at table 5.4.4.2.2-3. |NOTE 1: Additional rules defined in table 5.4.4.2.2-3 are also applied.<br/>NOTE 2: Attribute response status code is defined at table 5.4.4.2.2-3. |NOTE 1: Additional rules defined in table 5.4.4.2.2-3 are also applied.<br/>NOTE 2: Attribute response status code is defined at table 5.4.4.2.2-3. |NOTE 1: Additional rules defined in table 5.4.4.2.2-3 are also applied.<br/>NOTE 2: Attribute response status code is defined at table 5.4.4.2.2-3. |NOTE 1: Additional rules defined in table 5.4.4.2.2-3 are also applied.<br/>NOTE 2: Attribute response status code is defined at table 5.4.4.2.2-3. |
**Table 5.4.4.2.2-2: Definition of special Upper Tester commands**<aname="table_5.4.4.2.2-2"></a>
...
...
@@ -470,17 +484,24 @@ The rules for defining UtTrigger and UtTriggerAck primitives are:
**Table 5.4.4.2.2-3: Definition of ResponseStatusCode for UtTriggerAck primitive**<aname="table_5.4.4.2.2-3"></a>
|Response Status Code Description |Response Status Code Value |Interpretation |
|-|-|-|
|OK |2000 |The SUT receives successfully the triggering message from Test System |
|BAD\_REQUEST |4000 |The SUT does not interpret correctly the UtTrigger primitive |
|NOTE: Only above two response status codes are allowed to use in UtTriggerAck primitive. |NOTE: Only above two response status codes are allowed to use in UtTriggerAck primitive. |NOTE: Only above two response status codes are allowed to use in UtTriggerAck primitive. |
Protocol used for proceeding communications between Test System and Upper Tester Application is designated to the Hypertext Transfer Protocol (HTTP) protocol owning it is an application protocol that is widely supported by most all IoT devices and various intrinsic features such as persistent connection, ease of programming, flexibility, etc.
Protocol used for proceeding communications between Test System and Upper Tester Application is designated to the Hypertext Transfer Protocol (HTTP) protocol owning it is an application protocol that is widely supported and various intrinsic features such as persistent connection, ease of programming, flexibility, etc.
##### 5.4.4.2.4 Control Message Serialization
...
...
@@ -503,31 +524,73 @@ The Tester test component includes a set of ports, timers and variables that are
|vc\_aeSimu |variable |default |Reference to the default behaviour for an AeSimu component |
|vc\_cseSimu |variable |default |Reference to the default behaviour for an CseSimu component |
|vc\_primitiveContentRetrievedResource |variable |PrimitiveContent |Latest content of a RETRIEVE operation |
|vc\_myInterfaces |variable |Interfaces |Parameters for the ports of the given component:<br/>Port (mcaPort, mcaPortIn, mccPort, mccPortIn)<br/>Host (SUT IP address :port)<br/>Protocol binding<br/>Serialization |
Note that vc\_aeSimu and vc\_cseSimu are not common to the other defined test components, but those variables are required in Tester for the correct activation/deactivation of default behaviours.
|IutParameters |PX\_TCONFIG\_IUT |Time to configure IUT after a requested action |10.0 |
|TesterParameters |PX\_TS\_AE1 |AE1 component settings |aeIdStem = ""<br/>appId = "NMyApp1Id"<br/>mcaPort and mcaPortIn settings which include per port the following info:<br/>Binding:<br/>- bindingProtocol<br/> - bindingDesc:<br/> - tsAddress<br/> - localPort<br/> - sutAddress<br/> - remotePort<br/>Serialization |
|TesterParameters |PX\_TS\_AE1 |AE1 component settings |aeIdStem = ""<br/>appId = "NMyApp1Id"<br/>mcaPort and mcaPortIn settings which include per port the following info:<br/>Binding:<br/>- bindingProtocol<br/>- bindingDesc:<br/> - tsAddress<br/> - localPort<br/> - sutAddress<br/> - remotePort<br/>Serialization |
|TesterParameters |PX\_TS\_AE2 |AE2 component settings |aeIdStem = ""<br/>appId = "NMyApp2Id"<br/>mcaPort and mcaPortIn settings which include per port the following info:<br/>Binding:<br/> - bindingProtocol<br/> - bindingDesc:<br/> - tsAddress<br/> - localPort<br/> - sutAddress<br/> - remotePort<br/>Serialization |
|TesterParameters |PX\_TS\_CSE1 |CSE1 component settings |cseName = "CSE1\_NAME"<br/>cseId = "/CSE1\_ID"<br/>cseResourceId = "CSE1\_RESOURCE\_ID"<br/>spId = "//onem2m.org"<br/>supportedResourceType = {int1, int2, int3, int16}<br/>mcaPort, mcaPortIn, mccPort and mccPortIn settings which include per port the following info:<br/>Binding:<br/> - bindingProtocol<br/> - bindingDesc:<br/> - tsAddress<br/> - localPort<br/> - sutAddress<br/> - remotePort<br/>Serialization |
|TesterParameters |PX\_TS\_CSE2 |CSE2 component settings |cseName = "CSE2\_NAME"<br/>cseId = "/CSE2\_ID"<br/>cseResourceId = "CSE2\_RESOURCE\_ID"<br/>spId = "//onem2m.org"<br/>supportedResourceType = {int1, int2, int3, int16}<br/>mcaPort, mcaPortIn, mccPort and mccPortIn settings which include per port the following info:<br/>Binding:<br/> - bindingProtocol<br/> - bindingDesc:<br/> - tsAddress<br/> - localPort<br/> - sutAddress<br/> - remotePort<br/>Serialization |
