|Abstract: |< An abstract of the document and information that may be used in subsequent electronic searches> |
|Document Number |TR-0070-V-0.0.1 |
|Document Name: |Enhanced filtering and Queries |
|Date: |<mark>2023-06-15</mark> |
|Abstract: |This TR provides an overview about the current status of querying and filtering in oneM2M, formulates new use cases, and provides an overview about relevant query languages. |
|Template Version: January 2020 (do not modify) | |
...
...
@@ -65,32 +65,23 @@ NO REPRESENTATION OR WARRANTY IS MADE THAT THE INFORMATION IS TECHNICALLY ACCURA
[B.1.1 First subdivided clause of the annex](#b11----first-subdivided-clause-of-the-annex)
`EXAMPLE: The present document provides the necessary adaptions to the endorsed document.`
<mark>The Scope shall not contain requirements.</mark>
# 1 Scope
This document presents possible requirements for an enhanced filtering and query mechanism as well as a discussion about possible solutions.
# 2 References
<mark>The following text block applies.</mark>
References are either specific (identified by date of publication and/or edition number or version number) or nonspecific. For specific references,only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.
## 2.1 Normative references
<mark>As a Technical Report (TR) is entirely informative it shall not list normative references.</mark>
The following referenced documents are necessary for the application of the present document.
<mark>Clause 2.2 shall only contain informative references which are cited in the document itself.</mark>
The following referenced documents are not necessary for the application of the present document but they assist the user with regard to a particular subject area.
The key words "Shall", "Shall not", "May", "Need not", "Should", "Should not" in this document are to be interpreted as described in the oneM2M Drafting Rules <ahref="#_ref_i.1">[i.1]</a>
# 5 Introduction
# 5 User defined clause(s) from here onwards
<Text>
This clause provides an overview about the current filter am query methods in oneM2M.
## 5.1 Current Method of Filtering in oneM2M
## 5.1 User defined subdivisions of clause(s) from here onwards
<Text>
The current method for providing filters and queries in oneM2M requests is currently done by providing **_FILTER CRITERIA_** in a request. This is a complex attribute that contains a structure of matching conditions attributes and filter handling attributes that can be applied for certain RETRIEVE and DISCOVERY operations. The detailed description and the applicability is specified in TS-0001, clause 8.1.2 and table 8.1.2-2.
<mark>The following text is to be used when appropriate:</mark>
To summarise its use:
- Matching conditions are evaluated against resources. When the evaluation is true, the matched resources are part of the matching result.
- The filter handling conditions provide additional input used to determine the filtering result (e.g. maximum number of resources to be included in the filtering result).
- The filtering result may be composed of one or more resources and determines the target resources for Discovery-based operations.
# <mark>Proforma copyright release text block</mark>
<mark>This text box shall immediately follow after the heading of an element (i.e. clause or annex) containing a proforma or template which is intended to be copied by the user. Such an element shall always start on a new page.</mark>
Further details are also provided in TS-0001 [[1](#_ref_1)] , clause 10.2.6 "Discovery-related Procedures".
The *matching conditions* is a set of attributes that specify for some common and universal resource attributes matching (or comparison) a value and the kind of comparison that should be applied if present. For example, the value of the *createdBefore* attribute is compared against the *creationTime* attribute during matching, and it will be matched if a resource's *createdTime* attribute is before the timestamp specified in *createdBefore*. Through the specific nature of the attributes it is possible to compare against the respective attributes in resources not only for equality, but also for smaller, larger, before, after, etc.
Besides the pre-defined set of **_FILTER CRITERIA_** attributes it is possible to compare all these attributes attributes, and also the custom attributes in <flexContainer> specialisations, against equality. However, equality is the only means of comparison for most of oneM2M's attributes.
It is possible to add multiple queries and to logical combine them with setting the *filterOperations***_FILTER CRITERIA_** attribute. This attribute specifies the logical operator that is used for all the given filter expressions. No sub-queries are allowed, eg. to mix different boolean operations. Negation of query results is also not possible.
In addition **_FILTER CRITERIA_***filter handling* attributes are used to limit the number of filter result, specifying an offset in the filter result, or limit the search depth in the resource tree.
## 5.2 Semantic Discovery and Query
Both semantic discovery and query are not part of this report because they use a different method for handling queries, but the should be shortly summarised here.
Semantic queries are different from the filter method described in clause [5.1](#51-current-method-of-filtering-in-onem2m). The filtering methods described in that clause are part of the request handling. A result set of resources or references is created or filtered afterwards according to the given configuration in the **_FILTER CRITERIA_** request attribute.
For semantic discovery and query two different methods are defined that use semantic service components to compute the result set according not only by just comparing queries and values, but by deriving explicit and implicit information from the semantic information that are provided together with the resources as well as from external information that are semantically referenced.
Resources may have child resources of type <semanticDescriptor>, which contain additional semantic information that may describe resource resp. the data contained in that resource further. For example, the data from a temperature sensor may be annotated this way to explain that data producing device is a thermostat used indoor in an industrial context, and that the data is in degree Celsius.
The differences between semantic discovery and semantic query operations are:
- A *semantic discovery* uses the information stored in the <semanticDescriptor> child-resources together with other information to build a result set for a semantic expression. For example, "discover and return all the temperature data from devices that are used indoor".
- A *semantic query* is used to send and process a semantic expression by the semantic service component of the CSE. It may use all the available semantic information from the resources in the resource tree. The result, however, is not a set of resources or resource references, but a semantic result string (usually RDF triples).
The query in both cases is a SPARQL query.
## 5.3 Shortcomings of the Current Filtering Method
The current filtering method described in clause [5.1](#51-current-method-of-filtering-in-onem2m) has limitations that is preventing to execute more sophisticated RETRIEVE and DISCOVERY requests in the CSE. This means that an AE that want to make use of this feature needs to implement them itself, which in case of a restricted device is not always easy or even possible.
To summarise the limitations:
- Comparison operations such as "greater", "less", "earlier", "later" are defined for only a limited set of common and universal attributes.
- Other attributes, including <flexContainer> custom attributes can only be compared for equality.
- Filtering of individual attributes in complex attributes is not possible.
- Filtering on elements of a list is not possible.
- Filter expressions that contain multiple attributes can only be combined via "AND", "OR", or "XOR" logical operators.
- Filter expressions cannot be negated (logical operator "NOT")
## 5.4 Shortcomings of Evaluation Modes of <action> and <dependency> Resource Types
In addition to the queries and filters provided in requests some resources also use a similar method to perform their tasks. The <action> and <dependency> resources provide a simple method to react on changes in other resources without going through an application entity to perform the necessary requests. Both resource types have attributes that contain an evaluation criteria that is used for the comparison as soon as an update happens to a target resource. This is sufficient for simple scenarios, more sophisticated scenarios, for example evaluating multiple or complex attributes is not possible.
To summarise the limitations:
- It is only possible to evaluate a single "top-level" attribute of a subject resource.
- The comparison operations are limited to a few basic comparison operations (e.g. "in" to test for the existence in a list is not possible).
# 6 Requirements for IoT & oneM2M Resource Querying and Filtering
This section provides discussions on existing an new requirements for querying and filtering.
## 6.1 Existing Requirements in TS-0002
The following requirements regarding resource querying and filtering can be found in oneM2M TS-0002 [[2](#_ref_2)].
**Table 6.1-1: List of existing requirements in TS-0002**
| OSR-083<br/> See REQ-2015- 0593R02 | The oneM2M System shall be able to filter information from oneM2M Devices for a given set of parameters. | Not implemented |
| ONT-018<br/> See REQ-2018- 0057R01 | The oneM2M system shall support semantic query and discovery across heterogeneous ontologies including support of automatic ontology mapping and semantic reasoning. | 4 |
| ONT-018<br/> See REQ-2018- 0057R01 | The oneM2M system shall support semantic query and discovery across heterogeneous ontologies including support of automatic ontology mapping and semantic reasoning. | 4 |
The first requirement implies a technical filter or query while the other requirements are related to semantic querying and discovery.
## 6.2 New Use Cases
The following is a collection of new use cases that are interesting for IoT application as well as for additional functionality for common services within the CSE.
### 6.2.1 Searching for an IMEI in *label*
An application entity needs to store extra meta tags in a resource, e.g. the IMEI number of the hosting devices, and other tags. For this, the application entity uses the *label* attribute. Another application entity that wants to retrieve resources by the IMEI number wants to query for these resources by only the single tag in the *label* attribute.
**Requirement(s)**
- Filter on a single tag in a list attribute.
### 6.2.2 Monitor Temperature Values
An application entity wants to retrieve only those <flexContainer> [temperature] specialisations where the *currentTemperature* attribute is above a certain value.
**Requirement(s)**
- Filter on a custom attribute that is greater than a comparison value.
### 6.2.3 Search for a Tag in *label* or Where Multiple Attributes are Over a Specific Values
An application entity needs to retrieve resources that have specific tag added to a *label* attribute, OR which attributes *a* and *b* are BOTH over specific values.
**Requirement(s)**
- Filter on a single tag in a list attribute.
- Filter on multiple attributes that are greater than specific values.
- Combine comparison expressions with OR and AND.
### 6.2.4 Search for Subscriptions that Monitor a Certain Child-Resource Type
An application entity needs to retrieve all the <subscription> resources that monitor <contentInstance> resources. This involves a comparison to the *childResourceType* list attribute of the *eventNotificationCriteria* complex attribute.
**Requirement(s)**
- Filter on an attribute that is part of a complex type (child-attribute of another attribute).
### 6.2.5 Reacting on Multiple Attribute Change Using <action> Resources
An application entity wants to install an <action> resource that monitors a flexContainer> specialisation with attributes *a* and *b*. If both attributes in that resource rises beyond a certain value, the <action> resource shall set an attribute in another resource to a value.
**Requirement(s)**
- Be able to evaluate one or more attributes in one expression.
- Be able to evaluate complex type attributes.
- Support the same comparison operators as in other queries (e.g. "in")
---
<mark>TODO: Use case for parent- or child-resource attributes</mark>mark
## 6.3 Requirements
The following requirements are derived from the comparison above. In addition, some non-function requirements are formulated.
### 6.3.1 Functional Requirements
#### Compare Custom and Resource Specific Attributes Using Comparison Operators Other than only Equality
- Support the Same Functionality for <action> and <dependency> Resources
### 6.3.2 Non-Functional Requirements
- Size
==TODO==
# 7 Query Languages Comparisons
Data query languages are syntaxes designed specifically for querying and retrieving data from databases or data repositories. These query languages provide a simplified way to interact with data, making it easier for applications to retrieve the information they need. Here is a short summary of relevant data query languages:
1.**SQL (Structured Query Language)**: SQL is the most widely used query language for relational databases. It allows users to retrieve, manipulate, and manage data stored in tables. SQL uses a declarative approach, where applications specify what data they want to retrieve rather than how to retrieve it. SQL supports various operations like SELECT, INSERT, UPDATE, and DELETE, enabling users to perform complex queries and aggregations on structured data. Relevant for filtering is the WHERE clause where filter expressions resp. query statements can be formulated.
2.**XQuery**: XQuery is a query language specifically designed for querying XML (eXtensible Markup Language) data. It allows users to extract and manipulate data from XML documents using a variety of query and transformation capabilities. XQuery supports XPath, a language for navigating XML structures, making it suitable for querying and transforming complex XML data. ==examples==
3.**JSON Query** (also known as JSONPath) is a syntax used to query and extract specific data from JSON documents provides a way to navigate through the hierarchical structure of JSON data and select elements based on specific criteria. It uses a path-like syntax to traverse the JSON structure and select elements or values.
These simple data query languages provide efficient and straightforward methods for retrieving and manipulating data from databases or data sources. They offer different syntaxes and capabilities tailored to specific data formats and use cases, simplifying data retrieval and empowering users to work with data more effectively.
<mark>Notwithstanding the provisions of the copyright clause related to the text of the present document, oneM2M grants that users of the present document may freely reproduce the <proformatype> proforma in this {clause|annex} so that it can be used for its intended purposes and may further publish the completed <proformatype>.</mark>
<mark><PAGE BREAK></mark>
# Annexes
## <mark>Annexes</mark>
<mark>Each annex shall start on a new page (insert a page break between annexes A and B, annexes B and C, etc.).<mark>
