INTER-LAYER – Data and Semantics to Data and Semantics (DS2DS): Syntactic Translation of INTER-IoT messages
(Getting started) Scenario description
One of the ways in which INTER-IoT facilitates interoperability between platforms is the translation of messages.The translation process, divided into two parts - syntactic and semantic - ensures that the messages sent from platforms are understandable by INTER-IoT components, such as INTER-MW, and vice-versa. The semantic translation is done by IPSM component that INTER-MW configures and communicates with. Syntactic translation in INTER-IoT is implemented in INTER-MW bridges, although the translators can be used as a separate library implementation.
Technically, because platforms in INTER-MW are connected to bridges, the translation process starts there. The syntactic translation component is responsible for conversion between a format used by the platform/bridge (e.g. JSON, XML, YAML, etc) called FormatX and INTER-IoT JSON-LD used by IPSM and INTER-MW. The JSON-LD has a simple structure - it consists of two RDF graphs - metadata and payload. Contents of platform messages need to be put into the payload part of JSON-LD message. metadata describes other information about the message itself and is used by various INTER-IoT components. Because technically JSON-LD is just one of possible serializations of RDF, the syntactic translation can output RDF in any format, as long as it is a single graph.
The messages that leave the bridges that implement syntactic translation need to be in INTER-IoT JSON-LD and go through IPSM towards INTER-IoT. IPSM translates semantics of RDF payload, so that it can be parsed and understood by INTER-MW components and applications. metadata is, by default, not translated by IPSM.
A message that goes from INTER-IoT towards a platform is first semantically translated by IPSM, and the syntactic translation component needs to convert JSON-LD payload into FormatX. After passing IPSM the semantics of the message should be understood by the platform i.e. it should contain data structures and vocabularies that the platform is familiar with. Syntactic translation presents the data from INTER-MW JSON-LD in syntactic format that the bridge can send directly to the platform.
This recipe describes the process of producting a syntactic translator for a generic FIWARE platform.
Recipe ingredients
- FIWARE platform with a known data model
Prerequisites
- Java IDE with Java 1.8 SDK
- Apache Jena library, version 3 or above
- (optional) IPSM deployment for verification
- (optional) INTER-MW deployment with FIWARE bridge for verification
(How to Do it) The process
Preparation of message translation is a multi-step process and may require additional design and gathering knowledge before implementation. How complicated it is in practice, depends on how much information is available when a platform owner decides to connect the platform to INTER-MW.
Learn RDF information model
Because INTER-IoT messages are in JSON-LD, making a syntactic translator requires knowledge about RDF and libraries that operate on this format. Current implementation of the Messaging component in INTER-MW uses the Jena library, so it is recommended for implementation of syntactic translation.
Analyze your information model
In order to prepare a good syntactic translation an in-depth knowledge of FormatX is required. This includes the data structures, data types, format of identifiers, allowed encoding, special characters, and any other special considerations that may apply to the data format the the platform uses. If a platform uses more than one format it is recommended that they are tackled one at a time, as the lessons learned when implementing translation for one format may apply to the other formats. FIWARE platofmrs use NGSI compliant JSON format that differs slightly between versions. Here, we assume that the current - version 2 - is used, so FormatX means simply NGSIv2 JSON.
Prepare two way format transformation between your format and RDF
Syntactic translation between FormatX and RDF requires a design of a mapping between RDF and FormatX. This mapping can be simple or complicated, depending on how expressive FormatX is.
There is a number of design decisions that need to be made when mapping FormatX to RDF and vice-versa:
- The identifiers of FormatX can be mapped to RDF identifiers (i.e. URIs), or to a property assertions (e.g. "hasID" RDF property). Using RDF identifiers produces a more expressive RDF code and, simply put, a better graph. However, this requires a possibility of conversion if identifiers to URIs. The identifiers may not contain characters that are not allowed in an URI, and the URI base needs to be decided on. The same considerations apply to conversion of FormatX types and RDF types.
- A "naive" representation of FormatX in RDF should always be possible. This means mapping elements of FormatX (e.g. attributes, arrays etc.) to RDF entities that describe the respective syntactic elements (e.g. RDF class of Attribute, Array, etc). Such conversion is often technically simple, but verbose (produces large RDF) and may require more computation time than a more "intelligent" conversion. Depending on FormatX it may be possible to infer additional knowledge about data in FormatX e.g. is a parameter value is a two element array, often the two elements have very specific meaning (e.g. weight and height). In such cases it is better to put those elements in RDF as separate property assertions (for hasHeight and hasWeight properties), instead of an RDF Array entity with a name and two array elements.
- Existing syntactic translators may be reused, especially if FormatX is similar to another format, already syntactically translated. For instance, is two platforms use JSON, it is very possible that existing code to conversion of platform 1 JSON will be useful as a starting point for platform 2 JSON.
- The assumption that the syntactic translators are part of bridges is important. In other words the syntactic translator should not be viewed as a separate component that communicates with the bridge. This is because "context" information i.e. information that is known about the message, but not contained in FormatX, may influence the syntactic translation. The context information may be, for instance, REST parameters that were used in a REST call, that responded with a FormatX message. If syntactic translators are separated from the bridge, there is a risk of losing this information. Because of the above, it is recommended that syntactic translation is done in the bridge.
- We recommend using the Translator/Transformer code framework described below, as a way to produce modular and reusable code. INTER-MW code uses Jena, so it is recommended as the RDF library.
(Optional) Make an ontology
After the syntactic translation is done, the message is encoded in RDF and has some given semantics (i.e. semantics of the platform). These semantics should be made explicit in the form of an ontology, or an RDF vocabulary. Making, or acquiring an ontology is the initial step for semantic translation. In simple terms, it involves description of the structure of RDF resulting from syntactic translation, including used RDF classes and taxonomy, vocabulary (or hierarchy) of attribute names, datatypes and so on. This step is not absolutely necessary for people that are experts, when it comes to FormatX and underlying data model, but it is very helpful, and doubles as extra documentation.
(Optional) Make alignments to Central Ontology
In order for IPSM to translate messages semantically it needs to be configured with an alignment in IPSM Alignment Format (IPSM-AF). The alignments are prepared after an analysis of the ontologies prepared in previous step. The process and tools used to create alignments are described in other DS2DS recipes. This step in only required, if you want to implement the full two-step semantic and syntactic translation. In any case, the syntactic translation can work independently, and none of its components require IPSM.
(How to Do it) Code Design
The syntactic translators code base prepared for INTER-MW is in the https://git.inter-iot.eu/Inter-IoT/syntactic-translators.git repository. Concrete implementations can be found in all pilot, demo, and open call bridges prepared for INTER-MW.
FormatX and Data flow
The base code of syntactic translators in the package eu.interiot.translators.syntax defines one abstract class: SyntacticTranslator<FormatX>. The generic
public abstract Model toJenaModel(FormatX formatX)
public abstract FormatX toFormatX(Model jenaModel)
Once implemented, each method is responsible for syntactic translation in one direction - either towards INTER-MW, or towards the platform (towards FormatX). Each method accepts a different format than it outputs. Messages towards INTER-MW are translated to RDF in a Jena Model. One can then use the setPayload(Model model) method on an instance of a Message class from INTER-MW Messaging to put the translated RDF in INTER-MW JSON-LD.
The constructor of SyntacticTranslator<FormatX> takes two String arguments. The formatName is a human-understandable name of the format managed by the translator e.g. "FIWARE NGSIv2 JSON" and is intended only as a human-readable information for users. The baseURI is the base for the URIs of any RDF entities produced by this translator e.g. "http://example.com/myOntology#". This parameter is recommended to be used when constructing RDF code within the translator, and should be the default URI base, provided that the ontology of the platform that uses the translator does not require other specific URIs for its entities.
Transformers
While the SyntacticTranslator<FormatX> itself is responsible for changing the syntactic format of messages between FormatX and RDF, it can also be configured to use Transformers, which are components that change the contents of a message within one format. Two "archetypes" of transformers are defined - FormatXTransformer and JenaTransformer.
FormatXTransformers change one FormatX message into another FormatX message. The intention behind them is to provide a mechanism of adjusting the messages between the platform and syntactic translator. This mechanism can be used, for instance, to change a message, so that it is compliant with different version of the same protocol. This is useful, in case, where a syntactic translator is available for one version of FormatX protocol, and the platform only accepts FormatX messages for another protocol version. Assuming that different protocol versions are similar, there may be a way of transforming them between versions. In such cases, FormatXTransformers need to be implemented.
JenaTransformers change RDF code into another RDF code, in the place between the syntactic translator and IPSM or INTER-MW. They are intended to adjust semantics of RDF messages, so that it better fits the platforms ontology, or the semantics that are accepted by the syntactic translator.
The idea of transformers is a realization of modularization of syntactic translation. The core component of the syntactic translation is meant to be reusable, generic and semantics-independent. Transformers may be attached to the generic component in order to make the syntax translation more specific for a given platform deployment. Given that every deployment has its own protocol versions, acts in some specific domain, and may have custom extensions, transformers allow for customization and betterment of syntactic translation, while still reusing as much of existing code, as possible.
(How it works) FIWARE Example
Demonstration code for FIWARE syntactic translator is in the test/FIWAREDemonstration.java class. The main method there loads the *.json files from text/resources/FIWARE/v2 directory and applies the syntactic translator to convert FIWARE NGSIv2 JSON to Jena Model, and back to JSON, simulating a two-way syntactic conversion.
The implementation of the FIWARE syntactic translator is in the eu.interiot.translators.syntax.FIWARE package, where FIWAREv2Translator is the class that implements syntactic translation for NGSIv2 JSON.
The FIWARE JSON structure is assumed to be compliant with documentation available at: http://fiware.github.io/context.Orion/api/v2/latest/ and https://fiware-orion.readthedocs.io/en/master/user/walkthrough_apiv2/index.html
FIWARE JSON RDF Vocabulary
The (configurable) URI base has the default value of "http://inter-iot.eu/onto/syntax/FIWAREv2#". In this document we use the "fw" prefix.
The simple vocabulary for this translator has the following classes (types):
fw:Entity
fw:Attribute
fw:Metadata
fw:Array
fw:ArrayElement
fw:Value
The following properties are defined:
fw:hasId
fw:hasType
fw:hasValue
fw:hasName
fw:hasAttrValue
fw:hasAttribute
fw:hasEntity
fw:hasMetadata
fw:hasNumber
fw:hasElement
The properties and classes correspond to the relevant elements of FIWARE NGSiv2 JSON message. According to the documentation (link above) the JSON message can be a single value, an array, or an object (i.e. any of allowed JSON types).
Simple value
A single value is always translated to RDF as an instance of type fw:Value and a fw:hasValue attribute that stores the actual value. The fw:Value class is only used in this case.
Example: FIWARE NGSiv2 JSON- simple value
Input NGSiv2 JSON
23.0
Corresponding RDF (translation)
```[ a http://inter-iot.eu/syntax/FIWAREv2#Value ;
http://inter-iot.eu/syntax/FIWAREv2#hasValue
] . "23.0"^^http://www.w3.org/2001/XMLSchema#float
Note that such information without context is often useless for INTER-IoT, as it cannot be connected with any device, observation, or even measurement unit. The context in NGSiv2 is provided by the parameters of a REST call. For instance, the message in example above may be the answer for a query about specific attribute value, of a specific device. In order to be useful for INTER-IoT this additional context information should be included in the message. Syntactic translation of such values is included for the sake of completeness, but remember that INTER-IoT is not able to guess any additional information from a simple message that contains only a single value.
##### Array
A JSON array is translated to an object of type fw:Array. This object has a number of fw:hasElement property assertions that connect it to a number of individuals of type fw:ArrayElement. Each fw:Arrayelement has an index in the array, denoted by the fw:hasNumber property, and a value fw:hasValue, that stores the actual value of the array element. It can be any FIWARE type e.g. fw:Entity, fw:Array (for nested arrays), or a value.
This translation has the same form, regardless of where in the JSON structure the array is places (i.e. whether it is a top-level entity, or a value of an attribute).
*Example:* FIWARE NGSiv2 JSON- array
_Input NGSiv2 JSON_
```json
[
720,
23,
true
]
Corresponding RDF (translation)
[ a <http://inter-iot.eu/syntax/FIWAREv2#Array> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasElement>
[ a <http://inter-iot.eu/syntax/FIWAREv2#ArrayElement> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasNumber>
"1"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
"720"^^<http://www.w3.org/2001/XMLSchema#int>
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasElement>
[ a <http://inter-iot.eu/syntax/FIWAREv2#ArrayElement> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasNumber>
"2"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
"23"^^<http://www.w3.org/2001/XMLSchema#int>
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasElement>
[ a <http://inter-iot.eu/syntax/FIWAREv2#ArrayElement> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasNumber>
"3"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
true
]
] .
Object
An NGSiv2 JSON object can either be, or not be, a FIWARE Entity. According to the documentation, every such Entity has a JSON attribute 'id', in which case it is translated to an individual of type fw:Entity. Otherwise, the fw:Entity type is not asserted. The rest of the translation process is identical for Entity and other objects.
There is a number of JSON attributes that have special meaning, and are treated differently by the syntactic translator. Those are "id", "type", "metadata", and "value". The rest of the attributes are translated with a generically. Each generic JSON attribute corresponds to an instance of fw:Attribute that has a name (fw:hasName) and a value (fw:hasValue). The instance under fw:hasValue stores the JSON value, type, metadata, and any generic sub-attribute that it may have.
The non-generic attributes are parsed into their own RDF properties and classes. The "id" attribute is translated into fw:hasID, and type goes into fw:hasType (both are data properties). Metadata is translated into an instance of fw:Metadata and linked with the fw:hasMetadata property.
The JSON value attribute is translated into fw:hasAttrValue. This is done in order to distinguish it from fw:hasValue, which is used to model any value of an attribute, not the value of "value" attribute.
Because values of attributes in NGSiv2 can be JSON objects or arrays, not just plain values, the translator supports any level of depth when translating nested objects and arrays.
Example: FIWARE NGSiv2 JSON - non-Entity object
Input NGSiv2 JSON
{
"metadata": {},
"type": "Float",
"value": 23
}
Corresponding RDF (translation)
[ <http://inter-iot.eu/syntax/FIWAREv2#hasAttrValue>
"23"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasMetadata>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Metadata> ] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Float"
] .
Example: FIWARE NGSiv2 JSON - Entity object
Input NGSiv2 JSON
{
"id": "Room1",
"temperature": {
"metadata": {
"accuracy": {
"type": "Float",
"value": 0.8
}
},
"type": "Float",
"value": 26.5
},
"type": "Room"
}
Corresponding RDF (translation)
[ a <http://inter-iot.eu/syntax/FIWAREv2#Entity> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasAttribute>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Attribute> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasName>
"temperature" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ <http://inter-iot.eu/syntax/FIWAREv2#hasAttrValue>
"26.5"^^<http://www.w3.org/2001/XMLSchema#float> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasMetadata>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Metadata> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasAttribute>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Attribute> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasName>
"accuracy" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ <http://inter-iot.eu/syntax/FIWAREv2#hasAttrValue>
"0.8"^^<http://www.w3.org/2001/XMLSchema#float> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Float"
]
]
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Float"
]
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasId>
"Room1" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Room"
] .
As mentioned before, an NGSiv2 message can be an array with multiple Entities, like in the example below.
Example: FIWARE NGSiv2 JSON - multiple Entitiy objects
Input NGSiv2 JSON
[
{
"id": "Room1",
"pressure": {
"metadata": {},
"type": "Integer",
"value": 720
},
"temperature": {
"metadata": {},
"type": "Float",
"value": 23
},
"type": "Room"
},
{
"id": "Room2",
"pressure": {
"metadata": {},
"type": "Integer",
"value": 711
},
"temperature": {
"metadata": {},
"type": "Float",
"value": 21
},
"type": "Room"
}
]
Corresponding RDF (translation)
[ a <http://inter-iot.eu/syntax/FIWAREv2#Array> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasElement>
[ a <http://inter-iot.eu/syntax/FIWAREv2#ArrayElement> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasNumber>
"1"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Entity> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasAttribute>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Attribute> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasName>
"pressure" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ <http://inter-iot.eu/syntax/FIWAREv2#hasAttrValue>
"720"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasMetadata>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Metadata> ] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Integer"
]
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasAttribute>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Attribute> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasName>
"temperature" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ <http://inter-iot.eu/syntax/FIWAREv2#hasAttrValue>
"23"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasMetadata>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Metadata> ] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Float"
]
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasId>
"Room1" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Room"
]
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasElement>
[ a <http://inter-iot.eu/syntax/FIWAREv2#ArrayElement> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasNumber>
"2"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Entity> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasAttribute>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Attribute> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasName>
"pressure" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ <http://inter-iot.eu/syntax/FIWAREv2#hasAttrValue>
"711"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasMetadata>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Metadata> ] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Integer"
]
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasAttribute>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Attribute> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasName>
"temperature" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasValue>
[ <http://inter-iot.eu/syntax/FIWAREv2#hasAttrValue>
"21"^^<http://www.w3.org/2001/XMLSchema#int> ;
<http://inter-iot.eu/syntax/FIWAREv2#hasMetadata>
[ a <http://inter-iot.eu/syntax/FIWAREv2#Metadata> ] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Float"
]
] ;
<http://inter-iot.eu/syntax/FIWAREv2#hasId>
"Room2" ;
<http://inter-iot.eu/syntax/FIWAREv2#hasType>
"Room"
]
]
] .
Datatypes
The syntactic translator adds datatype information to RDF, and reads it back in order to serialize NGSiv2 JSON with correct datatype. The following datatypes are converted between JSON datatypes and RDF typed literals: Int, long, float, double, boolean. All other datatypes are assumed to be, and serialized as, strings.
Summary
The FIWARE syntactic translator implements a representation of the FIWARE NGSiv2 JSON in RDF. The structure of this JSON can be quite complicated and it allows nested structures e.g. attributes whose values are objects, which in turn have their own attributes and metadata. Therefore the translations can be quite complicated, and the translator itself must support every possible message. For this reason, no shortcuts were available when programming it. The RDF code produced and accepted by it is quite close to a universal representation of any JSON message in RDF, not just NGSiv2. Thus, it can be used as a starting point for implementation of other syntactic translators for systems that use complicated JSON. If the JSON structure is more restricted, the NOATUM WSO2 example might be a better starting point.
FIWARE Transformers
The implementation of FIWARE syntactic translation makes heavy use of anonymous RDF nodes. However, some FIWARE entities have identifiers that could be used as RDF URIs, or their fragments. The SimpleURIRefTransformer is an implementation of the JenaTransformer that does just that. It is configured with a URI base (by default "http://inter-iot.eu/syntax/FIWAREv2#"), and a FIWARE property type ("Ref"). When transforming to FormatX, any FIWARE entity that can store properties, and is identified by an URI in Jena model (i.e. it's not anonymous) gets a new FIWARE property of type "Ref", whose value is the URI. This way, the information from RDF is fitted into FIWARE JSON, and not lost in the process of translation. Conversely, when a message is translated to RDF, the transformer attempts to build meaningful URIs, either directly form the "Ref" property (if it exists), or by attaching the configured URI base to a html escaped FIWARE entity identifier.
Using this transformer makes the connection between FIWARE identifiers and RDF identifiers more explicit.